Simulation Photorealism is almost Irrelevant for Autonomous Vehicle Development and Testing
THIS IS NOT A GAME FOR HUMANS
This industry is once again focused on exactly the wrong approach. The IT, Silicon Valley and gaming world keeps trying to drag this domain into what it understands vs going to it and engineering this the right way. First, we had the untenable and reckless process of public shadow and safety driving and the use of bottoms Agile development approaches. Now fantastic looking, but physics limited, gaming-based simulation is crippling this industry.
If you want to create a legitimate digital twin vs a game and actually get to L4 without going bankrupt or harming people for no reason, you have got to stop centering the world on you. That means those pretty pictures for YOU largely do not matter to the autonomous system. What matters mostly by far is physics. And NOT adapted ray tracing physics built in your rendering engine. Or the physics of generic sensors, objects, weather, the world etc. You need EXACT objects and models that model how active sensors actually work. And all done in proper real-time under the most complex scenarios. The ONLY time visuals matter is for the human to play along in the development and testing, to show a system how to drive in imitation learning, to evaluate imitation and reinforcement learning and to train a camera system IF a real camera was looking at the screen. The latter is rarely being done and the items I listed before that do not require even mediocre visuals to be successful.
Why does using Proper Simulation Matter?
This is an issue of matching levels of fidelity to use cases or scenarios. When the right level of fidelity is not used, there will be development flaws. When complex scenarios are run, especially where the performance curves or attributes of any model are exceeded, the Planning system will have a flawed understanding of some aspect or aspects of the real world. The result will be creation of a flawed plan. In far too many scenarios, in the outcome will be creating real-world accidents, not avoiding them or their being worse than they need be. This will usually be caused by some combination of braking, acceleration, or maneuvering being improperly timed or the voracity being incorrect.
The simulation systems that are being utilized today are adequate if you are working on general or non-complex real-world development or testing. However, in order to run complex scenarios, the depth, breadth, and fidelity of the simulation is critical. The Autonomous Vehicle (AV) makers will need to keep track of every model’s capabilities for every scenario to make sure none is exceeded. If AV makers do not do this, especially in complex scenarios, the end result will be the development of false confidence in the AV system. Keep in mind that machine learning does not infer well, not nearly as good as a human as we have a lifetime of learning especially for object detection. Additionally, perception systems right now are far too prone to error. The famous stop-sign-tape test is an example. This means that development and testing must be extremely location-, environment-, and object-specific. You could test thousands of scenarios for a common road pattern in one place or have just a couple object differences, like clothing patterns, and wind up with errors if you think you don’t have to repeat most of that testing in most other locations. This raises the scenario variations into the millions.
What is Proper Simulation?
Real Time — This is the ability for the system to process data and math models fast enough and in the right order, so no tasks of any significance are missed or late compared to the real world. This needs to be extended to the most complex and math-intensive scenarios. And by math intensive, I mean every model must be mathematically precise as well as be able to properly function. That could be thousands running at a time. This is where gaming architectures, which are used by many simulation companies, have significant flaws even with the best of computers. The best way to architect these systems is to build a deterministic or time-based and structured architecture where every task or model can be run at any rate and in any order. Most systems out there are non-deterministic and just let things run. They will say modern computers allow them to do it so fast that the structure I described is not needed. I believe this is wrong. (At an event I attended at Jose De Oliveira, Unity Technology’s engineering manager for autonomy, spoke after me and confirmed my point of view.) The systems that are not deterministic run everything at one time at one specified rate. You can see accommodations made for these issues in gaming. Their play box is less than 60 sq miles and where they can avoid physics or math either by eliminating them, like when you can walk through trees, or dumbing them down as much as possible.
Model Types — The critical model types are: The ego-vehicle, tires, roads, sensors fixed and other moving objects, and the environment. Each of these needs to be virtually exactly like the target it is simulating. (Geo-specific vs geo-typical for example. The best technology can get this down to under five cm of positional accuracy.) This includes not just visual aspects, if they are applicable, but physical capabilities. You need to simulate or model the exact vehicle, sensor, object, etc. Not something like it or a reasonable facsimile. As I said before, this is important because machine learning does not infer well. As I stated above, this means the same road patterns will have to be developed and tested in a wide array of locations, at different times of day, in different weather with different signage, etc. All of this must be extremely articulately modeled both visually and physically. This means modeling how an active senor works in the real-world, not simply showing a visual representation using ray tracing.
Take radar, for example. You must simulate not only the Ego-radar but how the world and other systems interact with it. Every other radar or system emitting RF that would cause clutter or interference must be properly modeled as does how every radar’s signal is being affected by its environment. The reason for this is that the Ego-model’s received signal must be an accurate model of the culmination of all of these factors at any physical location in the environment or scenario. This is where I would like to address vehicle models. The Original Equipment Manufacturers (OEMs), simulation and simulator companies have been creating detailed vehicle models for some time. However, I would caution against assuming they are precise enough in all scenarios — especially in complex scenarios as the simulation companies have not likely instrumented these vehicles in all the relevant scenarios required here to ensure the performance curves are accurate. And keep in mind this is not just a function of the vehicle design data or specs. The model structure itself or the overall system being used could be flawed.
Another example is friction coefficients as they are applied to the road surface and tires. Each tire will experience a different part of the road. Those parts could contain segmented friction values based on the surface composition. It could be dry or wet, be painted, have oil or gravel on it, or any combination thereof. And that combination can be in varying segments under the tread pattern. The models need to properly account for this. Good models can divide these areas into segments of less than a centimeter. (Puddles would be an extension of this. Like the real world, a puddle can cause you to hydroplane or increase friction so much that it pulls your vehicle to the side of the puddle’s location.)
Examples
A Velodyne 128 LiDAR scanning an intersection in various degrees of rain. That LiDARs operation and interaction with exact objects and parts of them needs to be modeled dynamically and in real time and faster. That .23 degree beam at a certain power level progressively interacting with rain drops until it gets to the tire and a polygon returns the reflectivity value for rubber. Only to then progressively interact with rain drops of various density again. (The laser may not survive any part of that rain.)
10 vehicles with the Delphi ESR radar in an exact parking garage. Each radar and the cumulative 1st, 2nd and 3rd reflections, bouncing off exact objects, modeled in and faster than real-time. Or extend that to a packed intersection in NY City with 100 of those radars. The radar returns from every object would include associated RCS and reflectivity values.
Mixed friction coefficients under a tire at any point on the road. An example might be where there is an even 1/3 split under the tread across dry asphalt, a painted line and oil. And at that point a vehicle performs some aggressive operation.
Full-motion Driver-in-the-Loop (DIL) Simulator — When the real-world vehicle is replaced by simulation, you must use one of these devices to properly develop and test the system. This is whether reinforcement or imitation learning is used. The reason for this is humans cannot drive properly nor evaluate proper driving without motion cues or pressure on their bodies or inner ears in several classes of scenarios. The easiest to understand is loss of traction. It is simply not possible to drive or evaluate a system when you are driving in the snow without feeling what is going on. Other examples of this are where there is complex maneuvering, steep grades, running over something, or even bumping or being bumped by something else. Since it is desirable to run simulations faster than real time, the use of this device will be minimal. However, the scenarios in which it should be used for are critical.
(An example of proper real time where ground vehicles are concerned is where there is never more than 16 msec of latency from a driver action to the visual, control, or motion system of a full-motion simulator.)
Ask for Proof of Fidelity
Unfortunately, here is where the industry is at regarding simulation:
· The simulation companies do not know what capabilities are necessary or possible.
· The simulations companies are not utilizing the right or best technology by choice.
Whether unintentional or not, this results in misleading product information being conveyed. That will result in false confidence, flawed Planning, and real-world errors and tragedies. Given this, it is imperative that proof of model fidelity and real-time performance in a wide array of scenarios be provided, reviewed, and confirmed. I know of no simulation company who currently provides this data. (Begging the question: why not if the data is accurate and complete?) This information is critical both in the cases where you want or need to use a true digital twin and where you do not believe you have to do so, but want to ensure you have no negative impacts of that decision.
Some of the ways to validate the models include using the source data like instrumented performance data for vehicle performance, technical data from vendors, High Definition (HD) mapping data, Hardware-in-the-Loop (HIL) testing, satellite data, and, most importantly, data gathered from shadow driving.
Cloud-Based Systems
Cloud-based systems can be treated like local instances about the points I have made here, except for where the DIL simulator is involved. Getting that latency down to 16 msec is probably not going to be possible, especially in complex and loaded scenarios.
DoD/Aerospace Technology is the Solution
First let me address what is usually the immediate reaction upon hearing that DoD technology should be used. DoD does not have to deal with the same complexity as the commercial AV world. That belief is incorrect. The DoD autonomous ground vehicle folks not only have to deal with the same public domain and scenarios as the commercial side, but they must deal with vehicles driving off the roads on purpose, aircraft, folks shooting at each other, and electronic warfare. (That is where the enemy will try to jam, spoof, or overload sensors.) Trust me, the military has it much tougher.
This brings me to the resolution. The fact is that DoD has had the technology to resolve all these issues for well over two decades. And in most cases, like sensors, the target systems are far more complex than anything available in the AV domain today or probably will ever be. Proper and effective, not perfect, digital twins can be created for every model type needed here. And their real-time and federated model architectures can handle any scenario required, independent of complexity, model detail and math, or loading. Now having said this, clearly the effort here is not easy and will take a lot of work. This technology needs data to be tailored to meet the specific needs and targets in this industry. Keep in mind what we are talking about here is the impossible vs the possible, the doable vs the undoable. The current development and testing approaches are not remotely doable in many lifetimes. This makes the value proposition of making the switch brutally obvious from a time, cost, and liability point of view.
(With regard to computing power needed, the architecture being used is so efficient and performs so well that this does not take any special computing assets. In most cases, it will run on the gaming type system being used now. This includes the ability to run much faster than real time when compared to systems that do not use the proper architecture.)
Please find more information on my POV in my articles below
The Autonomous Vehicle Industry can be Saved by doing the Opposite of what is being done now to create this technology
Proposal for Successfully Creating an Autonomous Ground or Air Vehicle
Using the Real World is better than Proper Simulation for Autonomous Vehicle Development — NONSENSE
Please find more information on my POV in my articles below
The Autonomous Vehicle Industry can be Saved by doing the Opposite of what is being done now to create this technology
Using the Real World is better than Proper Simulation for Autonomous Vehicle Development — NONSENSE
How the failed Iranian hostage rescue in 1980 can save the Autonomous Vehicle industry
My name is Michael DeKort — I am a former system engineer, engineering and program manager for Lockheed Martin. I worked in aircraft simulation, the software engineering manager for all of NORAD, the Aegis Weapon System, and on C4ISR for DHS.
Industry Participation — Air and Ground
- Founder SAE On-Road Autonomous Driving Simulation Task Force
- Member SAE ORAD Verification and Validation Task Force
- Member UNECE WP.29 SG2 Virtual Testing
- Stakeholder USDOT VOICES (Virtual Open Innovation Collaborative Environment for Safety)
- Member SAE G-34 / EUROCAE WG-114 Artificial Intelligence in Aviation
- Member CIVATAglobal — Civic Air Transport Association
- Stakeholder for UL4600 — Creating AV Safety Guidelines
- Member of the IEEE Artificial Intelligence & Autonomous Systems Policy Committee
- Presented the IEEE Barus Ethics Award for Post 9/11 DoD/DHS Whistleblowing Efforts
