Has NHTSA backed itself and the Autonomous Vehicle Industry into a corner?
NHTSA recently announced it was investigating the twelve Tesla “autopilot” crashes into emergency vehicles in the United States. While this could wind up being a critical paradigm shift from enabling these issues, it could also make things far worse. Having said that, it does seem it has to do something, or it would not have isolated emergency vehicles from every other crash it is already investigating.
Questions, Observations and Concerns
Why did NHTSA feel the need to isolate the investigation to emergency vehicles?
NHTSA was already investigating many of the other Tesla AP/FSD crashes in the US. To date at least five people have been killed while using the system in the US. Nine total worldwide. (I say “at least” because several others have not been confirmed to be on AP yet.) While a dozen crashes into emergency vehicles have occurred, there have been far more crashes, including those deaths, going back six years to Joshua Brown. (And far, far more near calamities that were avoided by human Guinea pigs). Why did NHTSA have to add emergency vehicles to the mix, especially since it has been dragging its feet and enabling the status quo? To help inoculate itself from the Tesla machine and an echo chamber industry? After all who would fight an investigation into emergency vehicles? Or does NHTSA want to split hairs, deal only with emergency vehicles and let the public stay at risk? Given those emergency vehicles will likely involve the public in future crashes, that would be a ridiculous supposition. It seems to me NHTSA’s hand has finally been forced. How can they keep letting emergency vehicles be hit? They must be aware that it was pure luck no one, especially a member of the public or the first young child or family was killed in one of those emergency vehicle crashes. (In Japan, Yoshihiro Umeda was killed by a sleeping Tesla AP driver when he was run over while being attended to by people after he had already been involved in a prior crash. He had a wife and young daughter.)
Is the sensor system design root cause understood? And that it involves Automatic Emergency Braking as well?
Does NHTSA understand the sensor system design is fatally flawed and affects AP/FSD and AEB? The system does not properly detect stationary or crossing objects. A situation that likely has been made worse by eliminating the poorly performing radar.
The system routinely ignores stationary/crossing objects. This issue involves a combination of two areas, object classification and recognition and object location. Both now handled by the camera system and deep learning. Prior to removing the radar Tesla utilized radar for object location. It also acknowledged that the system struggled in these areas because of the radar. (Tesla also refuses to use LiDAR. It should be noted however most AV makers use LiDAR for determining position and object depth, not specific object detection, classification, or position. This is because they do not create “tracks” as radars do inherently. Some AV makers are now moving in this direction.) The radar Tesla utilized only had a couple transmitters and receivers. This forced the beam pattern to be wide. As distance increases that beam envelopes the entire road, areas to the sides and all objects within it. This leaves the system to merge objects as well as be unable to determine if they are on the road, next to the road etc. To avoid false breaking the system often ignores the objects. That leaves the cameras to do all the work. (I should note that mode L1+ systems out there use these low fidelity radars and have similar issues. However, they tend to be in L1 ADAS systems where the drivers not only do not cede steering and control of the vehicle. Having said thus everyone should evolves to high-fidelity radars or even LiDARs that produce tracks.
Given the plethora of information out there regarding this issue, both of crashes that have occurred and associated driver disengagements, before and after Tesla removed the radar input, it is clear this issue is systemic. If all the disengagement data were checked, NHTSA would likely see the issue is commonplace. Meaning every car out there with AP/FSD would have similar crashes if the drivers did not disengage. All 500k+ of them. This alone should be enough to impose the “autopilot” moratorium. (With respect to AEB, it needs to be determined if there is an inherent flaw in AEB or it only exists when AP is in use.) Beyond that is the ease at which this can be investigated. NHTSA need only acquire the Autopilot disengagement, design, and vehicle system data. As well as the AEB design and system performance data to verify it is systemic. (In addition to Tesla admitting the problem in the press, manuals, and NTSB crash report findings.) Worse case the scenarios can be easily replicated on a test track. Even if the root cause is the deep learning system not recognizing some objects versus the camera system determining their position, the moratorium and ease of investigation still applies. (It should be noted that camera systems struggle with direct light, weather, complex object patterns and 2D objects. This results in objects not being classified, detected or their position not being correct. Recently that included the moon being confused for a yellow light. The car braking as a result.
Note — At one point Tesla said they evaluated the dense array high-fidelity Arbe radar. They chose not to use it. That was a major mistake NHTSA needs to follow up on. Why wasn’t it used. I believe it is because Tesla’s main board has a major processing issue. It cannot ingest another sensor no matter what it is. And keep in mind radars produce tracks vs massive amounts of raw data bits like LiDAR. That means radars create a low processor load. Another argument for this point is Tesla getting rid of the existing radar. While it did not have high fidelity, as I mentioned above, it did have capabilities cameras do not. Tesla said they got rid of the radar because it did not perform well. (This after saying radar was crucial years ago) Why didn’t they just adjust the Kalman filter to minimize the radar’s issues? Again, I think this is because the main board can’t handle it.
The fatally flawed “Driver Monitoring” System
20 seconds or more for the system to alert the driver they need to take back control of the vehicle is virtually useless. Do the math on how far one goes in that time at various speeds. Fact is Tesla doesn’t want drivers to disengage. If they do many crash scenarios/threads cannot be tested and learned. (Beyond this there is the myth that handover can be made safe in time critical scenarios or that “safety driving”, especially with untrained/public drivers interacting with the public, is necessary at all. More on that below.)
Why would the investigation not already be complete or take more than a couple weeks to reach a preliminary finding?
Given the clear sensor system design flaw why has this taken six years to investigate, especially given this is easy to determine with a test car on a track duplicating crash scenarios? While getting in the weeds is necessary, NHTSA should be concerned with there being a fatal design flaw and stopping future crashes, injuries, and deaths. The why, below the obvious fact that the sensor system is flawed, is far less important.
Why isn’t there a moratorium?
Why isn’t NHTSA imposing an immediate moratorium while it conducts its investigation? In the time between it announced the investigation and the letter it sent to Tesla asking for information, another police car was hit.
Do they continue to allow the needless use human Guinea pigs?
The elephant in the room is Tesla’s use of their customers, others in the cars and the public around them, as human Guinea pigs. Tesla and most of the industry utilize machine learning to train the systems to handle scenarios. Machine learning “learns” by experiencing scenarios, trying to drive them correctly, failing, being corrected, and continuing that loop until complete. The repetition required could be hundreds if not thousands of times. And due to the current inability of the systems to infer or think, versus scenario and object matching and associated execution recall, it requires massive amounts of trial-and-error repetition to learn. In addition to that, especially when deep learning is involved, the systems scan objects from inside out and hyper classify them. They do this so they can learn the movement patterns of various object types or assign rules. Examples would be a person jogging and a sign. To apply those movement expectations or rules they must memorize enough detail about each item to classify them properly. This process has a nasty unintended side effect. It causes the system to get lost or confused when it sees very small differences in new or what it thinks is a new object. Dirt or branches in front of a sign or clothing patterns for example. To handle this the system needs to learn an insane number of objects and their variations. This brings us to why this process is untenable from a time and money perspective. RAND estimated it would take 500 billion miles to reach L4, where the system drives 10X better than a human. Toyota’s Gil Pratt said it would be a trillion miles. My very conservative math, for just vehicles, sensors and drivers and no engineering costs, which is the bulk of the expense, is $300B in 10 years.
Back to safety and the use of human test subjects. For many of the crash scenarios to be learned the system must experience them. While some scenarios cover others, current simulation technology helps and the system can learn enough from the moment the human disengages to avoid the impact of a crash, many of the scenario threads will have to be experienced in the real-world. That will cause the injury and deaths of thousands of people over time, especially when the threads involve a progression of interactions and steering, braking or acceleration actions. (Imagine complex scenarios in the snow when traction is lost.) The reason most of this is not accomplished in simulation is two-fold. It is believed there is no simulation system good enough to replace the real-world, which I will get to next, and you cannot make up or create enough scenarios in the real-world, especially edge cases. You must use the real-world to stumble on them.
Let me address the latter belief first. The issue here is time and money again. In Tesla’s Industry Day a year ago, Elon Musk used an example of a tractor trailer tractor towing several other tractor trailer tractors as an edge case. While I do not believe that example is rare enough to be an edge case, let’s go with it. Due to machine and deep learning requiring mass amounts of repletion to learn scenarios and objects and the need to learn variations of them, how many lifetimes do you think it would take to learn just Elon’s example? How many eons will go by so a car can stumble on that exact set of objects? Now how many more to stumble on the massive amount of color, position, quantity, environmental and other variations? Now multiply that by all the scenarios and their variations that need to be learned independently to get to L4. It’s insane.
(Of course, the elephant in the room here is that the industry uses the same basic development and testing process and human Guinea pigs. Tesla is far more egregious for the reasons stated here. As well as has far more vehicles with AP/FSD than all the rest combined. While this all means Tesla will harm or more people needlessly, the others will do the same. Does NHTSA have the courage to deal with this problem industry wide? Does it try to split hairs with Tesla? Or just continue to ignore it?)
The remedy is proper simulation
How do we remedy the situation? With simulation. First, we still use the real-world. Only much less of it. We still use shadow drivers (who maintain control of the vehicles) to learn the real-world. We then take that data, the plethora of data we have on objects, locations, weather, road patterns, driving patterns, crash data etc, and create scenarios and their variations using scenarios generation and coverage tools and Monte Carlos. Keep in mind we do not need to learn all the scenarios possible. Only enough to demonstrate due diligence and the statistical probability the system is some factor better than a human at driving. Yes, that is a lot of work. But it is doable. Where it is impossible in the real-world.
An associated reason the industry thinks it needs to rely on the real-world vs simulation is that simulation can’t replicate enough of the real-world to replace most of it. This is based on the extended belief that there is no existing simulation technology that can handle the processing loading or make models with high enough physics fidelity. (It is well established that the visual engine folks like Unreal and Unity, can make realistic enough visual scenes and objects.) Given the technology being used, they are correct. But that is the rub. The technology being used now is based on gaming-engine architectures and modeling approaches. Progress can be made using the gaming-based systems. They however have limited capability. Problems occurs when the fidelity or system run time degrades to a point there is negative development and testing. Often that will be hidden and cause false confidence. If not discovered that would lead to real-world tragedies. This is because the AV system will implement a plan that is not close enough to the real-world. That will wind up with timing or veracity flaws in maneuvering, acceleration, or braking. And in the case of sensors, especially when there is complexity and interference, it could be totally wrong. If aerospace/DoD/FAA level simulation technology is used, this is all resolved. (More on this below. And please ignore the fact that those are planes, or that air travel is not nearly as complex as the streets we drive on. What the model is called is irrelevant. And DoD deals with war games in urban environments that are more complex than what this domain needs because they include electronic warfare.)
Finally, there is a small group within USDOT that gets all of this. It is called VOICES. They are trying to leverage DoD to create a simulation environment to assist the industry to affect the necessary development, testing, and simulation technology paradigm shift. The problem is they are being drowned out by the larger USDOT organization and NHTSA echo chambers.
More detail here. Including how to do this right.
NHTSA should impose an immediate “Autopilot” moratorium and report initial investigation findings in 30 days
The Autonomous Vehicle Industry can be Saved by doing the Opposite of what is being done now to create this technology
SAE Autonomous Vehicle Engineering Magazine — Simulation’s Next Generation (featuring Dactle)
· https://www.sae.org/news/2020/08/new-gen-av-simulation
How the failed Iranian hostage rescue in 1980 can save the Autonomous Vehicle industry
USDOT introduces VOICES Proof of Concept for Autonomous Vehicle Industry-A Paradigm Shift?
Tesla “autopilot” development effort needs to be stopped and people held accountable
NHTSA Opens Probe on Tesla’s “Autopilot” Crashes with Parked Emergency Vehicles
My name is Michael DeKort — I am a former system engineer, engineering and program manager for Lockheed Martin. I worked in aircraft simulation, the software engineering manager for all of NORAD, the Aegis Weapon System, and on C4ISR for DHS.
Industry Participation — Air and Ground
- Founder SAE On-Road Autonomous Driving Simulation Task Force
- Member SAE ORAD Verification and Validation Task Force
- Member UNECE WP.29 SG2 Virtual Testing
- Stakeholder USDOT VOICES (Virtual Open Innovation Collaborative Environment for Safety)
- Member SAE G-34 / EUROCAE WG-114 Artificial Intelligence in Aviation
- Member Teleoperation Consortium
- Member CIVATAglobal — Civic Air Transport Association
- Stakeholder for UL4600 — Creating AV Safety Guidelines
- Member of the IEEE Artificial Intelligence & Autonomous Systems Policy Committee
- Presented the IEEE Barus Ethics Award for Post 9/11 DoD/DHS Whistleblowing Efforts
