More comments after a more thorough reading
This is by far the most thorough and revealing safety report I have seen. You are to be commended for the hard work and lack of hype.
Excellent staying thongs others do not like mikes and disengagement’s meaningless. And can’t shadow drive the miles. But you under represent that. It’s not billions of miles. It is up to a trillion to be at least 10x a human.
Excellent to mention you need detailed testing and for regulators to be part of that.
Use of frameworks and guidance rarely elicit excellence. Most often they provide legal air cover for the lowest of bars. Anything less than a complete and accurate scenario set for testing here will make things far worse than better.
Excellent you have defect types. Without detailed and thorough testing those defects will not be found.
Your list of scenario types is a good start.
Excellent to mention fidelity of sim environment. But miss real time vehicle tire and road models. Especially in degraded situations.
Page 9 you state simulation can only be used to generate scenarios that are theoretically possible or that have not yet been captured in on road testing. This is a fatally flawed determination. As a matter of fact proper simulation can produce real-world scenarios you may never stumble on once unless you drive the trillion miles or so it would take and in every location for which every core scenario would exist. This statement means you believe you have to drive most scenarios to know they exist? And as such would utilize public shadow driving to test them vs proper simulation? I would like to know if I am interpreting this correctly? What % of development and testing to do believe should or has to be done in simulation vs the real-world? I assume you believe simulation cannot be used most of the time let alone 99%? If so why? Have you seen a DoD FAA Level 5 simulated war game in an urban area? My concern is that if you do not understand you have to use simulation 99% of the time and it has to be proper simulation (not the vast majority of products in this industry) you will not be successful.
You do not mention the need for a full motion simulator for simulator shadow driving as well as to check for driver unease or sickness. Are you aware motion cues are critical to the development and testing of these systems? And that without them the system will be flawed and provide false confidence, especially when conditions are degraded, there is a loss of traction or model performance envelopes are pushed?
You do not mention the need to validate system real-time performance or models, especially in degraded conditions and when performance envelopes are pushed. This includes vehicle, tire and road models. (When manned simulator is used especially full motion added latency to visual, motion, control loading and other systems cannot be more than 16ms,)
It is wrong to state that accident scenarios are edge or corner cases. Accident scenarios are scenarios. The only difference being we do not intend to have them. An actual edge or corner case is something that is not possible or expected in the environment. Like when you search for a picture of a cat and get a tree. The problem with using these terms is that is may limit the amount of due diligence that is done in finding, creating, developing to and verifying a sufficient number and types of accident cases and their variations to reach an acceptable or even paramount state of safety. (Something I see you seem to realize when you mention that testing does have to include infrequent but dangerous scenarios.) The most problematic of these are the millions of scenarios that will need to bet tested to ensure 6 sigma levels of reliability regarding AI perception. Areas where AI can be fooled or confused.