How Can Resembler's Accident Database Improve Autonomous Vehicle Safety Testing?

Why testing to Accidents and Incidents

Traditional testing methodologies based solely on adherence to traffic regulations fall short of encompassing the complex, dynamic scenarios encountered on real-world roadways. While compliance ensures a foundational level of performance, it does not account for the full operational envelope or the unpredictable nature of actual accidents. In this analysis, we present three case studies—highlighting incidents involving Tesla, Uber, and Cruise—that generated significant social media backlash. Our evaluation illustrates that implementing A-B testing using empirical accident data from the Resembler database would have enabled engineers to detect critical vulnerabilities, thereby facilitating more robust risk mitigation strategies and potentially preventing these high-impact events.

Resembler Database Coverage

The Resembler database already has coverage of 500 million kilometres of real-world near-misses and accidents. Although a large amount of this data originates from Singapore, our analysis is focused on evaluating whether the three incidents could have been averted by implementing A-B testing against the Resembler database. This enables us to validate engineering assumptions and identify latent vulnerabilities that traditional testing methods might overlook.

Notable Incidents

Tesla’s Collision with an Overturned Truck

Tesla's advanced driver-assistance system (ADAS) did not identify an overturned truck as recorded on a Taiwan expressway. We verified the Resembler database, which contains multiple examples of overturned trucks, including instances of human driver near-misses in similar situations, as well as captured events of near misses where the truck overturned in front of the vehicle.

All these scenarios are successfully selected for highway and expressway ODDs and are available in CARLA simulation format to facilitate verificaiton.

Tesla model 3 driving towards the truck on a busy highway in Taiwan https://www.youtube.com/watch?v=X3hrKnv0dPQ

Cruise Vehicle VRU-Pedestrian Incident

In the Cruise incident, the sequence of events deviated from the standard collision scenario. An initial collision occurred between a non-autonomous vehicle and a pedestrian, which unexpectedly resulted in the pedestrian being propelled into the path of the AV vehicle.

It would be frowned upon if created as a synthetic test. However, the Resembler database has many instances of similar "erratic" and unforeseen accidents that are more often than not avoided by the very "smart" behaviour of human drivers.

Uber's Incident

In the Uber accident, a pedestrian pushing a bicycle in front of the path of the AV vehicle was visible for quite a while. Although the pedestrian did not follow traffic rules, this is a very common situation, and the Resembler database has multiple records of these events in different and adversarial lighting conditions and road topologies.

A-B testing with accidents and near misses as a proxy

Resembler provides a dataset and conversion tools—some in collaboration with Singapore CETRAN. This allows A-B testing to place the AV vehicle in the exact same situation as prior recorded incidents and accidents without necessarily studying the perception, planning, intention, or control system. Sometimes this is preferred as we do not always know the human perception and reasoning.