Driving autonomy into the metaverse

Nearly 400 crashes were linked to self-driving cars in the 10 months between July 2021 and May 2022. There is a critical need for manufacturers and control boards to understand these safety issues in order to earnestly embrace autonomous vehicles. Researchers at Carnegie Mellon are driving autonomy into the metaverse to understand safety-critical scenarios by way of digital twins.

Because car crashes are rare, affecting the average driver only once every 18 years, and heavily dependent on diverse environments like individual driving habits and road conditions, gathering data in the real world on safety-critical scenarios is nearly impossible. Students in Ding Zhao’s lab proposed the possibility of gathering that data in the metaverse to generate diverse safety-critical scenarios that can quickly adapt to various environments to provide numerous realistic testing cases, without the need to track hundreds of miles.

“Autonomous driving has demonstrated promising potential to reduce crashes, save people’s time, and combat climate change,” explain Zhao, an assistant professor of mechanical engineering. “But it’s evident that the guarantee of safety is still missing. We want to develop that missing piece for the large-scale deployment of self-driving.”

One way to generate safety-critical autonomous system scenarios is by finding failure cases by trial and error. Digital twins provide perfect test beds to create safety-critical scenarios without causing damage in the real-world. Wenhao Ding and Jiacheng Zhu, Ph.D. students in Zhao’s lab, proposed a new method, Learning to Collide, to identify risky scenarios leveraging the reinforcement learning technique. This method builds a framework where the autonomous system is a victim attacked by the scenario-generation algorithm.

Another efficient way to generate desired scenarios in digital twins is using causality, which describes the cause-and-effect relationships between objects. For example, an accident between a pedestrian and a vehicle is caused by another vehicle blocking the view of the pedestrian. The team developed a method called Causal Autoregressive Flow (CausalAF) to generate safety-critical scenarios in autonomous digital twins. It uses causality summarized by human experts and enables efficient generation to find diverse risky scenarios to self-driving vehicles.

We are building the bridge between the digital world and the real world, and we believe that it is the most efficient way to ensure the safety of people using digital systems.

Wenhao Ding, Ph.D. student, Mechanical Engineering

“What we are doing is unique because typically the cause of inference is studied by statisticians for theory, but we are applying it to the real-world,” said Ding. “We are building the bridge between the digital world and the real world, and we believe that it is the most efficient way to ensure the safety of people using digital systems.”

Ding and Zhu’s proposal, “Safety-Critical Scenarios Generation and Generalization for Autonomous Driving'' was awarded the 2022 Qualcomm Innovation Fellowship, which promotes innovation, execution, and teamwork.

“I am very excited to get this prestigious award,” said Zhu. “When we gave the presentation in the final round of competition, I noticed that we were up against very novel topics from students at other leading institutions. I am glad our work was recognized.”

I am happy to see that my students are controlling their own success.

Ding Zhao, Assistant Professor, Mechanical Engineering

“I am happy to see that my students are controlling their own success, and this award reflects that,” Zhao stressed. “They proposed this concep,t and now they get to drive the digital twin development.”

Understanding these scenarios extends beyond the road, as the use of autonomous machines is actively explored in manufacturing. Zhao’s group has created a digital twin of Mill 19, the home of Carnegie Mellon’s Manufacturing Futures Institute to more efficiently and cost-effectively explore safety in manufacturing. This project began as an MFI seed project in April 2022. Each seed project is funded with the goal that the research will be able to secure follow-on funding from external sources.

“Manufacturing scenarios have human-robot interactions, and through a digital twin we can assess working conditions.” Zhao went on to point out, “Scenarios with safety and security problems may be rare, but they do happen. It is critical for us to understand the why in order to prevent recurrences.”