Engineering meets extreme reality
We need to think about emergency preparedness, and one way to do that is to use both virtual and real-world data to train and equip first responders for dangerous work.
Less than three miles from Carnegie Mellon University, the Fern Hollow Bridge collapsed into the valley below it, making national news. Steep snow-covered slopes on both ends of the bridge created hazardous conditions for first responders. How did they rescue passengers from the bus that was trapped on the bridge? How did they ensure their own safety during their rescue mission? How would they survey the site for post-mortem analyses in the harsh environment?
Yang Cai, senior systems scientist at Carnegie Mellon’s CyLab, pondered these questions. Cai often conducts research outside of his lab, so he drove to the perimeter of the site and was shocked to see the magnitude of the disaster.
“Many engineered systems are not ready for extreme conditions,” and this is unacceptable explains Cai, who has been developing technologies for first responders. He advocates for “extreme reality engineering,” which entails creating innovative technology that can operate under some of the worst conditions imaginable.
“Extreme Reality (EXR) is an extended augmented reality (AR) that works in real or simulated extreme environments,” says Cai.
For more than a decade, Cai has worked on artificial intelligence (AI), augmented reality (AR), and translating cyber data into visual, audio, and tactile signals. However, it was during his extensive experience trekking through the rugged terrain of Val Camonica, Italy, researching Neolithic rock art, that he started forming his theories on extreme reality.
“Extreme Reality can transform our knowledge about reality technologies. Instead of building a virtual world, we must perceive reality, react to reality, and survive in a livable world,” says Cai.
Reality technologies have the capacity to transform safety equipment for rescue workers, and that has led NIST’s Public Safety Communications Research Division to award Cai’s team $1.91 million for two projects, including a current one, Extreme Reality.
In 2019, Cai’s team won the First Place Award from the NIST Haptic Interface Challenge for Public Safety. In collaboration with Mel Siegel, emeritus professor of robotics; Roberta Klatzky, professor of psychology; and UPMC Emergency Medicine Assistant Professor Lenny Weiss, assistant professor of emergency medicine a the University of Pittsburgh, the team developed a helmet that enables firefighters to communicate with haptic signals to navigate through dark, smoke-filled buildings. (The helmet communicates by sending a buzz to a firefighter’s head indicating which way to move.)
“Technically speaking, the haptic interface itself was not cutting-edge, but the interface design for human-machine teaming was carefully crafted with simplicity and thoroughly tested for use under extreme conditions, like when firefighters are wearing SCBA [self-contained breathing apparatus] masks,” says Cai.
The latest NIST project will allow researchers to further advance the helmet’s technology and “expand and ruggedize augmented reality interfaces that adapt to extreme environments, enhance communication, and provide a ‘sixth sense’ to first responders,” states Cai.
“We need to think about preparedness,” says Cai, and one way to do that is to use both virtual and real-world, real-time data to train first responders for dangerous work.
After his visits to the fallen Pittsburgh bridge, Cai’s lab started constructing 3D models of it. “Our goal was to build a digital twin of the disaster site for training first responders, recovery planning, and forensics. We found our rough model can provide situational awareness from different perspectives,” states Cai, adding, “the Fern Hollow Bridge should be a training case.”
Cai’s work at the bridge aligns with other projects he has underway. Just a week before the bridge collapse, he visited a local STAT MedEvac facility and 3D scanned the area around a medical helicopter launchpad. Again, the team plans to build a digital twin of the complex for navigation safety, emergency escape, and crisis-management training.
“Extreme Reality is about extreme interactions. We should teach this in engineering,” says Cai.