Capturing physiological signals with an app and a tap

For someone suffering from an acute illness or a medical emergency, the nearest healthcare facility can be several hours away in many parts of the world. Even in the U.S., hospitals in rural communities have closed or are at risk of closing. Researchers at Carnegie Mellon University’s College of Engineering are working to bridge this gap in patient care with machine learning.

Billions of people have cellphones, in both developed and developing nations. More than half of these phones are smartphones. Conrad Tucker, a professor of mechanical engineering, hopes to use this resource to help improve the health of people all over the world.

Tucker’s research is focused on developing algorithms to be deployed on a phone app that can unobtrusively capture someone’s physiological signals and correlate them to disease states. First, the user takes a video of themselves using the smartphone’s camera. The algorithm then analyzes the pixels of the video and the corresponding audio in order to quantify a person’s physiological signals such as pulse and respiration rates. “Unlike our human eyes, with a camera, you can magnify the intensity of the image that you’re capturing,” Tucker says. “When your heart is beating, there are very slight changes in the pigmentation of your skin, and that’s what it’s trying to extract and correlate to physiological signals and disease states.” 

For the mobile app to be effective across diverse populations and scenarios, the research team will work to ensure that their algorithms account for variability in lighting conditions, variability in people, and variability in available training data.

Variability in lighting conditions: if the lighting in an environment is too dark, the algorithm may have difficulty detecting someone’s face, which is the first step in extracting physiological signals. Tucker and his colleagues have conducted preliminary investigations into brightening and image smoothing techniques and will continue this line of research for data formats that are more commonly used in smart phones. This lighting component is also relevant for applications in which technology “sees”—most prominently, self-driving cars.

Variability in people: people have diverse appearances, which can hinder the performance of machine learning algorithms such as face detectors. If someone has a hat, face tattoo, or nose piercing, the app might not recognize them as a face. “The challenge with algorithms is that they are very literal,” Tucker says. “And so, if they’re not trained on examples of these kinds of diversity, then it’s difficult for them to sometimes group these as all being people.”

Variability in training data: rather than traveling to multiple parts of the world to collect training data to be used by machine learning algorithms, the research group seeks to use generative neural network approaches to actually generate data. If their team can generate artificial faces for example, they can achieve a robust dataset and reduce the amount of required travel. “We hope to use machine learning to create faces that have the tattoos, the hats, the different heart rates, so as to increase the robustness of our algorithms,” Tucker says.

This project is funded by the Bill and Melinda Gates Foundation with a $1,000,000 grant over three years. Collaborating with Amir Barati Farimani, assistant professor of mechanical engineering, and Laszlo Jeni, systems scientist in the Robotics Institute, Tucker is tackling the central question of the project: Does this have the potential to transform people’s lives?

Hopefully, the answer to the question is yes. As the project works toward deployment, the researchers can see it benefiting a wide range of people, both in the US and abroad, particularly in communities far from a medical facility. If people can send their physiological signals to their healthcare facility before they even get there, the medical professionals could be ready to start treating them as soon as they arrive. Another benefit of the app is that “it’s non-contact,” Tucker says. “You’re able to capture these physiological signals from just purely observation similar to taking a selfie.”

With this project, Tucker hopes to give millions of people control of their health. So, regardless of their healthcare or how far away the nearest hospital is, someone can capture important health information with an app and a tap.