Looking in the Mirror


Aug 15, 2017, 12:33 PM by David McMillin

looking in the mirror photo

When physicians prescribe medications, those scripts typically come with warnings of potential side effects. But just because physicians are knowledgeable about those reactions doesn’t mean they truly understand the pains of the patient experience. Tyler Gates, principal at Brightline Interactive, recently helped healthcare professionals take their understanding of the challenges of treatment to a new level and walk in the shoes of patients who struggle with Tardive Dyskinesia. The condition, which is caused by certain anti-depressants and anti-psychotic medications, impacts the nervous system and causes involuntary facial movements. “Tardive Dyskinesia isn’t necessarily life-threatening, so it doesn’t receive as much attention in the medical field,” Gates said in a recent interview. “Our client wanted to raise awareness that some people really struggle with these side effects.”

The client — who preferred to remain anonymous due to regulatory and compliance issues — worked with Gates to simulate the real feeling of battling Tardive Dyskinesia for professionals attending a medical conference. The experience took a cue from a surprising source: the Disney film “Moana.” “I had seen Disney use depth cameras to deliver a heightened augmented reality experience that translates facial expressions on to animated characters,” Gates said. “Instead of just feeling like they’re hearing Dwayne ‘The Rock’ Johnson’s voice, the technology translates his iconic cheeks and dimples on to the cartoon character. So we wanted to apply a similar approach and place scientifically rendered graphic content of the involuntary facial movements associated with Tardive Dyskinesia on top of people’s real faces.”

A Mask of Magic

Gates said that feature movies with massive budgets and extended production schedules enjoy a rare luxury: time. “In this case, we did not have time,” Gates said. “It was all being done live. In order to make it feel real, the images needed to change quickly based on different features and skin colors.”

The Tardive Dyskinesia experience started with a three-dimensional blank mask, designed to contour to an individual’s face. Using videos of actors simulating the movements of the condition, a combination of depth cameras and facial detection cameras translated that footage on to the mask. As participants approached a mirror, they saw a live feed of their own faces. The top half included their real eyes while the bottom displayed movements. “You can see yourself blink, but you’re watching what looks like your mouth move, too,” Gates said. “It offered a real sense of what it might feel like to be in a conversation with a friend or a public situation while dealing with this type of extreme vulnerability. It gave the medical community more than that statistics or photos of patients. Now, when they think of the condition, they can recall how frustrating it felt to lose control over their own movements.”

Tardive Dyskinesia is one of many conditions that physicians must understand. As augmented reality continues to grow more sophisticated, medical attendees will most likely see many new images reflected in the mirror. “There are ways of using AR to create a deeper understanding of conditions that aren’t even outwardly visible,” Gates said. “It can show what’s happening under the skin and on other body parts besides the face. The possibilities of this type of technology are relatively limitless.”

Want to learn more? Watch Tyler Gates 2017 Education Conference, DEI: Electrify Your Events with Augmented and Virtual Reality.