ASU researchers experiment with breakthrough tech exploring human communication, perception
Digital manipulation of video images, whether for blockbuster films or malicious intent, also known as deepfakes, has become commonplace. But the ability to create realistic-looking and -sounding digital images typically requires a good deal of time — until now.
Researchers at an Arizona State University West Valley campus lab have been able to capture a person’s likeness, motion and sound in real time and instantly swap that person’s image with another, with no lag time.
Beyond entertainment value, this technology opens the door to exploring such questions as how individuals perceive cross-gender, cross-racial or even cross-species communication. And for neurodivergent individuals, it provides a research window into the mysteries of social difficulties and problems with subtle communication.
Ben Falandays, an assistant professor in the School of Social and Behavioral Sciences at ASU's New College of Interdisciplinary Arts and Sciences, is leading this novel approach to exploring human interaction and behavior. At the heart of his research is a cutting-edge motion capture (MoCap) virtual reality (VR) laboratory, an innovative space that redefines how human cognition and social interaction are analyzed and understood.
Falandays' academic journey, deeply rooted in psychology and philosophy, led him to explore cognition not as a purely computational system, as it has traditionally been viewed, but as an entire ecosystem of processes. He integrates theories of neuroscience, perception/action, development, social coordination, language, culture and evolution to figure out how it all fits together. His lab, the PALM (Perception-Action, Language, and Meaning) Lab, is a hub for interdisciplinary research that harnesses advanced technologies including VR and MoCap systems.
In his MoCap lab, Falandays combines commercially available hardware with custom code to create a unique resource with unprecedented research opportunities.
"Our VR MoCap system, which I believe is the first of its kind, allows multiple people to interact in a highly realistic virtual environment. We're using hyper-realistic avatars, motion capture for real-time gesture replication, eye and facial trackers, speech data and various sensors to collect a rich dataset of interactions," Fandalays said.
"What's exciting is that it allows us to experimentally manipulate the 'parameters' of social interaction in real time — making people appear to behave differently than they do — so we can better understand which signals are crucial for effective coordination and communication. This technology opens up new possibilities for research across many fields, from education to team dynamics."
The advantages of this technology extend across academic disciplines. For instance, in education, the MoCap lab can help researchers understand how to enhance coordination between students and teachers. In the study of developmental disorders like autism, it offers new ways to study communication. Moreover, the lab's ability to collect multiple streams of data simultaneously — ranging from full-body motion to facial expressions and speech — allows for a more holistic understanding of human interaction, something that was previously difficult to achieve in controlled environments.
ASU students benefit immensely from this high-tech environment. The lab not only equips them with technical skills like coding, experience with research methodologies and statistics, but also provides hands-on experience with this custom-built research technology. These skills are invaluable for future careers in fields as diverse as business, data science and any discipline involving social coordination or technology.
Looking ahead, Falandays sees the MoCap system as a focal point for the lab's future research. He is enthusiastic about the potential applications of this technology, from improving virtual communication and team building to aiding in education and supporting individuals with communication disorders.
Simply put, the work being done in the PALM Lab is poised to make significant contributions to our understanding of human cognition and interaction, marking no less than a new frontier in cognitive science research.
More Science and technology
ASU, Mexico partner to build next generation of chipmakers, drive semiconductor innovation
Thousands of college students in Mexico will soon have the opportunity to enroll in Arizona State University’s new, free online course to learn the fundamentals of microelectronics and…
ASU, St. Mary’s Food Bank partner to tackle food insecurity in Arizona
Arizona State University and St. Mary’s Food Bank (SMFB) have joined forces to create an interactive data dashboard that tracks and maps food insecurity rates across Arizona. This innovative tool…
Study reveals genetic insight into desert survival
The deserts of the American Southwest are home to the Mojave and Sonoran desert tortoises, two seemingly similar yet genetically distinct species of turtles. These tortoises, adapted to different…