ASU researchers experiment with breakthrough tech exploring human communication, perception
Assistant Professor Ben Falandays and his students in the PALM Lab at ASU's West Valley campus explore human cognition and social interaction through motion capture and virtual reality technology. Photo courtesy of Danielle Castillo
Digital manipulation of video images, whether for blockbuster films or malicious intent, also known as deepfakes, has become commonplace. But the ability to create realistic-looking and -sounding digital images typically requires a good deal of time — until now.
Researchers at an Arizona State University West Valley campus lab have been able to capture a person’s likeness, motion and sound in real time and instantly swap that person’s image with another, with no lag time.
Beyond entertainment value, this technology opens the door to exploring such questions as how individuals perceive cross-gender, cross-racial or even cross-species communication. And for neurodivergent individuals, it provides a research window into the mysteries of social difficulties and problems with subtle communication.
Ben Falandays, an assistant professor in the School of Social and Behavioral Sciences at ASU's New College of Interdisciplinary Arts and Sciences, is leading this novel approach to exploring human interaction and behavior. At the heart of his research is a cutting-edge motion capture (MoCap) virtual reality (VR) laboratory, an innovative space that redefines how human cognition and social interaction are analyzed and understood.
Falandays' academic journey, deeply rooted in psychology and philosophy, led him to explore cognition not as a purely computational system, as it has traditionally been viewed, but as an entire ecosystem of processes. He integrates theories of neuroscience, perception/action, development, social coordination, language, culture and evolution to figure out how it all fits together. His lab, the PALM (Perception-Action, Language, and Meaning) Lab, is a hub for interdisciplinary research that harnesses advanced technologies including VR and MoCap systems.
-
Assistant Professor Ben Falandays explains to Roka Das, a current neuroscience undergrad, how the VR MoCap system will immerse her in a highly realistic virtual environment.
Photo courtesy of Danielle Castillo
-
Jacob Igo, a current computer science undergrad, excitedly observes real-time gesture replication in action at the PALM Lab.
Photo courtesy of Danielle Castillo
-
Hyper-realistic avatars replicating real-time gestures using eye and facial tracking technology.
Photo courtesy of Danielle Castillo
-
Student Rika Das and Assistant Professor Ben Falandays analyze key indicators of human cognition and social interaction.
Photo courtesy of Danielle Castillo
In his MoCap lab, Falandays combines commercially available hardware with custom code to create a unique resource with unprecedented research opportunities.
"Our VR MoCap system, which I believe is the first of its kind, allows multiple people to interact in a highly realistic virtual environment. We're using hyper-realistic avatars, motion capture for real-time gesture replication, eye and facial trackers, speech data and various sensors to collect a rich dataset of interactions," Fandalays said.
"What's exciting is that it allows us to experimentally manipulate the 'parameters' of social interaction in real time — making people appear to behave differently than they do — so we can better understand which signals are crucial for effective coordination and communication. This technology opens up new possibilities for research across many fields, from education to team dynamics."
The advantages of this technology extend across academic disciplines. For instance, in education, the MoCap lab can help researchers understand how to enhance coordination between students and teachers. In the study of developmental disorders like autism, it offers new ways to study communication. Moreover, the lab's ability to collect multiple streams of data simultaneously — ranging from full-body motion to facial expressions and speech — allows for a more holistic understanding of human interaction, something that was previously difficult to achieve in controlled environments.
ASU students benefit immensely from this high-tech environment. The lab not only equips them with technical skills like coding, experience with research methodologies and statistics, but also provides hands-on experience with this custom-built research technology. These skills are invaluable for future careers in fields as diverse as business, data science and any discipline involving social coordination or technology.
Looking ahead, Falandays sees the MoCap system as a focal point for the lab's future research. He is enthusiastic about the potential applications of this technology, from improving virtual communication and team building to aiding in education and supporting individuals with communication disorders.
Simply put, the work being done in the PALM Lab is poised to make significant contributions to our understanding of human cognition and interaction, marking no less than a new frontier in cognitive science research.
More Science and technology
From food crops to cancer clinics: Lessons in extermination resistance
Just as crop-devouring insects evolve to resist pesticides, cancer cells can increase their lethality by developing resistance to treatment. In fact, most deaths from cancer are caused by the…
ASU professor wins NIH Director’s New Innovator Award for research linking gene function to brain structure
Life experiences alter us in many ways, including how we act and our mental and physical health. What we go through can even change how our genes work, how the instructions coded into our DNA are…
ASU postdoctoral researcher leads initiative to support graduate student mental health
Olivia Davis had firsthand experience with anxiety and OCD before she entered grad school. Then, during the pandemic and as a result of the growing pressures of the graduate school environment, she…