Sign (language) of the times
Body-language interpretation research connects ASU computer science student with ASL community
Have you ever received a text message in which the sender's tone was unclear? That’s because crucial pieces of information are missing to fully understand the message.
Gestures and body language are important parts of speech that can’t be conveyed in the text- and speech-based input dominating today’s human-computer interfaces.
People who rely on text-based communication systems, such as those with hearing impairments, but who also have fully expressive communication systems like American Sign Language (ASL) could benefit from machine interfaces that can understand gestures as well as spoken or written words.
To address this need, computer science doctoral student Prajwal Paudyal is working on SCEPTRE in the iMPACT Lab. SCEPTRE is a smartphone interface that uses two wireless armband sensors to take in ASL sign gestures to communicate via computer systems.
“The ASL user trains the system by performing three iterations of a gesture, which is captured by a smartphone,” Paudyal said. “The data is then sent to the server to train a computer model. Later, when the user needs to use the system, she puts on the wireless armband sensors and performs the gesture, which is recognized in the smartphone, in a fog server or from the server, depending on communication channels.”
Finding a computing need and solving it
Paudyal became interested in working with technology to assist ASL users when he started his research for human-computer interaction using mobile devices to redefine the way people interact with machines.
“I quickly realized that having a syntactic language to interface with machines is very important. I started looking into sign languages because they are completely defined languages with full expressibility and flexibility. I started taking ASL classes myself and had the opportunity to interact with people who are deaf as well as with numerous users of ASL.”
He decided to continue on with this work for his doctoral research project because of its potential to help people.
“The immediate impact of this type of research on the real world and the potential increase of quality of life it promises to bring brought me a sense of fulfillment, and thus I decided to stick with this line of research for my PhD dissertation,” Paudyal said.
Recognition for research excellence
For his research efforts so far, Paudyal received the Spring 2016 Graduate and Professional Student Association (GPSA) Outstanding Research Award, which recognizes ASU graduate and professional students who exemplify excellence in research and development. Students are judged on the interdisciplinary character of their projects, the positive impacts on the community and the outstanding contribution of the research, among other factors.
Computer engineering graduate program chair and computer science professor Sandeep Gupta nominated Paudyal for the award. The iMPACT Lab faculty adviser thinks Paudyal’s research is very promising and pioneers pervasive personalized and extensible assistive technology for sign language users.
“He earned the GPSA award because his research is an interdisciplinary project that combines computer science (the Internet of Things and human-computer interaction), linguistics and speech and hearing science,” Gupta said. “It reflects that he is doing impactful research on a complex problem that involves knowledge from various fields. This kind of use-inspired research cannot only solve immediate problems but also spur fundamental research questions.”
Paudyal is thankful for the recognition he has received and said it inspires further commitment to his work.
“Receiving the Outstanding Research Award has helped me solidify my commitment to research and to the field,” Paudyal said. “The acknowledgment and support I get from peers, other researchers and the graduate community in general at ASU helps me stay focused and inspires me to do more.”
A love of discovery makes a good researcher
Paudyal said he owes his skill as a researcher to his desire for discovery and sense of curiosity.
“I like being on the cutting edge and discovering things for the first time,” Paudyal said. “In my ‘other life’ outside of the research lab, I am a photographer and an adventurer, and I like discovering new places. I think this curiosity and sense of adventure helps me a lot as a researcher because research to me is the ultimate intellectual adventure.”
He also enjoys documenting his life by writing blogs and marking trails at the places he explores to guide his future excursions and to learn as much as he can. His meticulous approach also helps him in documenting his research.
“I have a similar attitude about documenting a lot and staying vigilant when I conduct research,” Paudyal said. “This has helped me grow as a researcher because I can constantly reevaluate my approaches and improve my process. I think this vigilance more than anything else is what makes me a good researcher and has put me on an upward spiral of improvement.”
Continuing to help the world through accessibility research
Paudyal has taken his research beyond the lab to apply his work in ways that can help those who need more accessible communication options.
He and his colleagues submitted a grant to the National Institutes of Health that aims to do a usability study on emergency health-care scenarios for people with hearing and speaking difficulties.
He also is actively involved in ASL and deaf clubs around the Phoenix metro area.
Next, Paudyal plans to extend his research in two key areas to improve its practical and theoretical applications.
“First I want to incorporate fingerspelling, which is what ASL users use if there is no word equivalent in the ASL dictionary," Paudyal said. "I am also focusing on the theoretical background for sign language and gesture processing to see if I can make contributions in the theoretical realm as well as the practical realm.”
After he earns his doctorate he hopes to find a position that allows him to continue his research in accessibility, particularly music therapy and sign language processing.
More Science and technology
The science behind chronic stress
Stress comes in many shapes and sizes. There’s the everyday stress of preparing for a final exam or being stuck in traffic. And the more significant stress of losing a friend, family member,…
ASU planetary scientist to be inducted into the National Academy of Sciences
The National Academy of Sciences is inducting School of Earth and Space Exploration Director Meenakshi Wadhwa into the 2023 class of new members for her pioneering work in planetary sciences and…
Unlocking the potential of AI for homeland security
“Can we do what we're doing now cheaper, more efficiently, more effectively?” Adam Cox, director in the Office of Strategy and Policy at the Department of Homeland Security Science and Technology…