Sign (language) of the times

Body-language interpretation research connects ASU computer science student with ASL community

A closeup of hands holding a smartphone

Have you ever received a text message in which the sender's tone was unclear? That’s because crucial pieces of information are missing to fully understand the message.

Gestures and body language are important parts of speech that can’t be conveyed in the text- and speech-based input dominating today’s human-computer interfaces.

People who rely on text-based communication systems, such as those with hearing impairments, but who also have fully expressive communication systems like American Sign Language (ASL) could benefit from machine interfaces that can understand gestures as well as spoken or written words.

To address this need, computer science doctoral student Prajwal Paudyal is working on SCEPTRE in the iMPACT Lab. SCEPTRE is a smartphone interface that uses two wireless armband sensors to take in ASL sign gestures to communicate via computer systems.

“The ASL user trains the system by performing three iterations of a gesture, which is captured by a smartphone,” Paudyal said. “The data is then sent to the server to train a computer model. Later, when the user needs to use the system, she puts on the wireless armband sensors and performs the gesture, which is recognized in the smartphone, in a fog server or from the server, depending on communication channels.”

Finding a computing need and solving it

Paudyal became interested in working with technology to assist ASL users when he started his research for human-computer interaction using mobile devices to redefine the way people interact with machines.

“I quickly realized that having a syntactic language to interface with machines is very important. I started looking into sign languages because they are completely defined languages with full expressibility and flexibility. I started taking ASL classes myself and had the opportunity to interact with people who are deaf as well as with numerous users of ASL.”

He decided to continue on with this work for his doctoral research project because of its potential to help people.

“The immediate impact of this type of research on the real world and the potential increase of quality of life it promises to bring brought me a sense of fulfillment, and thus I decided to stick with this line of research for my PhD dissertation,” Paudyal said.

Recognition for research excellence

For his research efforts so far, Paudyal received the Spring 2016 Graduate and Professional Student Association (GPSA) Outstanding Research Award, which recognizes ASU graduate and professional students who exemplify excellence in research and development. Students are judged on the interdisciplinary character of their projects, the positive impacts on the community and the outstanding contribution of the research, among other factors.

Computer engineering graduate program chair and computer science professor Sandeep Gupta nominated Paudyal for the award. The iMPACT Lab faculty adviser thinks Paudyal’s research is very promising and pioneers pervasive personalized and extensible assistive technology for sign language users.

“He earned the GPSA award because his research is an interdisciplinary project that combines computer science (the Internet of Things and human-computer interaction), linguistics and speech and hearing science,” Gupta said. “It reflects that he is doing impactful research on a complex problem that involves knowledge from various fields. This kind of use-inspired research cannot only solve immediate problems but also spur fundamental research questions.”

Paudyal is thankful for the recognition he has received and said it inspires further commitment to his work.

“Receiving the Outstanding Research Award has helped me solidify my commitment to research and to the field,” Paudyal said. “The acknowledgment and support I get from peers, other researchers and the graduate community in general at ASU helps me stay focused and inspires me to do more.”

A love of discovery makes a good researcher

Paudyal said he owes his skill as a researcher to his desire for discovery and sense of curiosity.

“I like being on the cutting edge and discovering things for the first time,” Paudyal said. “In my ‘other life’ outside of the research lab, I am a photographer and an adventurer, and I like discovering new places. I think this curiosity and sense of adventure helps me a lot as a researcher because research to me is the ultimate intellectual adventure.”

He also enjoys documenting his life by writing blogs and marking trails at the places he explores to guide his future excursions and to learn as much as he can. His meticulous approach also helps him in documenting his research.

“I have a similar attitude about documenting a lot and staying vigilant when I conduct research,” Paudyal said. “This has helped me grow as a researcher because I can constantly reevaluate my approaches and improve my process. I think this vigilance more than anything else is what makes me a good researcher and has put me on an upward spiral of improvement.”

Continuing to help the world through accessibility research

Paudyal has taken his research beyond the lab to apply his work in ways that can help those who need more accessible communication options.

He and his colleagues submitted a grant to the National Institutes of Health that aims to do a usability study on emergency health-care scenarios for people with hearing and speaking difficulties.

He also is actively involved in ASL and deaf clubs around the Phoenix metro area.

Next, Paudyal plans to extend his research in two key areas to improve its practical and theoretical applications.

“First I want to incorporate fingerspelling, which is what ASL users use if there is no word equivalent in the ASL dictionary," Paudyal said. "I am also focusing on the theoretical background for sign language and gesture processing to see if I can make contributions in the theoretical realm as well as the practical realm.”

After he earns his doctorate he hopes to find a position that allows him to continue his research in accessibility, particularly music therapy and sign language processing.

More Science and technology


Older person seen from behind, seated in a wheelchair in a bedroom, looking out a window.

When skeletons hold secrets: ASU project helps identify hidden elder abuse

When the death of an older adult is considered suspicious, their body is sent to the Maricopa County Office of the Medical Examiner. The agency is required by law to investigate deaths…

Side-by-side images of bone scans.

ASU team creates 3D bone atlas to stop elder abuse

“Only our bodies know the truth. Bones don’t lie,” writes Tayari Jones in “An American Marriage.”No one understands the power of this statement quite like Katelyn Bolhofner. The assistant…

Emily Williamson carries the gonfalon for the School of Computing and Augmented Intelligence down an aisle in a crowded auditorium full of seated graduates

Computer science school looks forward on heels of record-breaking graduation season

This spring, at two packed convocation ceremonies, a crowd of newly minted engineers ebulliently cheered under a rain of fireworks, balloons and confetti as the School of Computing and Augmented…