Keeping memory on hand

As we age, our bodies deteriorate, making daily life more difficult. Along with physical challenges, the loss of cognitive function, such as memory loss, can also impede performance of basic tasks. From misplacing keys or a wallet to more serious memory issues like forgetting to take medication or leaving the stove on, memory loss is a prevalent issue among older adults.

Troy McDaniel, an assistant professor of engineering at The Polytechnic School, one of the seven schools in the Ira A. Fulton Schools of Engineering at Arizona State University, is using advancements in haptics, wearable technologies, machine learning and sensor fusion to alleviate memory issues in older adults through the use of intelligent, hand-centric, wrist-worn wearables.

Not many researchers have leveraged a hand-centric view of the world to understand activities of daily living and the associated challenges that affect older adults — making McDaniel’s research unique.

His unique hand-centric approach is being supported by a 2022 National Science Foundation Faculty Early Career Development Program (CAREER) Award.

“By 2050, more than 9 billion people will call Earth home, and of those 9 billion people, it’s estimated that 20% will be 65 or older,” McDaniel says. “Such changes will further exacerbate current health care worker shortages and we will be in desperate need of innovative assistive technology solutions.”

McDaniel is actively working toward contributions to gerontechnology — technology to benefit aging adults — to fill that gap and provide the support soon to be needed by a growing elderly population.

Using haptics to advance gerontechnology

So much of human activity revolves around the use of our hands; it’s how we interact with objects in our environment. Through the use of haptics, which is the study of how human senses can propel or be used in conjunction with technology, there are several opportunities to advance assistive solutions.

Much of McDaniel’s research aims to assist older adults, and this project is no exception. Taking into account humans’ heavy reliance on their hands, he began exploring “hand-centric” camera views to aid memory function in this population three years ago.

“I call it a hand-centric view because it’s a hand-centric view of the world,” McDaniel says. “It’s a camera worn on the wrist, pointed at the hand and fingers to capture visual data of our interactions with objects in the surrounding environment.”

In McDaniel’s “cognitive augmentation aid,” users load images of their belongings, like keys or a wallet, onto a device that can recognize and recall those images upon request, enabling users to locate misplaced belongings. This is just one of the technology’s many functionalities.

“Think of this device as a personal assistant,” McDaniel says. “Did the person take their medication? Did they pick the correct pill? Did they put it in their mouth? These hand movements were all captured by the camera and can offer important insight to the user so they can continue living independently and maintain their quality of life.”

Wearable assistive technology is also a significant development for older adults who move around frequently to support their health needs. Whether they are at home, in an assisted living facility or in a hospital, the technology is made to move with the user so it can be integrated into a daily routine.

Broader impacts

The application potential that can stem from this interactive system, which is made up of “sophisticated software and intelligent algorithms,” is extensive McDaniel says.

“From exercise adherence to compliance and safety, inventory tracking or manufacturing, the opportunities for this core technology are endless,” he says. “The video data collected from the hand-centric cameras specifically are also crucial for other researchers who want to explore the field of haptics, and its use extends beyond just seniors.”

Privacy issues and ethical concerns are at the forefront of the conversation. The technology is designed so data does not leave the device, ensuring data integrity. However, depending upon the application and with the user’s consent, it may be acceptable for video data to leave the device for analysis, for example, when used for rehabilitative therapies or inventory tracking.

In addition to broadening the application outlook, McDaniel has plans to design a new haptics course for the Fulton Schools to allow undergraduate students to explore this pathway. Experiential learning is important to McDaniel, so his course will offer a hands-on haptics kit for students to employ in various course projects. He also plans to develop an introductory haptics textbook for students and researchers new to the subject.

A multidisciplinary mindset

McDaniel says that none of this work would have been possible if it wasn’t for ASU’s interdisciplinary and collaborative research environment. During his undergraduate and doctoral studies in computer science at ASU, he says this was one of the qualities that set ASU apart from other universities.

For this project, he utilized his background in computer science and integrated neuroscience, psychology, human factors, costume design, ethics and more.

“I pulled from all of these areas to build the foundation for my research,” McDaniel says. “ASU presents an ecosystem where I can flourish and I firmly believe this is something I could do only here.”

Looking to the future

Along with refining this technology, McDaniel hopes to change the perception that there isn’t a space for wearable technology among older adults.

In his research endeavors, McDaniel discovered that older adults are open to technology provided it is user-friendly, doesn’t require daily updates, has an easy-to-read interface and maintains privacy and independence.

He has plans to put this technology into action with residents who are interested in participating at Mirabella, an intergenerational, lifelong learning community on the ASU Tempe campus allowing residents to test the new hardware.

With all of this in mind, McDaniel also hopes to improve the device’s accuracy and reliability and pursue licensing and commercialization to get the technology into the hands of people who most need it.

“This innovation will be a game-changer in the wearable technology industry,” he says.