Computer engineering doctoral student Alireza Bahremand, recent computer science graduate Paul Nathan and recent digital culture graduates Linda Nguyen and Marco Castillo also worked on the project.

GLEAM is short for Generates Light Estimation for Augmented Reality on Mobile Systems. The project involves an augmented reality application enabling the placing of virtual objects into physical spaces and illuminating those objects as they would be in those actual environments.

“This is about estimating the light in a space more accurately and robustly,” LiKamWa said. “People have been doing this for a long time in postproduction for computer-generated imagery in the film industry but we are bringing this capability to real-time mobile augmented reality systems and doing it in a way that improves the scene’s realism.”

With current systems of this kind, using even low-resolution virtual images requires a large amount of energy, he said, “so we have developed some tactics to actually reduce the energy use but still get better performance from the technology.”

Computer engineering doctoral student Jinhan Hu will present “Banner: An Image Sensor Reconfiguration Framework for Seamless Resolution-based Tradeoffs.”

Recent computer science graduate Saranya Rajagopalan and undergraduate electrical engineering student Alexander Shearer also worked on the project.

Banner involves technology that enables smartphones, headsets, tablet computers and similar devices to adapt camera settings to the needs of applications of computer vision and augmented reality to render images in appropriate resolutions for specific purposes.

Applications may need high-resolution image streams from the camera to capture distant or detail-intensive features of a scene, LiKamWa says, but they would be able to save energy and boost performance by capturing images at low-resolution settings when possible.

However, when applications try to change resolutions of image streams from a camera, mobile operating systems will drop frames, causing interruptions in the imaging.

With the Banner system, images at differing resolutions are captured and rendered seamlessly, with no loss in performance for app developers. At the same time, the system prolongs battery life by enabling low-resolution image streams.

“What we are doing is rearchitecting the parts of the operating systems, the device drivers and the media frameworks, so that we can actually change the resolution without dropping any frames, not even one frame,” LiKamWa said. “So you get a completely smooth resolution without any loss in the user experience.”

LiKamWa and his lab teams have come up with “two very impressive, elegant, technically demanding, well-implemented and well-evaluated solutions to very different problems,” said Rajesh Balan, an associate professor of information systems at Singapore Management University and a program committee co-chair for the MobiSys Conference.

The GLEAM project is a definite step forward in enabling “much more realistic scenes” using augmented reality imaging, while the Banner project “has high value for any application that processes a large number of photo or video images — for example, face recognition applications on mobile phones,” Balan says.

Balan’s fellow MobiSys 2019 program committee co-chair, Nicholas Lane, an associate professor of computer science at the University of Oxford in England, says LiKamWa is working at the forefront of research poised to produce “powerful mobile vision systems with capabilities that were until recently the domain of movies and science fiction. His work stands out because it rethinks how core aspects of these devices must function so they can better cope with the demands of high-fidelity processing and understanding of visual input.”

LiKamWa “has brought great energy to arts, media and engineering,” said the school’s director, Professor Xin Wei Sha. “He is inspiring students and colleagues alike and his METEOR Studio is blossoming with good students and innovative engineering research projects like GLEAM and Banner that are exploring fundamental experiential and technical aspects of mobile technologies and setting the stage for advances five to 10 years into the future.”

LiKamWa’s research, which has earned support from the National Science Foundation and the Samsung Mobile Processing and Innovation Lab, has led to provisional patent applications on the GLEAM and Banner technologies. His team will be releasing software frameworks for application developers to integrate these technologies into their solutions.

Joe Kullman

Science writer, Ira A. Fulton Schools of Engineering