Flying high tech: A new way to train fighter pilots

A graduate student at Arizona State University has teamed up with airmen from Luke Air Force Base to develop a 3D augmented reality system that could revolutionize the way U.S. Air Force pilots train for combat. The team has emerged as a top five finalist among more than 300 entries in an annual Air Force innovation challenge known as Spark Tank.

In the final round of the competition, the team will pitch their idea, called “Next-Gen Debrief,” to top U.S. Air Force leadership and a panel of industry experts at the virtual Air Force Association Aerospace Warfare Symposium on Feb. 26. The winner will be chosen by popular vote from thousands of virtual symposium attendees.

While the Air Force has invested heavily in technically advanced aircraft, tools used to analyze and learn from missions hasn’t kept up with the same pace of innovation. During postflight instructional sessions called “debriefs,” aircrews currently discuss flights using whiteboards, computer screens, laminated maps and model airplanes hot-glued to sticks. Among the pitfalls of the current debrief method, “there's opportunity to make mistakes or it's just harder to learn and complete,” said Maj. Kevin Hawkins, 56th OSS wing intelligence chief. 

Next-Gen Debrief uses data captured by sensors on F-16 Fighting Falcon and F-35A Lightning II aircraft during fighter pilot training at Luke Air Force Base. The sensors track flight characteristics such as position, airspeed and altitude. Next-Gen Debrief transforms this data into 3D interactive animations that provide an instant replay of missions during postflight training.

The system could revolutionize postflight debriefs at Luke Air Force Base, located west of Phoenix, which trains more fighter pilots for the U.S. and allied operations than any other air base in the world. Its 56th Fighter Wing graduates more than 400 pilots and 300 air control professionals each year. 

A Luke Air Force Base flight instructor conducts a postflight debriefing session with student pilots.

“The debrief process is fundamental to how we train to fly,” Hawkins said. He is one of three airmen on the Luke Air Force Base team, including Lt. Col. Alexander Turner, 56th Operations Support Squadron director of operations; and 1st Lt. Adam Treece, 56th OSS intelligence readiness chief. Through the National Security Innovation Network (NSIN) X-Force program, the airmen teamed up with Dylan Kerr, a student earning a master’s degree in digital culture (art) in ASU’s School of Arts, Media and Engineering. Kerr is also part of ASU’s Luminosity Lab, a student-driven, interdisciplinary research and development lab.

Here’s how Next-Gen Debrief works. Using Unity 3D’s real-time game development platform, Kerr developed an application that uses aircraft Air Combat Maneuvering Instrumentation data to create immersive and interactive flight animations — a radically new perspective for mission participants. He also built the application for the Microsoft HoloLens2 mixed-reality headsets to project the three-dimensional scene into the debriefing room, allowing participants to view scenarios from any angle during instructors’ postflight meetings with student pilots. The scene can be shared with multiple participants in the same room or with pilots on different Air Force bases to view and discuss missions together.

“Right now, if I want to debrief with someone who flew a mission from a different base we'd have to do that over the telephone,” Treece said. “But imagine if you put these headsets on and could debrief virtually, and both of you could see the same thing at the same time. That would make it a much more efficient process.”

“The project definitely changes my perspective on what’s possible with augmented or virtual reality,” Kerr said. “A lot of times we look at these tools as just something for entertainment. But what’s so special about Next-Gen Debrief is how much more effectively airmen can visualize their flight trainings.” 

ASU graduate student Dylan Kerr demonstrates the Next-Gen Debrief augmented reality system.

If Next-Gen Debrief emerges as a Spark Tank winner, the team will receive project management support and other resources to implement their idea at Luke Air Force Base, which Hawkins predicts could be accomplished within six months. The goal is to then grow the capability throughout Air Force installations across the country.

Last year, a team of Luke Air Force Base airmen and ASU engineering researchers tied for first place in the Spark Tank challenge. Their winning device, the Dynamic Unmanned Threat Emitter (DUMTE), is now used to train pilots to identify and respond to threat environments like surface-to-air missiles and anti-aircraft artillery. A threat emitter is a training system that has the ability to realistically mimic threatening emissions from an array of potential enemy air systems.

As a result of NSIN, a program office under the Office of the Secretary of Defense, Research and Engineering, ASU’s partnership with the U.S. Department of Defense continues to grow to rapidly innovate new solutions to problems that require minds from many disciplines. In addition to competing in the Air Force Spark Tank competition, ASU received direct funding from NSIN to further develop the software system.