Engineering reality

Margo Marchbank examines how virtual reality technology can help in the workshop

A giant turbofan engine hangs suspended in space, seemingly defying the laws of gravity. A young, goggle-wearing operator flicks her finger and the engine rotates to reveal an exploded view of hidden blades. Thanks to the goggles, her hands are free to work on a real engine using the turbofan hologram to guide her maintenance.

There’s a lot of glib talk currently singing the praises of virtual and augmented reality (VR and AR) and the many benefits of this technology, especially for technical training.

Amid the hype, is there some reality to the claims? Professor Lori Brown thinks so. She is chair of ICAO’s Next Generation of Aviation Professionals Outreach program. ‘For the first time, we have the ability to take the analog world and superimpose digital artifacts, creating mixed reality aviation simulations,’ she says.

This ‘mixed reality’, which seems to be the preferred term encompassing both AR and VR technology, is evolving rapidly. The hardware has come a long way since the first Google Glass release in 2013, with the current frontrunner appearing to be Microsoft’s second-generation HoloLens goggles, released in November 2019. And a growing group of developers, seeing the goggles’ potential, is designing software programs to capitalise on HoloLens 2 and other AR hardware.

Anneke Nabben, a training specialist with the Royal Netherlands Aerospace Centre, is working with Dutch airline KLM to develop training programs for maintenance engineers using HoloLens. Their collaboration has provided evidence of the value of mixed reality for such training.

In a proof-of-concept AR experiment, they allocated six experienced aircraft technicians aged between 40 to 60 and six base trainees aged from 16 to 22, to two groups. One group comprising three experienced technicians and three trainees completed a training course incorporating augmented reality, while the second (control) group completed a traditional training course.

Each three-hour course focused on an aircraft air conditioning system and had the same learning objectives. The traditional course comprised 180 PowerPoint slides and was based on the standard ‘talk and test’ model, while a second specially designed course integrated the AR technology as part of the training activities. This enabled trainees to collaborate through linked goggles, to see and understand how the system operated normally—the flow of fluids and air, the temperatures—to manipulate the height of the aircraft and see its effect on the air conditioning system and, finally, to see and analyse a system fault.

Nabben says changing the fundamental design of the training is important to fully realise AR’s training potential. Overwhelmingly, the AR training group valued the experience and felt confident about tackling the task in the real world (83 per cent), which none of the traditional training group did. Nabben and KLM’s findings with AR training mirrors the experience of others.

Frank van Liempt is an advisor to Spiral Technology, a company which is developing software for mixed-reality engineering applications. He is also a highly experienced aerospace engineer and was formerly head of engineering for Strata in the United Arab Emirates, manufacturing airliner parts for Airbus and Boeing.

Spiral Technology is developing immersive avionics maintenance training for US air force engineers. Their AR solution tethers existing training materials to an F16 cockpit. Students practise first in an interactive virtual cockpit. But the AR overlay works just as well in an actual cockpit, which can be used for final assessments. AR enables meaningful on-the-job-training, both on virtual and physical assets—he says that is ‘a game changer’.

The power of immersive training over conventional training is well known—a number of US studies show improved retention of information through AR learning over ‘talk and test’, reinforcing the old saying about experiential learning: ‘I hear and I forget. I see and I remember. I do and I understand’.

Bruce Hall, Pratt and Whitney’s general manager, customer training, says not only is there improved retention through the use of mixed reality learning—up to 90 per cent—but also AR brings up to 40 per cent return on investment through faster learning times and less down time for actual engines.

Through AR, the 3D virtual aircraft can be used for both theoretical and practical training. The Netherlands Aerospace Centre says in principle, it can model all complex mechanical systems in the aircraft, including systems which normally cannot be demonstrated in an aircraft on the ground, such as the landing gear. Difficult-to-access or dimensionally awkward parts of the aircraft can be replicated, so the invisible is made visible.

The ability to simulate normal and abnormal aircraft behaviour, coupled with the virtual aircraft being ‘available at any place and any time, without wearing out’, has enormous benefits for training efficiency and effectiveness, as training can take place in a more controlled environment, free of the pressure created by aircraft availability.

Using AR or mixed reality in training also satisfies younger students’ expectations of training, says Denis Manson, a former LAME now instructional designer with Aviation Australia. He contributed a chapter to a 2020 book addressing the issue of pilot and engineer shortages, Engaging the Next Generation of Aviation Professionals. ‘We can’t have PowerPoints for ever—it’s an evolution of technology which has to happen,’ he says.

We can’t have PowerPoints for ever—it’s an evolution of technology which has to happen.

He points to the coming generation of potential engineers who have grown up with immersive technologies and gamification, and with the corresponding expectation that their training will incorporate these. He is frustrated by the pace of change in aviation maintenance training in Australia which he says is impeded by two factors—the inherent conservatism of aviation engineering and, importantly, the lack of a critical mass compared, for example, to the United States.

The average age of the 301,516 certified aviation engineers in the US is 52 years, with 33 per cent of them over 60. Australia’s figures are similar. Therefore, the age of aviation engineers worldwide, and the fact they are retiring faster than they can be replaced, create further opportunities for using AR or mixed reality in engineering training.

Using AR to support repairs or maintenance in some remote areas is happening already. A senior engineer in the maintenance facility headquarters guides a less-experienced engineer at a remote location through the procedure, by projecting instructions on what needs to be done onto the actual aircraft.

Beyond training, van Liempt’s focus is on operational aircraft maintenance and manufacturing software. ‘AR is great for contextual overlay of information on the physical world—for display,’ he says. ‘Where the true power of AR will come is in the “capture” of information.

Think about using the AR headset for confirming an operation, creating quality records and error checking using its cameras. Think about guided inspection, where you record inspection results in AR straight onto a digital twin of the inspected object. Now your inspection result—and all previous results—are available for trend analysis to drive process improvement.’

While augmented reality can bring many benefits to aircraft maintenance training, there are limitations. Although AR goggles are improving all the time, Nabbens says it is still tiring to wear them for more than about 30 minutes and they still require set up and calibration. If you want to project onto a real aircraft, you have to take the environment into account—too much, or the wrong type of light will affect the quality of the hologram.

Then there is the cultural change required—instructors need to be fully on board or the technology may not be used to its full extent. Above all, the technology is only as good as the applications and the software available to run on it.

Augmented reality vs virtual reality vs mixed reality

The terms augmented reality (AR) and virtual reality (VR) are often used interchangeably, but there are important differences. According to Intel, virtual reality is fully immersive—it tricks your senses into thinking you’re in a cockpit or cabin, for example, apart from the real world. Wearing a head-mounted display or headset, the trainee experiences a computer-generated world of imagery and sounds in which they can manipulate objects and move around. Augmented reality, on the other hand, can overlay digital information on real-world elements, such as an aircraft engine.

Augmented reality keeps the real world central but enhances it with other digital details, layering new strata of perception and supplementing or augmenting, your reality or environment. It also offers the opportunity for collaboration, for working and learning together.

With evolving technology, a new term has been coined—mixed reality. It brings together real world and digital elements. In mixed reality, trainees can interact with and manipulate both physical and virtual items and environments, using next-generation sensing and imaging technologies. With mixed reality, they can see and interact with the world around them even as they interact with a virtual environment using their own hands—all without ever removing their headset.

While the fully immersive VR is ideal for procedural and familiarisation training—pilots can sit in a cockpit and try every switch—AR and mixed reality are better suited to supporting aviation maintenance and training. Engineers can move around an engine safely and have their hands free to work on a real aircraft. Nabben and van Liempt share reservations about VR. Nabben says, ‘With VR, you are completely immersed in an environment and you don’t know what’s going on around you’. van Liempt is not a big fan of VR. ‘You lose your bearings in the headset, you’re basically blind—I never saw it as practical.’