The technologies of virtual and mixed reality are revolutionising aviation maintenance.
Simulation has been part of aviation since at least 1929 when Edwin Link built the first of the thousands of trainers to bear his name using parts from the floor of his father’s musical instrument factory. But until relatively recently, aviation simulation meant flight simulation. That is starting to change, and the dividend is likely to be faster, more thorough training for engineers, greater safety in the hangar or on the apron and less opportunity for maintenance error.
Simulation in maintenance can potentially play a greater role than in operations, where its main use is for training and validation. The same technologies of computer modelling, high-definition 3D imaging and interactivity have the potential to change not just how the tasks of maintenance are trained but how they are done.
Engineers can now train for complex procedures in the same way surgeons do – computer systems can guide them through the job and experts can watch their progress, not just over their shoulders, but literally through their eyes.
Levels of reality: AR, VR and MR
Edwin Link’s ‘blue box’ Link Trainer created a form of virtual reality with its piano and pipe-organ technology that reproduced instrument flight for more than 500,000 pilot trainees during World War II.
Modern information technology has allowed a much more immersive form of virtual reality, defined by NASA as the use of computer technology to create the effect of an interactive three-dimensional world in which the objects have a sense of spatial presence.
- Virtual reality (VR) can be created in a full-flight level D simulator, or with head position-sensing goggles, headphones and haptic gloves. It has 2 virtual cousins:
- Augmented reality (AR) – simultaneous experience of the physical world with an overlay of digital elements. (Smartphone programs that display information about an exhibit in a museum or on a walking tour are an example.)
- Mixed reality (MR) – simultaneous experience of the physical world with an overlay of digital elements, where physical and digital elements can interact. These interactions can be visual, audible or tactile.
Tool or toy? The evidence
Academic consensus is that virtual reality can be an effective teaching tool. A systematic literature review published in the Journal of Computers in Education in 2021 concluded, ‘[Virtual reality] conferred a learning benefit in around half of cognitive studies, especially where highly complex or conceptual problems required spatial understanding and visualisation.
‘Encouragingly, most procedural tasks did show a benefit to utilising [mixed reality] and, furthermore, there was evidence that virtual skill acquisition could be transferred successfully to real world problems and scenarios.’
The review found 3 of the 4 studies that attempted to utilise VR as a means of teaching procedural skills, showed a distinct advantage over less immersive methods. ‘One study on emergency fire response found that 70% of those who utilised virtual reality training were able to perform the correct procedure in the correct order,’ it said. ‘This was 50% higher than the control group who were exposed to a presentation and reading material only.
Academic consensus is that virtual reality can be an effective teaching tool.
‘The ability to repeatedly practise a procedure in a safe environment whilst expending little resources could be one of the most advantageous and intrinsic benefits of [mixed reality] technology.’
Another academic study found VR/AR training translates well into acquiring real-world skills.
‘Skills acquired by simulation-based training adequately transfer to operative settings with firm scientific evidence of transfer from training in a virtual environment to real-world tasks,’ Bettina Mrusek and Stephanie Douglas of Embry-Riddle Aeronautical University wrote in a 2020 journal article, citing several earlier studies.
Maintenance training
Aircraft maintenance is at least as much learned by doing as by reading or lecturing. The traditional way of teaching involved air force or airline schools maintaining non-flying obsolete aircraft which were dismantled, examined and reassembled innumerable times by generations of students. Composite maintenance training devices, which were full-scale mock-ups of real aircraft, were sometimes used.
Examinations took the form of students performing repairs and maintenance under the eye of an experienced instructor. This system was effective, but costly, and not without danger in the case of hydraulic or electrical systems.
The first digital/simulation training used desktop computer systems and was useful for learning procedures, rather than techniques. Virtual and augmented realities make it possible to simulate and reproduce situations that would be costly, complex, dangerous, or difficult to replicate in the physical world. De-icing (an operational procedure) and undercarriage testing are 2 examples.
Virtual reality creates a safe and inexpensive environment for skills to be honed while allowing visible consequences for mistakes and allows repeated practice until the process is familiar.
Academic consensus is that virtual reality can be an effective teaching tool.
Maintenance training devices offer many advantages including:
- aircraft exploration through virtual aircraft
- component removal
- troubleshooting exercises
- lesson plan (reload any saved situation instantly)
- animated schematics
- simulation of built-in test equipment.
A hierarchy of devices has emerged: simulators and virtual trainers bridge the gap between the classroom and the aircraft, and augmented/mixed reality systems enhance training, ensure consistency and allow real-time support.
Canadian simulation specialist CAE offers Simfinity, a virtual maintenance trainer that creates a ‘virtual aircraft’ where systems familiarisation, maintenance procedural training and troubleshooting can be taught. The trainer offers the ability to display cockpit panels and instruments and save and recall layouts. It also includes a library of malfunctions and active schematics interacting in real-time in a high-fidelity simulated environment. Australian Defence Force engineers maintain the Sikorsky UH-60 Blackhawk and SH-60 Seahawk using the Simfinity system as part of the ADF’s next-generation simulated aircraft maintenance trainer. This system trains maintenance technicians to diagnose and troubleshoot aircraft systems, avionics and flight control systems in real-time.
Viral and virtual
The viral circumstances of the 2020s have stimulated virtual reality training. When the grim reality of the COVID-19 pandemic prevented Boeing technicians travelling to Australia to service the Royal Australian Air Force’s (RAAF) C-17A Globemaster III aircraft, mixed-reality devices replaced them.
In July 2020 the RAAF began a trial at Amberley involving mixed reality and a high-definition secure video connection, allowing the American engineers to observe as if they were not merely looking over the shoulder, but through the eyes of their Australian counterparts. Iris tracking in Microsoft HoloLens goggles enabled technicians in the US to see exactly what the RAAF engineers were seeing inside the aircraft.
Boeing C-17A field services manager Glen Schneider said the system would allow technicians to connect with the Boeing field engineering team while they are away on a domestic or international mission.
The first project was to replace the C-17’s floatation equipment deployment systems panels, which consist of explosive components that deploy life rafts in an emergency.
Real benefits
Airbus says mixed-reality solutions have cut manufacturing time by a third for some components and systems while improving quality. Digital information, such as instructions or diagrams, can be overlaid on a real piece of machinery to aid in complex or hard-to-reach tasks.
Airbus designers now test their designs virtually to see if they are ready for manufacture. The group says mixed reality cuts the time taken for this step by 80%.
In late 2019 Qantas began a trial of engine ground run training for engineers.
Since 2020, Rolls-Royce has offered its engine familiarisation course for the AE2100 turboprop (as used on the Lockheed C-130 Hercules and Alenia C-27J Spartan) as a virtual reality course. Students can log in from anywhere in the world to the training centre in Indianapolis, US, for a comprehensive overview of the construction, design and operation of the engine and engine systems using virtual reality. Rolls-Royce says, ‘This creates an environment for students to ‘learn by doing’, increasing their recall by completing multiple repetitions.’
The company also has developed a VR training course for the BR725 engine, as used in the Boeing 717 and, potentially, the upgraded B-52.
In late 2019, Qantas began a trial of engine ground run training for engineers. The project used HoloLens2 goggles and a virtual aircraft model instead of an actual aircraft. Using the goggles, Qantas engineers were able to train and practice this essential but potentially dangerous operation without the need to use a simulator or aircraft.
The safety case
For Lithuanian maintenance repair and overhaul organisation FL Technics, a major advantage of virtual reality training is safety.
‘The first thing that comes to mind is how to show students what can happen in a real environment in risky places,’ Ramunas Paškevičius, FL Technics’ Head of IT and Innovations says. ‘For instance, working on the emergency doors or something like that, there is a risk that if things are done in the wrong sequence, an explosion of the escape slide can be triggered within the hangar.
Virtual reality creates a safe and inexpensive environment for skills to be honed.
‘At best, that will be a financial loss; at worst, it will injure the mechanic and other mechanics. We did not have such cases in our hangars so, while all of our staff know theoretically that there is a risk, they have not seen how that explosion and deployment of the escape slide would look if it happened in the real world. Virtual reality is able to show mechanics that event and for them even to feel it.’
Further information
- Hamilton, D., McKechnie, J., Edgerton, E. et al. Immersive virtual reality as a pedagogical tool in education: a systematic literature review of quantitative learning outcomes and experimental design. J. Comput. Educ. 8, 1–32 (2021)
- Mrusek, B., & Douglas, S. (2020). From Classroom to Industry: Human Factors in Aviation Maintenance Decision-Making. Collegiate Aviation Review International, 38(2), 2021
