Virtual reality has much to offer aviation, with applications in training and simulation
About half an hour into the flight you realise the weather is different to what was forecast. Looking over your left shoulder you see a dark cloud line beyond the wing. It’s not like in the simulator—these clouds feel close, enveloping, threatening. Less than a minute later, fat raindrops are hammering on the windscreen as static crackles in your headphones. You go on instruments in the gathering gloom but have to lean forward to read them because you don’t have your glasses on. You’re sweating now, focusing on the panel and not daring to look outside. A little sightseeing now would end it all.
And in the midst of everything your phone rings, like a messenger from another dimension. You flip up your goggles, let go of the handgrips and walk from your study to your bedroom to take the call. Welcome to virtual reality flight simulation.
Virtual reality (VR) involves a simulated computer environment that users can explore in three dimensions, usually using goggles and headphones.
VR has been the hot technology of computer gaming since the release of the Oculus Rift headset in March 2016. The Rift headset fulfilled the promise of VR, which had existed in cruder forms for more than 20 years, by integrating several of the following discrete advances:
- A gyroscope, accelerometer and magnetometer that allow 1000 position readings per second. Predictive software guesses where an Oculus Rift user will look next.
- Faster reacting pixels on AMOLED screens eliminated nausea-inducing ‘latency’ delays between head movement and image change.
- Creating an optically correct appearance with simple lenses and low-cost optics, using a deliberately distorted image, rather than using expensive lenses to convert a raw image into an optically correct image.
- Positional tracking that follows head movement as well as rotation.
Until now, VR’s only intersection with aviation has been in the form of computer games, but now its promise is emerging.
Tim Robinson, editor in chief of Aerospace, wrote about his experience with VR aviation simulation for the Royal Aeronautical Society. ‘Landings become easier thanks to the ability to judge height and keeping the runway in sight in a circuit is a doddle. Weather, which for 2D simmers is a factor but never feels intimidating, suddenly becomes much more threatening. Fly into fog or thunderstorms which envelop you all around and you begin to experience a sense of dread.’
In addition, researchers are finding signs that VR delivers particularly vivid and memorable lessons, in any subject. Sun Joo Ahn from the University of Georgia told Popular Science that traditional media, such as TV and movies, and virtual reality initially have a similar effect. But after a week VR pulls ahead, and ‘they forget their exposure to everything but the VR scenario,’ Ahn said. ‘What they experience in a sensory way sticks with them over time. The effects are persistent.’
In another paper Ahn and colleagues said, ‘our preliminary results suggest that even a brief framed experience in VEs can be powerful enough to persist and change actual behaviour in the physical world.’
Interestingly for aviation, Ahn had this to say on the subject of simulator training. ‘It is suggested that behavioural modification for low-probability, high-risk events is most effective when using intensive virtual experiences to deliver visceral simulations.’
Ahn says the immersion of virtual reality is what makes it effective. ‘Virtual simulations tend to mimic what you might experience in the physical world so there’s a lot more detail for you to create a mental schema, a memory of this experience as if it happened to you in the real world,’ she told US NBC News. ‘When you are trying to recall that experience later on, it’s a lot more salient. You have a lot more details to recall. It’s easier for you to build this mental map of what you experienced in your memory.’
Networked Society Institute Fellow (Virtual Reality) at the University of Melbourne, Ben Loveridge, says VR has unique advantages in training for stressful situations. ‘People who experience something in VR and then do it in real life often comment it feels like they’ve been there before,’ he says.
The advantage comes from the physical sensation of being in a virtual world. ‘If you’re embodied, and you feel you’re in the location, that gets encoded into your brain much more strongly than merely watching or reading. It’s a bit like learning how to play a musical instrument; a physical process.’
CASA simulation team leader, John Frearson, says the regulator is open-minded about virtual reality simulation but concedes such a fast-moving area will stay ahead of aviation regulations for the foreseeable future. ‘A bunch of high school students in India can build an app years in advance of what a regulator can approve,’ he says.
‘Any device that qualifies you to make entries in your logbook needs a level of fidelity way beyond that of even the most impressive game,’ he says.
‘We have to do a lot of tests relating to fidelity. The last thing we want is people training in a device and having negative training, or thinking they are well trained when they are not.’
The deadly risks in simulator training are untrained instructors, unapproved syllabuses and unfaithful devices. Combine them and simulation becomes an accident generator in its own deadly right.
‘Combine an untrained instructor and unapproved syllabus and you get American Airways flight 587 that crashed in Queens New York City, in 2001,’ Frearson says. American Airlines’ simulator training had emphasised vigorous use of the rudder in response to wake turbulence, which was harmless in the simulator, but the pilot flying in AA587 used so much rudder it tore the tail off the aircraft.
Frearson says VR has already arrived at the top end of flight simulation. A full motion level D simulator is in effect a very large set of VR goggles, he says, as is a military simulator where the aircraft is in the centre of a dome with the world projected on it.
The difference is cost. ‘At a conference last year I put the helmet on and the next thing I was in a Cessna 172 in the circuit. I could see my computer-generated hand in the virtual world when I reached for the throttle on the desk. The whole setup was about $25,000, including goggles, yoke and throttle.’
‘In the past, complexity of training and complexity of device have increased proportionally so that airline pilot training required a $10 million level D simulator. Virtual reality breaks that. It’s conceivable the complex training could be done on a device at a hundredth the cost of a flight simulator.’
CASA is currently signing off on involved and expensive simulator upgrades to give high-level simulators upset recovery training (UPRT) capability.
‘I could see how VR could be useful to produce a simple, generic simulator for representative of generic fidelity for UPRT training,’ Frearson says, musing on the possibility that eventually a virtual reality simulator could be used from anywhere in the world, so long as the pilot had a set of goggles and a set of mocked-up flight controls.
‘It’s the training that matters, not the method of delivery. Does CASA care if it’s helmet based or a level D? Aside from our current regulatory and harmonisation obligations the eventual answer will probably be something like: “Of course not.” If there one day was a pill that could train a pilot how to fly a 777 all we need to do as a regulator is develop or adopt standards to enable us to approve that pill.’
Grounded in reality: other aviation uses for VR
A common complaint among older pilots (and older people generally) is to lament the youth of today. Frearson prefers to deal in facts.
‘The year 2017 was the year when the first generation of cadet pilots who’ve only ever known the digital world, began their airline cadet training,’ he says. ‘They learn differently to the paper and pen generation. Their method of communicating is not chalk and talk; it’s some type of digital device—indeed an array of digital devices. Even gaming programs might be used for learning and practising “what if?” situations.’
The new generation of pilots will be more reachable with VR than with paper, he says.
Swinburne University Senior Lecturer in Aviation Training, Chrystal Zhang, makes a similar point. ‘We talk about bringing the context of the industry into the classroom,’ she says, arguing that virtual reality is the ideal way to do this.
‘VR could create a global community in a virtual classroom. I think virtual reality will have a huge impact on people’s perception of distance,’ she says.
Zhang sees advantages for VR in cabin crew, maintenance and ground handling training.
For cabin crew, she imagines it could be valuable to create virtual cabins to teach emergency procedures and the subtle art of reading passengers. ‘It would be of great value. Cabin crew not only look after passengers, they ensure their safety, yet their safety responsibilities tend to be overshadowed by their customer responsibilities. I can see how VR could change that,’ she says.
‘We could also use VR training to create fault scenarios for maintenance engineers,’ Zhang says, noting that augmented and shared reality technologies are already under development in this field. ‘VR could be used for a briefing on what to look for in a section of an aircraft.’
For accident investigation and recreation, VR could create the flight crew’s point of view.
Ground handling staff are not as intensively trained and qualified as elsewhere in aviation, but their working environment provides a near perfect space for VR simulation, Zhang says. ‘I think ground handling adapts well to practical training. You could produce a lifelike scenario without unnecessary academic overlay.’
VR training could even be extended to airline passengers. Roxana Cardos and colleagues reviewed 11 studies of virtual reality exposure therapy (VRET) used to treat fear of flying by placing people in a virtual airline cabin. Cardos’ review ‘supports the efficiency of VRET in flight anxiety and encourages the use of this type of exposure both in clinical practice and the research field.’
Luca Chittaro and colleagues compared how quickly people trained in VRdonned life jackets, compared with those who learned from studying safety cards and watching cabin crew demonstrations. They found knowledge transfer to the real world is significantly better in those who used mobile VR. Moreover, users perceived the tool as significantly more engaging, simple and effective.
Zhang says, ‘The power of virtual reality is how you see, feel and believe resonates with your emotions. It’s learning not just on an intellectual level but on an emotional level. Virtual reality gives you an immersive experience.’
Further information
Ahn, S. J., Bailenson, J. N., & Park, D. (2014). Short- and long-term effects of embodied experiences in immersive virtual environments on environmental locus of control and behavior. Computers in Human Behavior, 39, 235–245. doi:10.1016/j.chb.2014.07.025. Retrieved December 2017. See: https://pdfs.semanticscholar.org/d048/ba1e778ad6275f2f7414ec1b1d535c60edc5.pdf
Ahn, S. J., Fox, J., Dale, K. R., & Avant, J. A. (2014). Framing Virtual Experiences. Communication Research, 42(6), 839–863. doi:10.1177/0093650214534973. Retrieved December 2017. See: http://gavel.uga.edu/wp-content/uploads/2014/12/Communication-Research-in-print.pdf
Cardoş, R. A., David, O. A., & David, D. O. (2017). Virtual reality exposure therapy in flight anxiety: A quantitative meta-analysis. Computers in Human Behavior, 72, 371–380. doi:10.1016/j.chb.2017.03.007.
Chittaro, L., Corbett, C. L., McLean, G., & Zangrando, N. (2018). Safety knowledge transfer through mobile virtual reality: A study of aviation life preserver donning. Safety Science, 102, 159–168. doi:10.1016/j.ssci.2017.10.012.
Couch, C., NBC News. (2016, December 31). Can virtual reality make you a better person? Retrieved December 2017. See: https://www.nbcnews.com/mach/technology/can-virtual-reality-make-you-better-person-n700816
Virtual Reality Flight Simulation Guide. See: https://www.youtube.com/watch?v=roRm1I8CpbY
GA pilot experiences VR. See: https://www.youtube.com/watch?v=gf4HRChsljI
