Total recall

If not for an experienced engineer’s long and detailed memory, a serious in-flight incident might have remained a mystery

By Robert Wilson

It’s something not meant to happen on modern passenger aircraft – but it did to Jetstar flight JQ15.

Approaching Japan’s Kansai airport after a flight from Cairns on 29 March 2019, the Boeing 787’s left engine – and then its right – rolled back to below flight idle power.

The incident was not particularly dramatic, despite its serious implications. At about 16,000 feet on approach to Kansai International Airport, the engine-indicating and crew-alerting system (EICAS) display showed ENG THRUST R and ENG CONTROL R messages. About a minute later came an ENG FUEL SPLIT VALVE R message, indicating a problem in a fuel metering valve that was part of the autothrottle system.

Three minutes later, as the jet banked right over Komatsushima, a serious and statistically unlikely message appeared – ENG FAIL L – but disappeared after a few seconds. Altitude was now about 12,000 feet. The pilot-in-command could not feel any malfunction in the left engine but the aircraft’s continuous parameter logging (CPL) recorded an rpm value below idle lasting for 8 seconds in the left engine. A minute later, ENG FAIL R appeared on the EICAS.

The pilot disengaged the right autothrottle and set the right engine thrust lever to the idle position in accordance with the checklist for unstable parameters in the right engine. The CPL recorded the right engine below idle for 81 seconds. On both engines the rpm drop activated the auto relight feature. A further ENG CONTROL L message showed as the aircraft was on downwind leg over Osaka Bay. Thankfully, the aircraft landed uneventfully about 26 minutes after the first EICAS message. A go-around in those circumstances would have been a high-stakes gamble. As it was, none of the 313 people onboard was harmed.

The old-time engineers said they wouldn’t use Kathon because it had a salt problem.

Recorded data showed that engine rpm oscillation of both engines had occurred  during the first flight after a routine biocide treatment, during engine starting and cruise, and again on the incident flight. The oscillations had not been noticed by the flight crew and were not strong enough to prompt EICAS messages.

As the ICAO ‘State of Operation’ of the aircraft, Australia was entitled to take part in the investigation. Participation took the form of a daily international conference call including CASA and the Australian Transport Safety Bureau. Two CASA engineers specialising in large turbofan engines, Will David and Alan Silva, joined the daily discussion.

‘Everybody was trying to work out what had happened,’ Silva says. One of the first facts to emerge was that the 787 had been ferried from Melbourne to Auckland, New Zealand, 2 days before the incident, for biocide treatment in its fuel tanks. A distinctive characteristic of diesel and jet fuels – surprising considering their poisonousness to human beings – is that bacteria, fungi and yeast live in them. These microorganisms live where fuel and the water contained in it meet, and can pose a serious clogging and corrosion hazard to filters, pipes
and pumps.

You can’t have a safety system based on the oldest guy in the room having an uneasy feeling.

Aviation uses 2 types of biocide, Biobor and Kathon FP 1.5, which uses active ingredients (methylchloroisothiazolinone and methylisothiazolinone) widely used in paints, plastics cleaning products and cosmetics.

‘At the time I thought there might have been an overdose of the Kathon fuel additive,’ Silva says. ‘Boeing and [engine maker] GE have warnings in maintenance manuals about overdosing with Kathon.’ However, checking of the machine used to dose the 787’s tanks in New Zealand found it to be properly calibrated.

Japanese investigators used cotton swabs to take samples from the aircraft’s fuel tanks. But no microbial or biocide residue could be found. Briefly, it seemed the Kathon path was a dead end.

Silva felt a nagging dissonance during the fuel systems discussions in the daily meetings. He had joined CASA a year earlier as the latest move in an engineering career, which had begun as a Qantas apprentice in the early 1980s and taken him to senior engineering positions with Qantas, Ansett, Singapore Airlines and Boeing.

At first Silva couldn’t say for sure why he felt something had been missed. Then he recalled what his supervisors had told him when he was a trainee doing the sometimes arduous job of fuel system treatment. The memory was the source of his unease.

‘The old-time engineers said they wouldn’t use Kathon because it had a salt problem,’ he says.

‘At the time I was starting, Qantas was just about to get GE engines for the [Boeing] 767. At the time it was operating Pratt and Whitney and a few Rolls-Royce engines.’

He remembered conversations about fuel treatments and what additives were to be used. ‘They said GE engines were less tolerant of fuel contamination, compared to Pratt and Whitney and Rolls-Royce. This wasn’t just hangar talk – it was backed up with documentation. At the time I was doing fuel treatments on 747s coming out of heavy maintenance so it was interesting and relevant to me.’

Soon after, an analysis of the Jetstar 787’s fuel system by control systems manufacturer Woodward confirmed contamination in the aircraft’s fuel control unit. The unit uses spool valves which slide back and forth on a shaft. Contamination on these was able to be wiped off with a cloth but, due to the very fine tolerances of the unit, this contamination was enough to jam the unit beyond the capability of its servo motors.

‘Earlier engines had hydromechanical units, using cable control. Contamination issues back then would have been solved by shoving the throttle forward in its quadrant, against quite a strong resistance,’ Silva says. ‘These sorts of contamination incidents probably manifested as autothrottle failures.’

Woodward had also analysed the gel material. ‘They told us, “we think it’s a magnesium compound”,’ Silva says. ‘But when I looked up the material data sheet for Kathon, there was no mention of magnesium – it was actually very vague. I thought, “there’s got to be some more information somewhere”.’

From his industry experience, Silva knew the International Air Transport Association (IATA) published a range of detailed manuals and he ordered the latest edition of its fuel guidance document. ‘Sure enough, it listed all the components of Kathon FP 1.5, including all the magnesium salts.’

After that, things happened quickly:

• Jetstar suspended biocide treatment using Kathon FP 1.5 inside Boeing 787 fuel tanks of the same type of aircraft. It began a review of training for infrequent or unusual maintenance procedures.
• GE issued Service bulletin 73-0086 for aircraft equipped with GEnx engines, notifying operators to suspend biocide treatment using Kathon FP 1.5.
• Boeing deleted biocide treatment procedures using Kathon FP 1.5 from the aircraft maintenance manual for 787 aircraft using the GEnx engine. It also updated the aircraft maintenance manuals for all other Boeing models to more accurately describe the maximum allowable biocide concentration ratio, and record the calculation of biocide amount and the amount used.
• In March 2020 the US Federal Aviation Administration issued Special Airworthiness Information Bulletin NE-20-04 and the European Union Aviation Safety Agency issued Safety Information Bulletin 2020-06 to notify operators, repair stations and aircraft and engine manufacturers about suspending the use of Kathon FP 1.5.
• CASA issued Airworthiness Bulletin 28-018 that strongly recommended suspending use of Kathon FP 1.5 for biocide treatment.

Who knows? When expertise evaporates

Silva is modest about his contribution. ‘I’m sure the investigation would have made the connection before much longer,’ he says. He sees the larger issue as organisational knowledge, and how to preserve and benefit from it.

Boeing is among the aviation organisations that take retention of corporate knowledge seriously. The company harnesses and rewards knowledge and experience with several programs. About 2000 of the company’s 50,000 employees are recognised as Boeing Technical Fellows, described by the company as ‘a network of talent with deep expertise in a wide range of fields’. Fellows are deployed to solve technical, engineering and scientific challenges across the company. A further 191 staff are recognised as Boeing Technical Principals. The company also has a network of 3000 Boeing Designated Experts, with expertise in about 9000 distinct fields.

Airbus has had a formal knowledge management program since 1996. The company’s knowledge management systems include Reuse, Improve and Share Experience, a database of ‘lessons learned,’ and a company yellow pages directory of all Airbus employees throughout Europe, listing their experience and expertise.

An Airbus Expertise Transfer program, ExTra, obtains and codifies the accumulated knowledge of departing staff before they leave the company. Its actions include listing lessons learnt, contributing to a book of knowledge, conducting training modules and conferences, and managing internet forums.

Silva says knowledge and succession management are essential and unappreciated aspects of maintenance safety. ‘If this incident has a lasting wider lesson, it would be that you can’t have a safety system based on the oldest guy in the room having an uneasy feeling. There has to be a better way.’