DBEA55AED16C0C92252A6554BC1553B2 Clicky DBEA55AED16C0C92252A6554BC1553B2 Clicky
March 29, 2024
Care to share?

It seemed unthinkable a few months ago that the Boeing Dreamliner, the technologically most advanced airplane in the world and a display of American superiority in technology that had finally entered service after a three and a half year delay, would be grounded.  At this point in time, we don’t know whether this will be a short or long process, but given what we know and what we are hearing, don’t believe a quick solution will be forthcoming.

In the best case scenario for Boeing, the plane could be granted authority to resume service with frequent inspections and perhaps limitations on how far over-water it can fly (ETOPS) to ensure there is always an airport close enough for emergency landings.  Boeing has submitted such a proposal that would entail additional inspections, but we do not believe it will be accepted given the severity of the situation.

Our expectation would entail a redesign of the battery system, and entail a 6-9 month process to certify the changes and bring all of the 50 aircraft currently in service and those in production up to this new standard.  This would be a delay similar to the six- month delay in redesigning the power distribution panel that failed on a test flight in 2010.

The worst case would be a review of the entire electrical system, which could take longer, but is a real possibility if it is determined that over-charging damaged the batteries, as discovering the cause would require additional forensic engineering.  If overcharging was not the cause of battery failure, finding the cause could take additional time.

The on-going investigation and review, with the FAA, CAA Japan, Boeing, National Transportation Safety Board, the airlines, component suppliers and other international regulatory authorities involved, may take a week or two to make a determination of risk, and decide whether the ban on flying should be lifted, or remain in place until a permanent solution can be found.

What’s Happened Recently to Force the FAA to Change its Mind?

On January 11th, the FAA initiated a comprehensive review of the 787 program, while proclaiming the airplane safe, and the Secretary of Transportation personally indicated that he would have no reservations flying in the 787.  Five days later, the FAA reversed itself, and along with other international regulators, and grounded the 787 on January 16th due to a fire involving Lithium-ion battery on January 7th onboard a Japan Air Lines (JAL) flight that had landed safely in Boston. Nine days later, a second battery incident that resulted in a meltdown but not a fire, on an All Nippon Airways (ANA) flight caused an emergency landing at Takamatsu.

This grounding came quite quickly after the FAA and Department of Transportation launched a “comprehensive review of the 787 and its critical systems, including the design, manufacture, and assembly of those systems.”  That review was to focus on the electrical system, before the second battery event with ANA prompted the emergency grounding of the aircraft.   The fire in Boston involved one of two Lithium-ion batteries on the Boeing 787, in this incident the one located in the belly of the aircraft to power the Auxiliary Power Unit, or APU.  The photo below of the fire scene illustrates smoke coming from the aft compartment, which took firefighters 40 minutes to extinguish, finally removing the battery and placing it on the tarmac during that ordeal.

webjetfire001                                                   photo: PictureBoston

The second incident entailed an in-flight smoke situation involving a Lithium-ion battery contained in the forward avionics bay of the aircraft, used as a primary backup to the electrical generators that draw power from the engines.  This battery was severely damaged, with an indentation in the case that contains the battery, and caused smoke in both the cabin and cockpit prompting the emergency landing, as shown in the photo below.  Passengers and crew were forced to evacuate the plane using the emergency slides after landing, but fortunately no major injuries occurred.

images-1

These two incidents were the most serious of a recent spate of glitches with the 787, that ranged from a cracked windshield to a fuel leak on another JAL aircraft from a faulty valve.  In all, nine separate incidents occurred with the 787 over the last few weeks, resulting in the FAA deciding to undertake a special review of the aircraft before the second battery fire incident and issuing the airworthiness directive prohibiting flight.   The significance of that flight prohibition should not be taken lightly, as it was last used in 1979 after the crash of an American Airlines DC-10-10 in Chicago that resulted from improper maintenance and the use of a fork-lift truck to remove engines, weakening the engine mounts.  Fortunately, in this instance, there was neither a fatal crash nor any loss of life.

While some industry participants, and Boeing, believe the grounding to be an over-reaction, the FAA would rather err on the side of caution and ensure that any potential problems with the aircraft are sorted out on the ground, rather than in the air.   The Emergency Airworthiness Directive issued by the FAA on January 16, 2013 states: “This AD requires modification of the battery system, or other actions, in accordance with a method approved by the Manager, Seattle Aircraft Certification Office (ACO), FAA.”  It is notable that this notice does not say inspect, or monitor, but specifically uses the term “modify.”  The implication is that the FAA believes that changes will be necessary to the airplane.

images

Modification to the battery system on the aircraft is not simply swapping a Ni-Cad battery for a Lithium-ion battery and putting the aircraft back in the air.  The electrical system for the 787 was designed for Lithium-ion batteries, and has a high power level (1.45 megawatts) five times higher than prior aircraft because it has replaced pneumatic (bleed air) and hydraulic systems with electronic controls in order to save weight.   The 787 has six generators on board, compared with the normal three on more conventional aircraft, and has two Lithium-ion rather than one Nickel-Cadmium battery on board to generate back-up power.  These batteries are re-charged in flight from power generated by the engines in the more complex electrical system.  Included in that electrical system are controls to ensure that a batter does not overcharge, and that any problems will result in a cockpit warning.

Why is this a Potential Safety Issue?

The answer is quite simple. These failures could lead to a fire in the electronics bay, housing the avionics and computers that operate the aircraft.  In addition, an on-board fire could result in damage to critical systems and even structures, much of which on this aircraft are made from Carbon Fiber Reinforced Plastic materials.  An on-board fire, particularly on an aircraft that relies on electronic controls that could be severed in a fire, is clearly a significant danger.

Prior to the 787, aircraft typically utilized both hydraulic and pneumatic systems to manage flight controls.  On the 787, controls  are provided by a “fly by wire” system in which they are electronically controlled.  This innovative technology saves considerable weight, as heavy hydraulic and pneumatic controls are replaced with lightweight electronics, and wiring rather than piping fluid or air throughout the airplane.  However, a problem with the electrical system that interfered with operations of those flight controls could result in loss of control of the aircraft.  Plus, these computers require a smooth flow of power, which is one of the benefits of Lithium ion batteries in a back-up situation.   While the electrical system on the aircraft has 6 generators and the batteries are primarily for back-up power, they are an integral part of the system redundancies and multiple back-ups that provide an extra margin of safety.

Why are Lithium-ion Batteries Suspect?

Lithium-ion battery technology is not new, and was initially introduced by Sony in 1999.  Early on, however, there were problems with some laptop batteries that overheated and exploded on some laptop computers, which have been solved as the technology has evolved.  But these batteries generate a lot of heat, as anyone who has a MacBook Pro knows.  If you place the computer on your lap while it is charging, you will soon want a pillow between the bottom of the computer and your legs, even if you are wearing jeans.

There is a history of Lithium-ion batteries catching fire, and shipments of Lithium-ion batteries that caught fire in the cargo compartments of two freighter aircraft have been confirmed, and suspected, as the cause of two aircraft crashes, respectively.  Today, TSA requires that any additional laptop batteries are checked in the cargo hold of the aircraft rather than carried on-board.   Lithium-ion batteries have several advantages, including higher capacity than traditional NiCad batteries, lower weight, and the ability to be designed specifically to fit within a specific area.  Boeing chose Thales SA, based in France, as the supplier of the integrated battery system for the aircraft.

Lithium-ion batteries have a history of “runaway failures” from overcharging, where once cell of the battery can literally explode into the next, causing a chain reaction and a fire.  With the high voltage requirements of the 787, about 5 times those of traditional aircraft to run everything from flight controls to the electrochromic window dimmers, the robustness of the electrical system is a necessity.  On the 787, fail-safe systems are installed to monitor the batteries and prevent a battery meltdown or catastrophic failure from overcharging.  Given the picture below, which shows the battery from the ANA flight and a new one, those fail-safes may not have been activated early enough, or failed to prevent the batteries from entering an unsafe condition. Regulators now suspect that an overcharging condition may have existed on the airplane.  That could result in a more detailed investigation of the aircraft’s electrical system.

2020162345                                                 Photo: Seattle Times

If it is the batteries, how long will it take to fix the problem?

That batteries that failed on the ANA and JAL aircraft were supplied by GS Yuasa, who indicated that it may take six months to determine whether it was an issue with the battery, or longer if there is an issue with the electrical system on the airplane.  And if it takes six months to diagnose the problem, add on a few months more for correcting it.   It is possible that both of the defective batteries could have been produced from the same batch, but that begs the question of why Boeing’s quality control practices could not adequately test the batteries or determine if they were defective before installing them on the airplane, and what they will do to ensure that this won’t happen again in the future.

An earlier electrical failure on a power distribution panel during flight testing in 2010 resulted in a six month delay to the program.  We would expect a similar time frame for problem resolution in this situation, with additional time to retrofit and certify the fix.   A key element for the 787 is software, as the battery system is integrated into the electronics systems, with computer controlled fail safes and other features.  How would a change in specification to this component impact other components on the aircraft?  With 17 million lines of computer code driving this aircraft, just reviewing how this system interfaces with other systems is a massive, but necessary task.

Given that the ANA battery leaked flammable electrolyte from the battery, and that the incident was not fully contained by the case, the top of which was bulged, there may be a requirement to increase shielding around the battery.   The 787 has a carbon fiber reinforced plastic structure, with about 50% of the plane made of plastics.  Plastics react differently to heat than metal, and a carbon fiber panel burning will release small particles that could negatively impact nearby electronic systems – which on an all electronically controlled plane could be catastrophic.

A fire is the last thing you want on an airplane, and a battery leaking flammable electrolyte while overheating enough to deform its case is not a good thing.  It is likely that the failsafe system to stop charging the battery may have contributed to the failure on the ANA flight  – but what if this incident happened over the open ocean rather than near Takamatsu?

Interestingly, the ANA 787 did report a battery warning before takeoff — but it was on a screen pilots are unlikely to look at while running their pre-takeoff checks.  At a minimum, an enhancement to the systems should be made to flash this alert in the cockpit to warn the pilots not to take off and not go further through the checklist, but return to the gate. But a problem with hi-tech airplanes is information overload.

In the Airbus A330 crash off Brazil, when the pitot system failed, hundreds of warning messages came to the pilots – impossible to comprehend in a short time, as every system that failed as a result of the first system generated its own message, cascading into information overload.  For a programmer, this is simple – if error, then send warning message.  For the pilot and passengers, it can be fatal, as it distracts from the fundamental job of flying the airplane.

The 787 is much more sophisticated than the A330.  Should somebody look at the 17 million lines of code and develop a mechanism to sort out which faults are primary, and which faults are secondary, and which are the most important for the flight crew to see?  Or should they be illustrated differently on a screen, so that the first fault in a system can be readily identified, and the flight crew better understand what to do?   Perhaps a critical systems fault should trigger a transponder code change to the emergency code of 7700 – at least then Air Traffic Control would ask the pilots “what is the nature of your emergency”, and serve to alert the crew that the airplane should be put on the ground ASAP.

Did the Certification History of the 787 Play Any Role in its Grounding?

The FAA has historically relied on company personnel, who are seconded to the FAA as Designated Engineering Representatives and Designated Airworthiness Representatives, who have the power to approve designs on behalf of the FAA.  The FAA has neither the staff nor engineering resources to independently review aircraft designs.  In this process, the FAA relied on Boeing data regarding the electrical system and battery configuration.  But there is a built-in conflict of interest that nobody likes to talk about in the industry – effectively the self-certification of airplanes by the manufacturers.

There is always a difficult dilemma for a DER or DAR – do I side with the FAA, to whom I have oversight responsibility, or do I rock the boat with my employer, who signs my paycheck?  To truly be effective and ignore the inherent conflict requires a very strong engineer to stand up to his CEO and say no, and put his job, future promotions, and career in the industry at risk.   Knowing that Boeing was rushing to get the plane completed as quickly as possible, after running into problem after problem during its design, regulators may not be comfortable that things weren’t rushed a bit in the efforts to get the 787 back on schedule, and the initial review, prior to the grounding, was to cover design, manufacture and assembly.

Does Boeing Have a Quality Problem with the 787?

With the delays in certification of the 787 of 42 months, Boeing had started production of aircraft that, one the design was finalized, required significant rework.  That rework, necessary before the airplanes could fly, resulted in non-standard production processes for initial aircraft.   This begs the question of whether this rework resulted in a reduced level of quality?  Was Boeing in too much of a hurry to generate cash?  Was there shoddy workmanship – as potentially evidenced by mis-wiring in the battery compartment of a United aircraft that was inspected after the first JAL incident?  Such connections are normally double and even triple checked at the factory, and signed off by inspectors, so a mis-wired aircraft should not be allowed leave the factory if processes are working well in organizations that espouse six sigma standards.  The investigation will likely include recommendations for Boeing in that regard that could negatively impact its production ramp-up.

How much will this cost Boeing?

It is impossible to determine at this point in time, but already some airlines with 787s are demanding compensation for the loss of service with these airplanes, much as they asked for compensation when the aircraft were initially delayed.  Because the Emergency FAA AD prohibits Boeing from test flying new aircraft coming off of the assembly line, it cannot finish aircraft that are currently work-in-progress, and will likely need to retrofit those aircraft with whatever fix is required.   With respect to the backlog for the airplane, this is unlikely to cause cancellations of orders with Boeing, because Airbus is also sold out at their current production rate for several years.

When the 787 was initially delayed, the A330 received a spate of additional orders, and as a result, delivery positions are sold out for a the next three years, (Airbus has orders for 306 A330 family models, and produces about 100 annually), and with 576 orders the A350 is sold out for its first few years of production.  So there is no instant relief available, except to keep flying the older 767s that were scheduled to be retired. But some airlines bought the 787 for growth, and will need to curtail their expansion plans and cancel planned new routes, unless they can substitute other used aircraft for a short-term solution.

The reputation of the 787 was just beginning to improve after its entry into service before these new difficulties emerged.  It has reversed itself, and is certainly a black eye for Boeing.  Will it be able to recover, and will the program generate new orders?  Earlier this week, American Airlines confirmed its orders in bankruptcy court, and several airline CEOs are publicly standing behind their decisions to purchase the aircraft.  But in the back of their minds, they are also thinking about plan B, and how limited their options appear to be.

Will the reputation of the 787 be rehabilitated in the future?

The delays in development have already negatively impacted the reputation of the aircraft, and of Boeing’s ability to deliver a new product on time.  Once the problems are settled, the key may be the development of the next model in the series, the 787-10.  If that can be done on-time, and successfully introduced without problems, the reputation of the aircraft could be rehabilitated.  But this will take some time.

+ posts

7 thoughts on “787 Grounding – AirInsight Perspectives

  1. The cracked windshield sounds like a teething issue. This electrical problem seems more systemic, very serious indeed. Was the fire in the test 787 related to the batteries? Either way, once Boeing figures it out and I know they will, seems like we can all put this behind us. i’m glad they decided to ground the planes and take the time to figure it out. In the end this means it will only be a provision on the income statement and no loss of lives. I guess we can put a dollar amount on a six month delay and I think we all would rather see that then any loss of life. Couple of questions I’d love to get answers to:
    1) what other planes use these lithium ion batteries
    2) do those others come from the same provider. If so, I guess the big question is: what is different here…
    3) is the electrical system the same. I.e. do batery vendors and electrical system vendors usually do joint design or does integration happen at oem (boeing, airbus, bbd, etc)?
    4) will the a350 or cseries use li-ion batteries?
    5) after the swissair crash a couple of years back (fire above the pilots in after market A/V system) that rendered it impossible to fly the plane bc melting plastic was dropping onto the pilots), wouldnt it makes sense to treat the cockpit as a secure box. I.e no batteries, transformers, converters or anything that can cause fires in front of, beside, behind, above or below the pilots. Move the batteries to the belly in a secure box within a secure box within a secure box. And only send the wires to the cockpit. I think we need to secure the cockpit from fires and if that means only sending minimal wires and fiber optics to it, then lets do that…
    Of course I know very little on AC electrical systems but just a thought worth 1.5 cents…
    PierreBus

  2. Sir, I think Boeing has to concentrate on quality, I am sure rushing the work and introducing more products / plane will not help if quality of work is poor. I am sure Boeing can reach its target by investing more in planning, training, coaching and testing. Boeing as much as they can has to relay on her manpower to supervise and certify others work. Without this Boeing will move from one trick to other and so on.

  3. It would be hard for airlines to explain the customers the aircraft is 100% safe and ETOPS 60..

    Re those batteries, Airbus uses them too, but in a more conservative lower risk function ( less powerfull, smaller, less frequent use/ charging), for emergency lighting on the A380. Be sure they have a second look for the A350. Airbus selected a less electrical dependent system buildup anyway.

  4. To date, the best story on this subject I have read so far…and I have read them all…at least twice. Good work.

  5. I am a Boeing employee working in QA and worked on the 787 program for 5 years. Like many QA professionals that had any amount of experience, I was stunned when the FAA signed off on that airplane. Although 787 has a “Quality” organization, it is “Quality” in name only. In reality, what they do has much more to do with manipulating paperwork and supplier ratings; a corporate culture which has yet to change. Like many Boeing employees, I am angry, saddened, and embarrassed that a company that was, at one time, so great, has stooped to a level of sleaze that can only be compared to Enron, and until Boeing cleans house; REALLY cleans house with the highest levels of 787 QA management, getting the 787 program on track and making it the great airplane it should have been will never happen.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.