Hacker News new | comments | show | ask | jobs | submitlogin
Delays in Boeing Max Return Began with Near-Crash in Simulator (www.bloomberg.com)
173 points by pseudolus 6 months ago | hide | past | web | 191 comments | favorite

FWIW, the near-crash in the simulator was apparently not directly related to MCAS, but occurred when Boeing "simulated what would happen if gamma rays from space scrambled data in the plane’s flight-control computers."

> In one scenario, the plane aggressively dove in a way that mimicked what happened in the crashes on the grounded jetliner, the people said. While such a failure had never occurred in the 737’s history, it was at least theoretically possible.

> Because at least one of the pilots who flew the scenario in a simulator found it difficult to respond in time to maintain control of the plane, it needed to be fixed, according to two people familiar with the results.

However, the next sentence seems to belie the implication that this was just an edge case:

> The answer was to modernize what was a relatively antiquated design on the 737.

So, of all the things that caused Boeing/FAA to realize and the accept the 737 needed to modernize its decades-old design, it was a simulated edge case involving "gamma rays from space"? Hate to be overly cynical, but that sounds like the kind of CYA thing you'd say when you don't want to admit it you made a huge mistake by deferring modernization in the first place, of which the MCAS concept was a major and disastrous symptom.

> a simulated edge case involving "gamma rays from space"?

People have nearly died because of exactly this situation. See, for example, Quantas Flight 72 [1], in which a pair of sudden, uncommanded pitch-down manoeuvres that caused severe injuries -- including fractures, lacerations and spinal injuries -- to several of the passengers and crew.

A long and detailed investigation essentially concluded that these events were most likely caused by the CPUs not being sufficiently rad-hardened. The pilot was diagnosed with PTSD, caused in part by being totally out of control, and quit his career [2, 3].

[1] https://en.wikipedia.org/wiki/Qantas_Flight_72 [2] https://www.smh.com.au/lifestyle/the-untold-story-of-qf72-wh... [3] https://www.harpercollins.com.au/9780733339745/no-mans-land-...

That's not what the Wikipedia links says:

"A number of potential trigger types were investigated, including software bugs, software corruption, hardware faults, electromagnetic interference and the secondary high energy particles generated by cosmic rays. Although a definitive conclusion could not be reached, there was sufficient information from multiple sources to conclude that most of the potential triggers were very unlikely to have been involved. A much more likely scenario was that a marginal hardware weakness of some form made the units susceptible to the effects of some type of environmental factor, which triggered the failure mode."

Cosmic rays would be a very convenient excuse. (Cue the ECC RAM lobby, which will probably downvote this.)

> a simulated edge case involving "gamma rays from space"?

Soft-errors are a prime concern when designing safety-critical systems, both in hardware and in software. It's not some outlandish edge case.

Incidentally, while soft-errors in an airplane are probably primarily particles from space, on the ground we're still sometimes talking about it, but in reality it's mostly chip packages that are low radioactive and emitting a alpha particle or so.

I don't disagree that this is an important thing to test for. I question why this is the reported explanation for why Boeing now decided that the computer systems needed an extensive overhaul. Ostensibly, "gamma rays from space" affecting flight control computers would be an issue that affected Boeing's planes long before MCAS.

But it's not the gamma rays that are the reason for the systems redesign, it's the fact that "at least one of the pilots who flew the scenario in a simulator found it difficult to respond in time to maintain control of the plane".

Boeing's assumption that pilots could respond quickly enough to a loss of control in the cockpit was an underlying motivation for designing MCAS with inadequate safety measures.

Also, there is a regulation that requires no design have a single point of failure capable of resulting in a catastrophic outcome.

That's why they have to rearchitect the system. If those bit-flips happened to the Flight Computer in command, they could lose the plane. Therefore, redundancy must be built in.

It wasn't originally evaluated as posing a catastrophic outcome in the safety analysis, which is why no one paid it any mind.

Single Event Upsets are a real thing and if they had a failure mode that missed that, it's a problem that needs to be fixed.


Airplanes are flying above the vast majority of the atmosphere, which provides a majority of the shielding for us on Earth. It's a real concern, and extensive care is taken when designing spacecraft electronics for exactly this reason.

>at least one of the pilots who flew the scenario in a simulator found it difficult to respond

And I'm guessing their not using pilots of average skill here...

According to [0] the FAA was looking for numerous candidates to get a good range of experience, so I would think that you might’ve had at least a few average pilots.

[0]: https://www.reuters.com/article/us-ethiopia-airplane-faa-boe...

Yeah there is a push for using more avg pilots.

...but on mission critical testing with billion dollar implications. You want to tell me they're pulling that name out of a hat?

FAA, right. Trump's pick for FAA head was confirmed only because the majority in the senate (you guess which party) decided to look the other way about allegations that while at Delta, he mistreated a whistleblower who raised safety concerns to him. Oh, and he's a lawyer. I wouldn't trust the current administration's FAA on this. Yes, politics does come into it whether we like it or not.

Correct, I’d wager the best pilots in the world are safety and test pilots.

Maybe. I don't know the intricacies of testing flight control software but I imagine Boeing will have a bunch of professional 'test pilots' augmented with regular pilots they 'borrowed'.

I find it hard to believe that they are completely re-designing the avionics/flight computers because of this gamma ray incident. There must be another reason.

If it's seriously the main concern, wouldn't it be much easier to just encase the computer in several inches of lead shielding?

Is this an issue on non-MAX 737s as well?

If not, and thus the affected computer system has been added in the MAX, why not design it properly with 3+ redundant copies of the system given that it was designed in the 2010s when both engineering theory and technological resources are plentiful?

It isn't an issue on non-MAX 737's. The flight computers on NG, and others don't have final say level control over the horizontal stabilizer, and there are several overrides the pilots have built in that allow the pilot to lockout trim commands from the Flight Computer, such as hauling back on the control yoke.

The can of worms has been officially opened, and this may turn out to be a case study in how bad software and outsourcing your centers of excellence can bankrupt a company.

If TFA is to believed, Boeing has embarked on a project to take what is currently a master-slave failover design and jury-rig it into a master-master real-time system. This is based on bad performance in the simulator—not because of MCAS—but while testing errant memory errors from a simulated gamma ray strike!

The obvious question of how can dual-master ever work when they disagree if there isn’t a 3rd source of truth to vote out the failure? This means the software running in both computers has to somehow agree which one is wrong, while one of them is in a potential failure state. This... isn’t how flight control systems are designed from first principals.

I’m not a flight system designer, but there must be hundreds of physical, electrical, and architectural considerations taken at every point in the design process which enable multi-master controllers with the ability to vote out a failure. Everything from the particular sensor suite, the number of sensors, the way they are wired, the way data is acquired and bused through the system, the timing and synchronization of the system clocks, the way that control outputs are calculated, queued, and ultimately issued to downstream controllers... none of the necessary pieces will be in place in a system which up until now makes you select a single master controller before embarking on your flight.

The only conclusion I can reach is that Boeing has lost its mind, and this project is absolutely doomed to fail. The architecture that Boeing has apparently committed itself to now is extremely difficult to design from first principals and a blank slate. I just don’t see how it’s something that can realistically be papered onto a legacy dual-computer system after the fact.

The part in TFA about “adding a wire” practically made me spit out my coffee. The sheer level of arrogance that Boeing management must have to think this would be possible - is just a classic example of an elitist MBA management group totally disconnected from any technical domain expertise. Boeing said they would have this ready for certification by the end of this year?!

Classic case of hitting the date but crunched out with technical debt and buggy software, a tell that management/finance/marketing is driving the decision over the engineer/designer/product people.

People forget a project was late if it succeeds, they never forget a project on time that failed because it was rushed, buggy and not polished production.

MBA/business and marketing need to go back to letting the product people and engineers create value, then they can extract the value and sell it. It is much easier to sell value of a good product, it basically sells itself. For some reason American business has completely forgotten the rule this millennia, product then marketing and selling.

You see this same thing happen at game companies when they put a date out there and it is driven by management and marketing over reality of engineering/product/design that makes a fun and solid game.

Boeing was greatly diminished in engineering/product excellence by the McDonnell Douglas management control takeover, everything newly produced since has the tell of this.

People forget a project was late if it succeeds, they never forget a project on time that failed because it was rushed, buggy and not polished production.

The problem is the 737 Max was hustled quickly to market to beat airbus and especially to offer a "drop in replacement" to existing earlier generation 737. Boeing would have definitely lost market share if they hadn't gotten their plane out then.

Choosing technical debt involves economic logic for today's marketplace. I suspect the real problem is companies wind-up deluded by their choice - to hustle the 737 Max to market, Boeing had to pretend is wasn't a, uh, piece of crap. Or not a piece of crap but a terrible mix of high tech and low compromises. And this pretending had to go through the whole enterprise. The marching order couldn't be "we'll add few more stiches on this Frankstein and hope the guts don't spill out", they had to be "this will be one more product of Boeing excellence" and everyone either believes or acts like they do. and this situation, combined with the huge investment involved, makes cutting losses very hard.

> People forget a project was late if it succeeds, they never forget a project on time that failed because it was rushed, buggy and not polished production.

#notallMBAs, but the surest way for your product to fail on the market is for your company to go bankrupt before you’ve polished it enough.

In this specific case it was about greed, but sometimes MBAs push for release, because according to the engineering teams the product is never truly perfect

A crunch type culture and pushing can also create the issues seen in these problem projects. In fact I personally believe this is why more software and products are bad today, the product professionals are being overridden by marketing/finance and exact dates to meet some marketing goal not a good product goal.

Product people and engineers know how to manage time probably better than business because they are doing the work.

If something fails because it doesn't meet the market there are numerous reasons for that besides not hitting the date.

A good product, that hits the date, can still fail due to market timing or market movement and many other things. Whether a project is launched at a certain date has little bearing on the success of that project long term, short term it might meet marketing and financial quarterly goals, but long term success is always better when it is a good product, and for it to be a good product the people that make products and create value should determine or at least have some say when it is complete and ready.

"Perfect" is of course unattainable, but I'd at least hope that engineering could be able to push for -- and be granted -- the time to build a system that meets what should be some pretty obvious safety requirements around redundancy and failure detection.

"Dual-dual" is a thing in this space. There are safety-intent microprocessors which have a couple of cores running a couple of cycles out of sync with a delay and comparison on registers. If they disagree after that delay, the processor concludes it's insane and you can take action based on that hardware-level signal.

It's not my favorite solution for every application, but it's a good tool for a handful of situations. And it's a good way to avoid having to think about picking who to trust with only two sockets. There are other approaches, of course. For example, if your system can afford more than a couple clock cycles of latency (e.g., if you don't believe you can crash a plane with a couple milliseconds of control error) you can do the same thing with well-designed software and a bit of extra hardware. It gives you more freedom since there's more hardware to buy, at the expense of engineering and test effort to prove that your solution does what you want.

Doing this over large physical distances also adds some challenges. There are debates around dedicated signal wires direct between control centers vs data networks. Every situation is a little different, so there aren't really easy answers that you can swoop in and prescribe as best-practices. In general I find that people are more-skeptical of packet-switched solutions (e.g., Ethernet) than I would be. But most folks are also better at reasoning about the failure modes there than they are at thinking about the lower-level issues that Ethernet solves for you. And of course there are tons of different signaling systems - not just Ethernet - to consider.

Consider just a few different possible applications and the constraints that they impose. A car has less severe failure modes (many crashes are nonfatal) but less freedom in time. Your trajectory is simply more cluttered. A passenger plane has dramatically more energy, so failures are typically more severe, but you have a lot more time to deal with them because your trajectory is empty (much of the time, anyway). And if you're sane, you've designed it to be passively aerodynamically stable because twitchy maneuverability isn't critical when you're not a fighter jet. A rocket is a different animal altogether. Those are often rigged with self-destruct explosives, and you don't really get much time to change your mind if you erroneously choose to trigger them. Plus they're typically less stable (performance is more dear, gravity isn't helping you, and purely aerodynamic solutions don't take you outside the atmosphere) and are thus more dependent on their control systems.

I'm confused as to what Boeing is changing then - your description of 'dual-dual' systems sounds like what is happening with the master/slave failover system already in place. It sounds like what they're changing is to have the two systems cross-check each other in real time, rather than have the in-band delayed core be the sanity check for each system, which seems like a big difference because it means that there are now disagreement scenarios whereas an in-band sanity check always is just checking one system against one set of invariants and thus can't disagree with it's own observations. Or am I misunderstanding?

You're forgetting there's a third system: the pilot.

If the majority of computer aids turn off, the plane doesn't simply drop out of the sky. It reverts back to a more complicated, but human-controllable, vehicle.

From the article (and who knows how much Bloomberg understood) the active-active sounds more like recognizing, not recovering from, errors.

Which isn't the worst idea. Pilots being, well, pilots.

I think the problem is that the 737 Max is the product of a long line of planes engineered to fly the original 737 but now flying very differently due to the massively increased engine size, which supposedly creates a much more unstable aircraft.

So you're asking the pilot to, in the midst of some other failure, to start flying a weird plane they ever flown before. You could solve that problem by training all your pilots in this BUT this solution is very expensive, so expensive the airlines might just demand their money back instead. And that wouldn't be a solution for Boeing.

All true. But from where we are today, that seems like a better solution from a safety perspective.

And now, the FAA (or EASA) might just mandate that anyway.

I wouldn't be surprised if CAAC refuses to ever recertify the 737 MAX, especially if a trade deal isn't struck.

As a tangent, there have been times when HN is ahead of the curve in understanding where something is headed before the investing public.

The benchmarks and excitement around AMDs offerings are a great example. When you see something that presents a paradigm shift, often the market will not have fully digested it.

AMD is one of those cases, and I believe BA is as well. Their stock is essentially unchanged from where it was before the Lion Air crash in October 2018. This company is worth $200B and we’re talking about 5,000 units of a $100mm plane (that’s $500B in revenue right? I’m not adding a zero there?)

$100mm is the list price. Actual prices are decided by the market. I've read about $50mm for the 737-8, the baseline 737MAX model. That was of course before the MCAS disaster.

My layman understanding is that actual prices would be around 40% off list price for models that sell really well, with little competition (for instance A321). Much steeper discount above 60% for models that have a hard time selling (A330NEO) or for end of line to close a production gap (777-300ER is very cheap to buy while Boeing is working on the 777-9).

That, or this article missed the boat somehow. I find it hard to believe that a total redesign is happening and that they expect to be done by the end of the year. Either this article is wrong, or something is terribly wrong at Boeing.

I'm a huge MAX skeptic, but this article just has to be wrong. Someone who had dropped out of a CS degree would know not to do what the article says they're doing.

You would be surprised, if managament is broken to it's core, the situation is "Do this unethical thing or we'll find someone who will!"

Computer Scientists aren't usually taught engineering ethics and definitely get fired all the time for arbitrary reasons.

Is that true that CS students don't take an ethics course? I have a 10 year old SE degree and I took 3, only 1 of which was engineering specific. (I took a philosophy ethics course, which really isn't that relevant to issues like these anyway, but also a course that was specific to ethical issues in computing.)

OTOH, I'm skeptical those courses change behavior, and in this particular case others in comments are saying this is a common architecture in the space, so how could anyone solve this ethical conundrum if getting to the base level of whether this is an ethical conundrum to begin with is difficult?

In my opinion courses wouldn't change behavior, modern computer science is designed to infantalize coders and make them interchangeable. Think of all the effort startups put into marketing themselves as "COOL TO WORK FOR"/"WE HAVE BEER ON FRIDAYS". These reinforce the idea that you are small and need management to tell you exactly what to do everyday, because you're a dumb idiot who spends all day watching anime and playing video games.

In that environment, serious ethical concerns could easily be brushed aside as "coders just being coders". When Google got found out for making drones, the employees of a company staged a 5 minute walkout.... that was it, I'm sure some people quit, but not enough to make any impact

I took one course that briefly touched on engineering ethics in my ECE program. Nothing even remotely resembling the ethics courses taken by the Mechanical/Civil/Structural guys.

I had one ethics course. Class of 2013 in a prominent Pennsylvania university.

SE = structural engineering? That's cool, I'm glad your discipline is receiving some ethical training. I've never heard of any CS program including any sort of engineering or general ethics courses in their curricula.

I wonder if someone talked to this reporter about a long-term redesign they're doing, and the reporter conflated that with the short term patches they're doing to get the plane flying again. It just doesn't make sense otherwise.

And, 100 scenarios instead of 10? I sure as heck there's more than that on an architecture redesign!

No kidding! How sad is it the entire time I was reading this I kept looking back at "Bloomberg" and was thinking to their other reporting like the server spying that they never did explain.

The National Enquirer may be more reliable than Bloomberg at this point.

>Landing contracts for key systems on the Max cemented a relationship with Boeing that Collins had long cultivated, in part by creating what its chief executive in 2002 called an “enormous sea change in a very long legacy business." For years, the company had been “a technology leader, an innovator,” Clayton Jones, then the chief executive, told Fortune magazine. “Unfortunately, along the way they forgot to hone their financial skills.” In 1998, Boeing executives summoned Mr. Jones to Seattle, he later recalled in a speech, and made clear that, to get more of Boeing’s business, Collins would have to cut prices dramatically. In response, Collins introduced what it called “lean electronics,” its take on a belt-tightening philosophy popularized by Toyota.

>Collins reorganized business units and retrained managers, with an eye toward efficiency and speed. It pushed its suppliers to do the same, and established partnerships with companies such as HCL Technologies, which provides outsourced, lower-cost engineering services from India.

>Along the way, Collins unseated Honeywell as the provider of flight control computers on a predecessor of the Max, the 737 NG, and supplied numerous systems for the Boeing 787, which went into operation in 2011.

>When Collins secured the contract for the Max displays in 2012, it credited the belt-tightening. “There were a lot of cost-saving measures — a lot of tough decisions — that had to be made,” one manager said at the time in a company publication.


If there was a problem with the master-slave operation on the MAX, not necessarily connected with MCAS, might not the same problem exist on the 737 NGs?

not likely the NG didn't add new systems. not saying other cost-cutting won't be exposed but nothing as egregious as MCAS.

the MAX added new engines, which required forward placement on the wings, which changed COG, which necessitated MCAS.

They changed flight control computers from Honeywell to Collins when going from the 737 to the 737 NG. Seems like a new system.

Right, but the recent change was from the 737 NG to the 737 MAX.

I hope they aren't expecting people to fly on these planes.

A split brain in a plane is a bit of a nightmare. I'm not sure how they can work around it with just two computers.

Big "disagree" alarm for the pilots and let them decide to keep either computer or go direct. Airbus has the concept of flight control laws, surfaced and changeable by the wet backups.


>go direct

That's not really a thing in modern airliners. The computer is still heavily involved even when hand-flying the plane.

"Thankfully", the 737 series isn't a modern airliner, and isn't fly-by-wire. All control inputs are directly linked to the relevant outputs, though sometimes with power assistance.

It is on the Airbus, one of the control laws, see link above. It's still FBW though, so one might argue that "mechanical backup" is really what you want.

As for "modern", the A320 had control laws in 1984 and the 777 in 1994.

And Boeing will make that disagree alarm an optional extra?

Am optical extra that isn't actually properly functioning as intended in numerous delivered products? Almost as if it wasn't sanity tested in one production unit, let alone some random sample of units?

Which computer calculates that there's a disagreement, and how can we ensure it's not indicating a false negative?

Either. The alarm must be an OR.

Then, IMO, it should be two independent indicator lights, not 1.

my CS201 class 19 years ago taught me this. education just isn't as good these days..

I didn’t feel this way at first, but now I feel like I will never fly on this plane or allow my family members to fly on this plane.

tl;dr : The Kruger-Dunning effect is most visible in the upper-management circles, where politics and charismatic talents are necessary skills.

Former engineering powerhouses like HP, Boeing, and IBM slowly get hollowed-out into incompetence when it’s assumed that Engineering and technical skills don’t require a considerable depth of knowledge and experience.


could it also be the normal generational rotation of people through the organization?

Everyone grew at a tremendous rate since the 1960's. Many, many more engineers. And then they retired.

I don’t think it’s generational. It’s just unlearned lessons of history.

The most effective leadership for a technical company is leadership with actual technical creds to back it up.

The technical side is a must because you need someone who actually understands the business and the innovations that must takes place. But you also need someone who can provide the people-leadership skills and the business wheeling&dealing.

These are all crucial skills and take up a persons time, so it’s not uncommon to see this come in the form of a duo (Apple and Steves, or Microsoft with Bill+Steve).

> bad software

The software was fine (or at least not to blame). Unstable aeronautic design and lack of redundant sensors was the real problem.

Looking at the current article the software is also low quality.

If what we read is true this is then the second time that Boeing is trying to fix hardware shortcoming with software in a hair raising manner.

What is TFA?

They're redesigning the architecture and expect to be done by the end of the year? That is insane. The MCAS change was a patch of one fairly isolated system as I understand it. Redesigning the entire architecture of the flight controls seems like a years long process. Is this article correct? If so, I don't think I'll be riding on a MAX any time soon.

These should not fly for a decade. They're rewriting the avionics from scratch. This should require a years-long process of testing, external auditing, and approvals. I'd even go so far as to say in flight, we should require competitor review. That's right, no trade secrets. If you want your Hello World up in the air, you better let your competitor vet the source code.

This is an engineering boondoggle and an embarrassment for Boeing. Software does not need to be this complicated. Design a damned airframe that's airworthy without needing stabilization hacks.

Agreed, the MAX was only designed to beat the competition to market with the least amount of re-certification and re-training. That gamble (with Boeing's reputation, the FAA reputation and most importantly, people's lives) did not pay off. They won't get away with rushing though certification of a complete rewrite of their flight software. The world is watching now. Time to cut losses and junk the MAX entirely.

I'm not sure why this keeps getting repeated.

Boeing certainly bears culpability for failing to execute, and whatever terrible lapses in certification were made to get there.

But why were they doing a dumb thing in the first place?

Because American Airlines asked-told them to. (Because they didn't want to have to pay to retrain pilots)

Boeing deserves to have substantial portions of management jailed over this, but there's blood on the legacy 737 operators' hands as well, for asking them to do it in the first place.

Boeing could've said no.

I agree with you.

This sounds like the ratings agencies during the last recession, where they knew they were handing out good rating for junk bonds because if they didn't do it they'd lose a customer to the competition.

It's totally fucked.

In which case American would have bought exclusively new A320neo's.

Boeing was caught flat-footed and underinvested in a new product in 2011.

So while Boeing management could have said "No" to their largest (?) customer, that would have been a hard decision to make. And probably would have led to the board chopping C* heads for breach of fiduciary duty.

It's not as if the current situation, in which Boeing is currently causing all their customers lots of financial pain each day the MAX is grounded, and the CEO of the commercial division has been fired, is any better.

The Airbus order backlog is almost a decade out as it is, so American just shoving it all into A320neos that would take longer to show up wouldn't exactly be realistic either.

Boeing should've just gone with a clean-sheet design.

It's still Boeing's fault. They're the ones that designed and built the damn thing. AA just asked for it; they didn't decide on the specific engineering defects that killed two planeloads of people.

Of course every customer ever is always going to ask for products on a faster timeline at a cheaper price. That doesn't absolve the manufacturer of its responsibility to build a safe product.

Is it that the airlines told them to? Or was Boeing trying to avoid retraining in order to undercut their competitors prices?

2006 - 2011, Boeing maintains the next 737 will be a clean-sheet redesign.

In December 2010, Airbus launches the A320neo, featuring new engines (LEAP included).

In July 2011, American Airlines releases a press release [1] containing the following:

"As part of the Boeing agreement, American will take delivery of 100 aircraft from Boeing’s current 737NG family starting in 2013, including three 737-800 options that had been exercised as of July 1, 2011. American also intends to order 100 of Boeing’s expected new evolution of the 737NG, with a new engine that would offer even more significant fuel-efficiency gains over today’s models. American is pleased to be the first airline to commit to Boeing’s new 737 family offering, which is expected to provide a new level of economic efficiency and operational performance, pending final confirmation of the program by Boeing. This airplane would be powered by CFM International’s LEAP-X engine." (emphasis added)

In August 2011, Boeing announces the 737 MAX program, featuring LEAP engines.

American Airlines literally ordered a plane that didn't exist. And then Boeing tried to build it.

There are structural failings (e.g. why Boeing wasn't better positioned by investing in a redesign in the 2006+ period), but American shares a fair amount of blame for this clusterfuck.

Boeing certainly could have said "No", in which case American likely would have bought additional planes from Airbus. So American had leverage, they used it to pressure Boeing into building what they wanted, Boeing failed at delivering that, and we're here today.

[1] https://www.sec.gov/Archives/edgar/data/4515/000119312511191...

[2] https://en.m.wikipedia.org/wiki/Boeing_737_MAX

I don’t see how that proves anything about American pressuring Boeing. An equally likely scenario is that Boeing promised a plane they were developing and attempted to secure a market for it before it was completed. That alone doesn’t sound unusual. The problem is that it appears they were overly aggressive in their estimates and tried to circumvent the process with a software update.

Slightly redesigning the wings and fitting 4 engines which are sized similarly to ones in 737-400, instead of 2 large engines might solve the problem.

>Slightly redesigning the wings

>fitting 4 engines

Changing the number of engines would most likely require a total redesign of the wing.

It certainly would. Most of the rest of the airframe, too, I suspect. These things aren't modular.

Four engines are generally way more inefficient than two big ones. That's why they're being phased out. A340,B747,A380, BA146, etc.

Contrary to popular belief, quad engine planes are not inefficient [1], at least not in terms of fuel consumption (we are talking 1% to 2% difference). In terms of dispatch reliability however I think they are doing somewhat worse than twins, as they are more likely to have one engine out of service. And it not a small problem: a plane flying revenue passengers is an asset, but a plane not flying is a liability.

At least two of the planes you list are not being phased out because they were quads. The A340-300 was doing very well against the 777, however the A340-600 did very poorly against the 777-300ER. This is because the frame was too long and narrow, requiring extra reinforcement not to bend, thus ending up much heavier than the frame of the 777. The frame itself was grossly overweight, the extra engines were a rounding error.

The A380 was (and still is) extremely efficient (in terms of fuel burn per seat, at equivalent seating density), with a fuel consumption similar if not better than that of 787s or A350s, despite engines one generation older. It is, however, way way too big. In the end flying empty seats is very inefficient. But that has little to do with having 4 engines instead of 2.

The re-engined 777-9 will certainly be the most efficient plane flying when it is launched in 2021. Yet sales are lacking. It may, like the A380, turned out to be too big.

[1] https://leehamnews.com/2015/12/11/bjorns-corner-twins-or-qua...

The transition from 4 engines to 2 was accompanied by a great increase in systems reliability. Look up ETOPS for details. The trade off between safety (loss of airframe) and reliability( plane has all engines functioning ) is complex. A 2 engine aircraft is more reliable than a 4 engine aircraft, but a 4 engine aircraft can still fly safely with less than 4 engines. ETOPS made the design margins increase such that 2 engines would have identical safety and reliability.

One engine out situation is called for while planning the flight.

The entire point of the Max was to use these two new large higher-efficiency engines.

>>Software does not need to be this complicated. >>Design a damned airframe that's airworthy without needing stabilization hacks.

In effect you are stating that you understand better than the hundreds of engineers involved with this project why these design trade-offs were made. Careful with that line of thinking. You're nearly guaranteed to be wrong.

There are a host of reasons for the design in place: Efficiency, ease of implementation, familiarity with the components/technology, reliabilty of subsystems, availability of components, cost, maintenance complexity, etc. etc. etc.

And, don't forget, efficiency standards. As the whole world freaks out about CO2 and global climate change/cooling/warming, and the insta-crowd 'air travel shames' those who use these magnificent machines, it's important to realize that some of these design decisions maximizing efficiency are the hereditary descendants of cultural pressures, too.

Not to excuse any engineering f-ups, but there's a lot more to it than just 'build a simple tube with wings and an engine.'

> ...why these design trade-offs were made.

While your line of reasoning is sound in most scenarios, in this case, the why is pretty obviously written on the wall. The overriding reason why this hacky-hack software solution was slapped into the airframe was to keep from having to reclass the plane following the addition of too-large engines. Your reasons listed played into the design decisions that became MCAS, but the underlying reason as to why any of this nonsense was needed in the first place is because non-engineers forced a shitty situation on the people who actually implement these airframes.

Hacker News is frequented by engineers from multiple disciplines, including software engineering, and many of those engineers know the kinds of terrible design decisions forced by management that come about to make a quick buck. People are pissed about it due to this fact, and rightly so.

I get the impression half of HN is pissed at Boeing because they understand terrible management-driven engineering decisions.

The other half just think we're "pissed off at Boeing" and makes claims like "well, they should have just built the plane right."

And so you get engineers taking umbrage (IMHO, rightly so) for the ridiculous simplification being made by the second half.

The lovely thing about HN is that that first half is probably indeed half of the people reading; whereas this same conversation on reddit might be 10% engineers, 90% echo chamber. That's at least my experience.

The reason for the MCAS hack was not any engineering wisdom but simply allowing them to build a completely different plane that they could claim was in fact the very same plane, and then sell it to airlines as such. While the aircraft manufacturing industry was decades ago heavily safety-focused, this oversight was possible because the FAA has resigned its regulatory powers to the industry's own "self-regulation".

When the purported self-regulation of a stock market company stands between making large profits, guess which one budges?

> The reason for the MCAS hack was not any engineering wisdom but simply allowing them to build a completely different plane that they could claim was in fact the very same plane, and then sell it to airlines as such

The idea that MCAS is some kind of 737 emulator intended to keep the plane under the 737’s certification, and that without MCAS it would have needed more thorough training, is a persistent myth on HN.

The reality is that MCAS exists because without it the 737 MAX’s aerodynamics are uncertifiable no matter how much training pilots were given.

During testing the MAX was discovered to have an inverted force curve on the stick approaching a stall — as the aircraft approaches stall angle, forces flip around and it becomes easier to pull the stick back (into the stall) than push it forward (out of the stall angle).

This violates a fundamental airworthiness requirement — commercial aircraft cannot be certified as airworthy if the stick forces invert.

MCAS “solves” this by commanding the stabilizers down when it thinks the airplane approaches stall — effectively using the stabilizer to put the forces on the stick that are “missing” due to the MAX’s aerodynamics, ensuring the force curve never inverts. MCAS isn’t really allowing the plane to pretend to be anything other than “a plane that doesn’t want to stall mid-air”.

Desire to keep the MAX requiring minimal training for 737 pilots (probably) drove the concealment of MCAS mentions from the manual, but MCAS would need to exist even if the airframe had been totally recertified from the ground up.

> In effect you are stating that you understand better than the hundreds of engineers involved with this project why these design trade-offs were made.

We don't know that any engineers approved this design. It could have been completely compartmentalized and approved by management only. We'll never know what the engineers said until there's a complete investigation and the engineers are subpoenaed.

Here's what we know so far about the 'engineers': 1) The MCAS software was outsourced. 2) MCAS as originally designed and submitted to the FAA didn't have enough authority to affect the plane in the needed amount, so Boeing increased it's authority and never told the FAA.

Point to speaks to a failure in basic engineering or a deliberate deception of the FAA. A competent engineering team should have been able to calculate exactly how much effect was needed before the plane left the ground. Sounds like they were winging it on this one.

I think the key thing is how the MCAS added new failure modes that the pilots where not briefed about.

There should be way more strict user interface requirements. Including how humans are supposed to react to failures and whether that makes sense from human psychology point of view

If the MCAS had informed pilots of a potential sensor failure, then there would have been a very good chance that the pilots would have been able to land the planes safely.

> If the MCAS had informed pilots of a potential sensor failure, then there would have been a very good chance that the pilots would have been able to land the planes safely.

Perhaps on the two crashes. We don't have any data on successful landings with MCAS disabled in very adverse flying conditions.

What if the plane is unsafe to fly without proper MCAS operation?

There should be a test that results in a plane crash with 50% of the simulations with MCAS behaving properly (IE, extreme weather, low fuel, land or die). Now during that same simulation, disable MCAS randomly. Also, there seems to be potential for the MCAS to come in and out based on disagreement (loose sensor wiring, etc), and see what the crash rate is for intermittent MCAS failures.

It's clear Boeing didn't do any of this kind of testing because they would have easily caught the failure modes that caused the plane to nose-dive into the ground. Completely predictable based on the behavior of the system.

Isn’t that an appeal to authority fallacy though? With everything that’s come out about the Max, it’s clear that something is rotten in the process.

An extensive rewrite in only a few months (they expect a return to service by the end of the year), involving a whole new paradigm of 2 computers monitoring each other vs the old failover mechanism. I'm sure the aviation industry has extremely stringent coding standards, but that's just not realistic and asking for bugs (aka risks to the safety of passengers and crew).

>involving a whole new paradigm of 2 computers monitoring each other

If I read this article correctly that was already the case.

>They're rewriting the avionics from scratch.

So claims Bloomberg. Since there seems to be overwhelming consensus in this thread (that I also agree with) that it would be colossal stupid, I'd wait until we see confirmation from some other source. Bloomberg hasn't had the best record with facts or nuance in many stories recently.

Here's a Seattle Times article from Aug 1st describing the redesign, citing "three sources":


> Design a damned airframe that's airworthy without needing stabilization hacks.

The 737 Max is stable, as all commercial aircraft have to be. MCAS is not a system to take an unstable 737 and make it stable. It's a system that was meant to take a changed 737 and make it fly nearly identical to a previous 737, so pilots didn't have to get a new type rating (a very big deal for airlines).

Juan Brown, a commercial pilot and certified flight mechanic, on YouTube has a (great series of videos)[https://www.youtube.com/playlist?list=PL6SYmp3qb3uPp1DS7fDy7...] talking through the mess that is Boeing made of MCAS, including what the actual problem is.

Isn't there massive avionics commonality with all other modern Boeing AC? On the Airbus side, all types from A320 to A380 share a significant common core.

A decade is too long but I agree with your sentiment. The underlying problem with the 737 Max was commercial pressure leading to cutting corners and creating an unsafe plane (or plane with an unsafe core component) ... and now commercial pressure is again being applied on their R&D teams, which are probably scrambling because they know every day means millions in losses. That's not a great environment for quality and safety.

Boeing has to come out and say that the plane won't be back until at least 202x to take the pressure from their engineering and testing teams, so they can salvage this plane (and it may not be salvageable), instead of doing this month-to-month thing where they are hoping the FAA will sign-off on whatever latest change they put out.

Boeing makes up 8.8% of the Dow right now, the largest component. There is a lot of pressure from the finance community and the government to get this recertified to keep the stock market from going down.

That is incredibly misleading. The Dow is not representative of the entire stock market anymore. The S&P 500, which is only the 500 largest public companies by market cap, has a combined market capitalization of ~$25+ trillion. Boeing has a market cap of ~$200b, so that represent 0.8% of the total. If you include the top 3,000 public companies (i.e., the Russell 3000), the combined market capitalization is over $32 trillion. The idea that regulators at the FAA are going to let them fly a dangerous plane "because of the stock market" is absurd.

>The idea that regulators at the FAA are going to let them fly a dangerous plane "because of the stock market" is absurd.

They already let fly a dangerous plane. Given they have already done so, an explanation like 'stock market pressure' is no longer absurd.

the Icon A5 is also a certified death trap. but i guess it allows the consumer to chose risk with their wallet...

You can argue for days about your favorite index, but the Dow is printed on every financial news channel, every newspaper, every finance and trading site, every evening news show. You are naive if you think the Dow doesn't matter, or that the largest military supplier doesn't have large influence over government and finance.

Tough shit. Index funds may take a hit, but better a market crash than and airline crash.

How many people actually agree? Yes, if you ask them, the social costs and gains of each answer being given means most will disagree. But is that actually what we value more when their isn't a social cost to being honest with the 'wrong' answer? Is that what our actions speak too? I think we live in a world that puts a value on the plane crash and sees a certain level of cost from plane crashes as acceptable in the payoff is high enough. And this isn't unique to just plane crashes, look at what movie and music stars are allowed to get away with that would quickly get you or I a 20 year prison sentence.

The thing is, if they don't actually get it right, and another plane crashes, the stock is going to take a massive hit. So even if all you care about is the stock price, you have to get the fix right.

Let them take a hit, that's the best time to buy more.

I had to comment, nobody in finance pays attention to the Dow. It's a meaningless index. Look into how it is constructed if you're curious as to why.

But to your point, it makes up 0.78% of the S&P500 which is not insignificant for sure.

It makes up a far larger portion of US influence on aviation, which might bring the real push from the government.

The Dow is also extremely outdated

They may pressure FAA and maybe EASA into a certification but with other agencies (the Chinese) I think it would be more difficult.

Not to mention what happens if one of re-certified planes falls out of the air. People will refuse to board these planes.

Master-Master setups are notoriously complex. You don't just "spin one up" and things "just work".

Are the simulators run by Boeing or the FAA? Are the test pilots employed by Boeing?

What changes have been made to the regulatory framework to prevent Boeing from signing off on their own safety tests? What changes has the FAA made to bring more public transparency to the flight certification process?

When is the sentencing date for Boeing executives? They are not getting a plea deal I hope?

I'm not worried about the overly complicated flight control software or MCAS, I'm worried about the next system that will fail because nobody at this company seems to care about engineering any more.

> sentencing date for Boeing executives

Are you talking about criminal court? Call me pessimistic, but I don't think our societal arrangement is such where the people at the top pay the price. I think regulation reform is a more realistic goal.

I'm not sure the FAA is at fault here. From a big-picture perspective, the industry has never been safer.

Obviously, continuous improvement is a core part of quality, and the FAA can always learn and improve their processes, but you can't expect a regulator to shoulder the core responsibility of certifying a plane. The primary responsibility is always going to be on the manufacturer because no regulator will ever have the manpower to test and verify everything nor the deep visibility into to R&D process that a manufacturer would.

They absolutely can with the right amount of funding to maintain attractiveness.

The regulator should be adversarial; period. A well-meaning adversary, but adversarial never the less. Cutting manufacturers as much slack as has been is exactly what got us to the point we're at; a regulator that collected rubber stamped reports and only heard about things going wrong after tragedy has already struck.

It is better to have an active regulator able to intercede than to have the manufacturer coordinating everything internally, and asking for help when needed simply out of interest for removing the possibility to hide a problem discovery by never opening the floor to being questioned by the regulator.

If you tie the regulator's hands, then it isn't a regulator anymore. It's a postmortem service.

I didn't argue a regulator should be rubber stamping anything. What's with the strawman?

Sorry for the delay, but I reject the assertion that you're making here:

>but you can't expect a regulator to shoulder the core responsibility of certifying a plane. The primary responsibility is always going to be on the manufacturer because no regulator will ever have the manpower to test and verify everything nor the deep visibility into to R&D process that a manufacturer would.

In a financially incentivized market-based system where a fiduciary responsibility is built into the very underlying fabric of the corporate calculus, you cannot afford to be blind to the fact that a market actor has every reason and opportunity to stuff off every cost they can to improve their bottom line. This is why we need regulators in the first place, due to opposing optimizations between the interests of shareholders/executives, and the public.

The FAA as a regulator must be capable of requesting and having delivered any piece of information relevant to the goal of airframe certification. It is the job of the manufacturer to satisfy the regulator as to the objective safety of the plane, and it is the regulator's job to ensure nothing is left out for expediency sake. When regulator's start talking about streamlining things for the regulated, I start to get worried.

To go into more detail, no; I do not see the FAA adopting the actual physical task or logistics of testing a plane; however, I do see them as the final authority in terms of "Is the design complete" and "is your testing sufficient?"

This means that an Engineer, free of the inherent bias that comes from being dependent on the manufacturer for their paycheck, and acting in the public's interest as an external agent, needs to be as fully briefed on the entirety of the operating and physical details of every plane. It doesn't need to be the same person with it all; but the point is between the FAA as an external agency, and the manufacturer, there should be two independent agencies with enough understanding of the product that it can be demonstrated the manufacturer has done their due disclosure in informing the flying public of every facet of the aircraft's behavior that nothing like the MAX boondoggle should ever even be considered as being a reasonable course of action ever again.

You had people inside Bboeing who couldn't understand why MCAS was the way it was. Given that, it is evident that the most important stakeholders in being fully informed (buyers and operators) as a consequence were also not informed before regulators cut Boeing loose to sell on the market.

In 2018, legislation was passed that made it even more difficult for the FAA to exercise it's purported authority so long as a corporate representative assured them the issue was being handled internally.

I do expect anyone in an oversight position to be capable of observing things within their purview; and in terms of evaluating designs, the tangible nature of the principles and forces involved with aviation should be conducive to clear communication and reproducibility between the manufacturer and the regulator. The difference to a business in a functioning regulatory regime is that the manufacturer should see it's job as revealing new ground to a regulator, and leveraging the regulator as the source of of friction that peels away any uncertainty from the design. This can only happen in an environment where a "no more secrets" approach to business is maintained.

Yes, I agree that the aviation industry hiding behind the streak of years of no accidents has contributed to how bad Boeing became. They were using that statistic as a shield and a hammer to justify further deregulation.

This is a great example of how looking at the world through the prism of stats really limits your understanding of what's actually happening.

That is a cynical way of interpreting the situation, and let's be clear, you have no basis for claiming this.

There has been plenty of cynical reporting and editorializing that has made this very argument, that in fact Boeing and the industry more widely has been hiding behind the safety streak up until the moment of the first MAX crash.

The fact that you find my post novel and shocking just shows you're not reading very much about aviation in the past few months.

Abstracting out of planes to software on general - this is what happens when your testing surface dramatically expands. You can have a product that has been working just fine, generally, but when you start adding tests for new situations, you can suddenly get a LOT more tests....with most of them failing.

TDD advocates (and I'm a fan) will be feeling smug, be that doesnt apply here - the issue is not the initial tests, but explicitly tests after the fact, on criteria that werent in the initial tests. Be it by oversight or deliberate choice, TDD is in the same boat here.

All of which underlines how hard complex software can be. Boeing made lots of mistakes, and many of us might recall happier examples from our past (new criteria, but a well-written suite passes it all with minimal effort), but such examples are selection bias - if we exclude the code we know is a mess, the remaining mix of did-well and did-poorly code looked GOOD before. (Here I'm generalizing from my experience and the war stories I've heard)

Which brings us back to the Waterfall vs Agile issue. We know that we generally stink at anticipating all the requirements. We also know that the better we do at anticipating those requirements the less likely we are to have a sudden spec change derail us (not because we can prevent the spec change, but because our code tends to work)

Anyone asserting that such problems are simple to resolve hasn't worked on enough such problems. We are learning, but this field is still in its infancy and we've not even finished understanding some of the earliest principles the pioneers in the industry laid out.

They say the MAX issues might delay the introduction of the 797, which I find rather strange. The 797 is the plane Boeing should have made instead of butchering the design of the 737...it's the right size and has the ground clearance necessary for the large efficient engines they jammed onto the 737...in other words it's perfect for mid-capacity long range point to point flights between secondary cities.

I say get the 797 out ASAP.

Considering that MCAS is an added ‘feature’ to avoid stall, you would think they could just remove it and be done. After all, it wasn’t part of the original 737.

Sadly, from my limited understanding, due to the engine location change on the airframe, the plane naturally pitches up... making it necessary.

One thing seems clear to the external observer, this is a 737 in name only.

due to the engine location change on the airframe, the plane naturally pitches up... making [MCAS] necessary.

Most jet airliners have a pitch up tendency. The problem with the MAX is two related issues...

First, FAA has a requirement that stick pressure must follow certain patterns - namely, the harder you pull/push the stick, the more the plane reacts. Due to engine placement, the MAX did not meet this requirement.

Second, Boeing needed to create the MAX with minimal (approaching zero) re-training of pilots.

MCAS was supposed to meet both needs - by having MCAS adjust the stabilizer, the stick behavior/feel came back within requirements. And because it was all automatic, the pilots didn't need retraining in the simulator. Correcting the stick feel without a system like MCAS would have required more substantial changes to the plane, which would have likely required simulator time, which was likely a deal-breaker for some airlines.

Well, it IS a 737, that is part of the problem.

The 737 has a feature almost no other new plane have: you can order it with a internal staircase so people can leave the plane without a specialized airport terminal.

This feature requires the plane to keep its current overall shape, otherwise the staircase thingy would not work.

So to keep this feature, while putting a bigger engine on the plane, they had to move the engine forward, and change its shape too, otherwise the engine wouldn't fit between the wing and the ground.

Then THAT caused the necessity of MCAS.

So long story short: wanting to keep backward compatiblity with the 737 staircase, led to the engine hack, that then to maintain backward compatiblity with the handling led to the MCAS hack...

So, it IS a 737, in the sense they kept the staircase and to do that ended needing the MCAS...

The other option would have been abandon the staircase entirely and make a taller plane, this would allow a bigger engine with no handling changes, but although this would remain a 737 from handling perspective, it would not be a 737 from the airports perspective: it would need to remodel the airports to install bridges or purchase of ladder trucks.

The other option would have been abandon the staircase entirely and make a taller plane, this would allow a bigger engine with no handling changes, but although this would remain a 737 from handling perspective, it would not be a 737 from the airports perspective: it would need to remodel the airports to install bridges or purchase of ladder trucks.

No that would not be the major problem. Specifically since practically any airport nowadays has the infrastructure to service such a plane. This was quite different 50 years ago.

The reason why they couldn't have heightened it was that a new type certification would have been required and Boeing wanted to avoid exactly that at all cost.

Even at the cost of 346 dead people.

Just skipping the staircase it not enough. To make the plane taller it needs a new landing gear. But that doesn't fit into the existing gear wells. So now you need to redesign the gear wells. But the gear nearly touch in the middle already. So you neet to make them taller, and attach them to the wing further out. Which the Wing can' take. So you need to redesign the wing. At this point you better just admit that the 737 design from 1964 is at the end of its life.

TIL: The airstairs are still an option of most (all?) 737 variants. RyanAir appears to have them on part of the fleet. And many BBJ have them, as not all general aviation terminals have jetways.

I assumed the stairs were not longer installed, and the height of the 737 was just an anachronism.

Video for those that haven't seen them: https://www.youtube.com/watch?v=unZeusTrDX4

The 737 MAX 10 has taller (extending) landing gear. Not sure if the airstairs will still be an option.


That landing gear only extends when the nose gear is off the ground. It's really just to prevent a tail strike. An alternate choice would be a little extra wheel up on the tail, as was done with the Concorde.

It was more than a staircase issue- they were trying to avoid time-consuming and costly pilot-retraining.

The Homer car comes to mind whenever these kinds of convoluted and unchecked engineering adventures have the light shone on them.

It is my understanding it is only necessary to make the Max 'compatible' with the 737 so nobody needed to be retrained and recertified.

My understanding is that, without MCAS or some other design change to fix its unacceptble handling characteristics near the stall, the 737 MAX would have been uncertifiable even as a new type. It is the way that Boeing went about implementing that fix, in an attempt to avoid an additional training requirement, that has caused all the trouble.

Let's be clear here, though: it's the airlines who refuse to buy an aircraft with additional training requirements. Pilots can only fly one aircraft type, and retraining to a different type is an expensive proposition. If Boeing could make an aircraft that fit under the 737 type, they really had no choice but to do it.

They also probably could have succeeded, too, if not for other systemic problems within the company.

It is not clear to me that they probably could have succeeded in satisfying all these conflicting goals, if not for other systemic problems in the company. Even now, with (one hopes) everything in the open and the systemic problems pushed aside, they are having trouble getting it done.

You say Boeing had no choice, but one choice was to acknowledge the problems, aim to keep the type certification with additional training, and negotiate with its potential customers on that basis. There is no law of mankind or nature that says Boeing is entitled to a certain number of sales at a particular profit.

Yes, that was the gamble. But as others have pointed out, instead of actually ‘updating’ the plane Boeing changed it materially. Calling it a 737 was a stretch at best.

Yeah they're gonna have to fix the MCAS - duck-taping the engines on a bit further back isn't an option.

If I understand correctly "fixing" MCAS would be quick but audit found out more problems and Boeing started basically a refactoring of their computer architecture (including hardware changes) which itself takes a lot of time and didn't pass new audit yet.

Yes, that's the official explanation. But these problems (such as gamma rays scrambling the flight computer, causing the plane to aggressively dive) are ostensibly not new, or related to MCAS. The issue is not just that gamma rays theoretically – and MCAS, in real-life and quite tragically – cause the plane to dive, but that Boeing overestimated pilots' ability to react to the underlying systems error. I think it seems suspicious Boeing only now decides "Oh, looks like we should also modernize the 737 computers", while asserting that MCAS itself was still a great idea.

Didn't I read from another hacker news poster that the software engineering and design was outsourced? Anyone know?


They didn't work on the Flight computer supposedly. Just the Flight displays.

There was a Bloomberg article with more detail.


Outsourcing is a symptom of the real issue. I don't think outsourcing necessarily causes these problems. Subjugating engineering to a lower class and elevating politics/accounting doesn't really generate great engineered products, go figure.

As an engineer in the defense/aero industry, this is so true. Legal, HR, and IT run everything, with the engineering staff shat upon.

In the article they mention that they outsourced at least some of it.

“ As Boeing and the subcontractor that supplied the flight-control computer, the United Technologies Corp. division Collins Aerospace Systems, worked through these changes, it has at times created tension.”

It is extremely common that various parts of design and manufacture of aircraft are "outsourced". The airframe manufacturer is essentially never the manufacturer for whole subsystems and there is a large subcontracting network.

This is an entirely normal and long-standing practice.

Didn't say it wasn't, I was just answering GP's question :).

Probably multiple levels of outsourcing. Boeing > Collins > Indian for $9/hr.

There are also issues with 737 NG[1]:

> At least three Ryanair Boeing 737s have been grounded due to cracks between the wing and fuselage but this was not disclosed to the public, the Guardian can reveal.

> The budget Irish airline is the latest to be affected by faults in the “pickle fork” structure, which has sparked an urgent grounding of 50 planes globally since 3 October.

[1] https://www.theguardian.com/business/2019/nov/06/boeing-737-...

But that's metallurgical failure, not software, and it's something that's showing up in use almost decades later.

I'm saying this to highlight that what you're pointing out, while not desired, is kinda expected. That's the point of the routine inspection and maintenance - to catch these.

It's not the same as the conversation we're having under the OP: catastrophic failure due to bad assumptions in software (in this case, memory safety)

> I'm saying this to highlight that what you're pointing out, while not desired, is kinda expected.

If it was expected, it wouldn't be newsworthy. And metallurgical issues in Boeing planes can be as critical as software issues:


It's not newsworthy. That's why the recent metallurgical stress topic regarding Ryanair's jets rapidly died from the news cycle - because to some extent, it's expected, and it's why maintenance processes exist. A quick search on Google News yielded exactly one recent result, and that was about Ryanair's response to the Guardian article: https://www.irishtimes.com/business/transport-and-tourism/ry...

> And metallurgical issues in Boeing planes can be as critical as software issues:

This much is obvious, and it's exactly why the maintenance exists. Crashes happen; the precursors are baked right into new processes and procedures. When said processes pick up on similar symptoms in the future, it might be in the news for a day or two, and then it disappears because it's expected.

The precise problem with the MAX is that there's an entire body of knowledge around the MCAS and other automation included with the plane that was never shared, which meant that unlike metallurgical issues which in many cases are largely unforeseen, Boeing's problems here were entirely preventable.

Items that stay in the news for a while tend to be the novel things.

> This much is obvious, and it's exactly why the maintenance exists. Crashes happen; the precursors are baked right into new processes and procedures. When said processes pick up on similar symptoms in the future, it might be in the news for a day or two, and then it disappears because it's expected.

At least on three occasions, multiple people have died or were injured because 737 NG planes developed serious cracks after exactly 8 years in service. Nothing like this is expected or considered to be normal. Therefore, when 50 planes of the same type are urgently grounded by multiple airlines in a very short period of time, it's considered to be newsworthy.

> The precise problem with the MAX is that there's an entire body of knowledge around the MCAS and other automation included with the plane that was never shared, which meant that unlike metallurgical issues which in many cases are largely unforeseen, Boeing's problems here were entirely preventable.

The previous metallurgical issues in 737 NG were also entirely preventable:


That makes the real source of any new issues with 737 NG questionable.

My point is, that MAX and MCAS is not the first instance of Boeing's negligence. And new things that went wrong with 737 NG might still be discovered.

That is not nearly the same issue. It is a relatively tiny few airplanes with very high numbers of legs, i.e., landings, which is the most likely culprit (exacerbated by hard landings of inexperienced, low cost pilots). And most of the grounded planes seem to just meet lowered inspection thresholds; many of which will simply enter service again after inspection.

I believe the Max will never fly again, at least not outside the US. The FAA might approve the final "fixed" design based on political pressure, but what international regulator will do that? So then you have a plane design that only the US considers safe.

The flying public in the US will naturally start asking questions. I guess airlines could offer discounts for Max flights. Then we'll see what level of discount is sufficient to get people on a plane that historically had a strong predisposition to nose-dive.

This may be cynical, but I suspect this issue will be drawn out way longer, because Airbus (including maybe the nascent Chinese airline industry) will ... logically ... take any and all advantage of this self-inflicted flesh wound to keep Boeing on the ropes by at least shining the brightest of bright lights on any and all flaws they can find.

This will, as the article implies, come in combination with the justified and rightful concern that Boeing is and will continue and increasingly get more desperate and frantic to "fix" things (literally and, likely increasingly figuratively), as far greater testing and auditing requirements are placed on recertification.

I don't know about everyone else here, but I sure as heck am going to be quite careful not to fly on any MAX for quite some time if they are even ever re-certified again. The replacement for the 737, which the MAX was supposed to delay, was not scheduled to reach market until the 2030 timeframe (which could mean 2030 or 2039). I think it is anyone's guess whether Boeing has the resources to drastically accelerate that timeframe or the MAX village fire is draining all resources and it may delay that 2030 timeframe.

Please convince me otherwise, but I could see this MAX issue essentially crushing Boeing as it eats away at many different aspects of the enterprise over time. How long can you keep the concerns at bay and run on inertia? I think there may be hell to pay next year if this isn't really an "easy" issue that just takes some time and Boeing can scathe by. Maybe someone with deeper insight into Boeing's operations can substantiate why I am totally off base or … hopefully not … validate that my concerns are not unfounded.

This article is incoherent:

> “It’s really complicated,” John Hansman, an aeronautics and astronautics professor at the Massachusetts Institute of Technology who is not involved in the repair, said of revising aircraft software. “It totally makes sense why it’s taking longer.”

But then...

> While the fix became more complex and politically charged after the second accident -- the crash of an Ethiopian Airlines jet on March 10 -- the changes to MCAS remained self-contained and relatively simple. “I could have a bunch of graduate students and rewrite MCAS in a couple of days and be done,” Hansman said.

So is it "really complicated" or "relatively simple" to the point where "a bunch of graduate students [could] rewrite MCAS in a couple of days and be done"? It can't be both.

The impression I got from the article is that they are bringing all (safety-critical?) control-system software up to 21st-century standards of redundancy.

It was not clear to me whether it would be possible, in theory, to fix MCAS alone, and if so, who decided that it was necessary to go beyond that - Boeing? The FAA? Other countries' regulators? The airlines? None of these entities want another round of "you didn't fix it right", though their perception of risk and tolerance of it may vary.

AFAIK from this article, none of the earlier generations of 737s will be so modified, even though their systems are presumably very similar, in all non-MCAS aspects, to what the MAX first flew with. That's quite arguably a rational choice, once you recognize that there is a cost - benefit tradeoff even in air-travel safety.

This is a sort of technical debt situation. Ironically, if Boeing had not tried so hard to game the regulations, MAXes could be flying today with less fault-tolerance than they will have when they return.

Read it as patching MCAS would be the grad students over the weekend fix, and the really hard stuff as redesigning avionics logic.

OK, fair enough, but the complete redesign is only necessary because the simple patch didn't work, so the grad-student-in-a-couple-of-days comment is vacuous.

They're talking about two different changes:

"bunch of grad students"="patch MCAS to not auto-crash"

"really complicated"="redesign entire compute architecture to be multi-master and failure tolerant"

"really complicated"="make an architecture usually done with three computers work with just two which were incidentally architectured and dimensioned for a different job. Get this done in a lot less time than usual and btw. you don't have the usual freedom to discuss constraints one usually has at this point.".

Possibly stupid question but will they be able to apply these fixes remotely? Or will Boeing have to fly engineers / mechanics to every grounded 737 Max in the world?

Commercial aircraft have a large support staff that 'touch' the aircraft ever day (week). Certified mechanics and technicians. A software update is something they do on a regular (yearly maybe()) basis. By following instructions provided by the OEM. Upgrading hardware is also done by the customer. People are trained and qualified to do this work all over the world.

I could well image a middle ground for software updates: patching needs to be done by ground crew (and while not in flight, obviously), but not by Boeing engineers, just by regular maintenance crew.

But the article reads as if new hardware was necessary as well, which might change the picture.

> Boeing decided to make the two systems monitor each other so that each computer can halt an erroneous action by the other

I thought the whole point was that you needed 3 computers monitoring each other because if you have only two, you can't tell which one is faulty and which one is correct. How can they make it work here ?

> simulated what would happen if gamma rays ...

So now I am curious. How do other airliners (newer Boeings, Airbus, etc) fare when subjected to the gamma ray test? I would like some context.

The article could have used the industry term, Single-Event Upset, rather than use the more eye catching gamma ray term over and over. It's a failure mode that aircraft and spacecraft have to deal with.

You keep CRCs of data in critical areas of memory and constantly check them to make sure you are only processing valid data, or use equivalent of ECC memory.

I'd be fine if they just junked the whole fleet.

In fairness, it's a good sign that they're catching such potential glitches in the simulator.

I’m a rational person who enjoys flying, and I just never want to fly on a Max.

Based on one story from Bloomberg? That might be a tad premature - they never did address concerns with their "reporting" on the China server spying issue.

It’s not one story — it’s a pattern of stories from numerous outlets for over a year now.

If it was just bloomberg then I'd be with you on this one, but trustworthy news agencies have also published similar stories.

Aircraft should be required to be fully recertified by the FAA.

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | DMCA | Apply to YC | Contact