Engineering and Software Disasters

Lead Summary

When the Boeing 737 MAX fell from the sky twice in five months, killing 346 people, investigators did not find a single broken part. The MCAS flight-control software was working exactly as designed — it was the design itself, the organizational choices that produced it, the regulatory machinery that allowed it, and the informational concealment that hid it from pilots that collectively caused the crashes. The same pattern repeats across the canonical engineering disasters of the past forty years: the Therac-25 radiation overdoses, the Ariane 5 explosion, and the 2024 CrowdStrike global outage. In each case, technically competent people working inside recognizable institutions produced catastrophic outcomes through the systemic interaction of design choices, schedule pressure, regulatory failure, and organizational culture.

This article surveys what that body of disaster is, what theoretical frameworks have emerged to explain it, and what remains genuinely contested. It also examines who bears the costs — because disasters do not distribute their consequences evenly.

---

The Case Studies

Therac-25: Software Without Safety Discipline (1985–1987)

The Therac-25 was a computer-controlled radiation therapy machine built by Atomic Energy of Canada Limited. Between June 1985 and January 1987, it delivered massive radiation overdoses in six known accidents — described as the worst series of radiation incidents in 35 years of medical accelerator history — causing deaths and severe injuries. The technical core of the failure was a race condition: the hardware-control task and the operator-interface task both read and modified the same memory location without synchronization primitives. When an operator corrected treatment settings within a roughly eight-second window, the hardware could act on a different value than the interface had set, activating a high-power electron beam without the required physical apparatus in place.

A second vulnerability complemented this: a flag variable tracking system state was implemented by incrementing it rather than setting it to a fixed non-zero value. Arithmetic overflow caused the flag to wrap to zero, bypassing safety-check logic entirely. Neither bug would have been lethal in the Therac-20 predecessor, because that machine relied on hardware interlocks — physical energy limiters and turntable sensors that made unsafe activation mechanically impossible. The Therac-25 removed those interlocks and concentrated all safety responsibility into software, without validating that the software was adequate to carry that responsibility alone.

Nancy Leveson's investigation emphasized that the accidents were not caused by the race condition alone. Multiple systemic failures converged: cryptic error messages that gave operators no indication of hazard; a single-keystroke mechanism to resume treatment after errors; failure of dose-rate monitors when the turntable was mispositioned; absence of independent interlocks; and organizational failure to investigate early reports of unexplained injuries. AECL's initial response was to dismiss operator complaints as user error rather than investigate the software. By the time the machines were shut down in 1987, additional patients had been harmed that more rigorous incident response would have protected.

The post-Therac-25 regulatory response produced the IEC 62304 standard for medical device software development, and the FDA revised its guidelines to move beyond substantial equivalence toward explicit software validation requirements. The case remains a canonical teaching example precisely because attributing the outcome to "operator error" would have obscured every systemic factor that actually caused the deaths.

---

Ariane 5 Flight 501: The Cost of Unvalidated Reuse (June 4, 1996)

The first operational launch of the Ariane 5 rocket ended 37 seconds after liftoff when the vehicle disintegrated, destroying a payload valued at approximately $370 million. The cause was a 64-to-16-bit floating-point conversion in the inertial reference system (SRI) — code reused unchanged from Ariane 4 that assumed horizontal velocity would remain within Ariane 4's lower-performance flight envelope. Ariane 5's higher acceleration caused the velocity variable to exceed the representable range, generating an arithmetic overflow exception.

When the primary SRI (SRI 2) shut down on the exception, the on-board computer switched to the backup unit (SRI 1) — which had failed from the identical overflow condition 72 milliseconds earlier. Redundant systems sharing identical code and operating under identical conditions cannot provide independent protection against systematic defects. Worse, both units transmitted diagnostic data rather than flight data when they failed, and the on-board computer had no mechanism to distinguish diagnostic from flight data. It acted on the diagnostic output as if it were valid attitude information and issued control commands that tore the rocket apart.

The Ariane 5 failure illustrates three compounding design flaws: (1) software reused outside its validated operational envelope, (2) redundant systems sharing the defect that defeats them both, and (3) an interface boundary that propagates diagnostic errors as operational commands.

The alignment code that triggered the overflow was not even needed for Ariane 5 — it had been necessary for Ariane 4's ground calibration procedure but served no function in Ariane 5's operational flow. The system included vestigial code that had no purpose in the new deployment context but retained its operational hazards.

The Ariane 5 case established several principles now foundational to aerospace software engineering: software components must be explicitly revalidated for new operational contexts; redundant systems must use diversified implementations to prevent correlated failure; and interface boundaries between subsystems must include defensive filtering to prevent one unit's failure mode from propagating as valid commands to another.

---

Boeing 737 MAX MCAS: When Concealment Kills (2018–2019)

On October 29, 2018, Lion Air Flight JT610 crashed into the Java Sea shortly after takeoff from Jakarta, killing all 189 people aboard. On March 10, 2019, Ethiopian Airlines Flight ET302 crashed near Addis Ababa, killing all 157 aboard. Both crashes were caused by the Maneuvering Characteristics Augmentation System (MCAS), a flight-control software feature that neither pilots nor airlines had been told existed.

MCAS was introduced to compensate for handling differences created by moving the 737's larger LEAP engines forward and higher on the wing. Its single-sensor design violated fundamental aviation safety requirements for critical control inputs: if the one sensor feeding MCAS malfunctioned, the system activated persistently on false data with no cross-check. Aviation certification practices under DO-178C make single-point-of-failure reliance for critical control functions formally indefensible. The design choice was driven by cost and schedule pressure, not technical necessity.

The concealment compounded the technical failure. Boeing deliberately excluded MCAS from pilot manuals and FAA certification documentation to avoid triggering full simulator training requirements, which would have imposed significant costs on airlines and undermined Boeing's competitive positioning against the Airbus A320neo. Boeing's argument was that the 737 MAX was essentially an updated NG variant requiring no new type rating — a claim that held only if MCAS was invisible to pilots. When MCAS activated on both fatal flights, the crews did not recognize the system or understand its behavior.

The regulatory dimension is equally important. The FAA's Organization Designation Authorization (ODA) program delegated substantial certification activities to Boeing-employed engineers, who were both responsible for validating designs and employed by the manufacturer those designs would profit. The FAA's JATR final report documented that the FAA had "inadequate awareness of the MCAS function" prior to the crashes and that ODA unit members were subject to undue pressure to approve certification activities. George Stigler's 1971 theory of regulatory capture predicts exactly this: information asymmetry between manufacturer and regulator enables manufacturers to shape certification to industry interests, and delegating certification to manufacturer-employed engineers concentrates both the information and the incentive misalignment in the same place.

Marver Bernstein's life-cycle theory adds a temporal dimension: regulatory agencies drift toward capture over years of industry consultation, with capture deepening until a disaster temporarily restores public pressure and forces reform. The 737 MAX crashes triggered precisely this reset. Public Law 116-260 (December 2020) mandated FAA recentralization of certification authority, prohibited undue pressure on ODA certifiers, and required manufacturers to implement Safety Management Systems under FAA oversight.

The organizational suppression of dissent is also documented. Boeing engineer Ed Pierson formally warned leadership in writing before both crashes about unsafe conditions in the production environment. His warnings were ignored. Boeing quality engineer John Barnett, who raised concerns about manufacturing defects in the 787 Dreamliner, faced systematic retaliation including inaccurate performance reviews, shift reassignments, and harassment — a manager explicitly stating "I am going to push you until you break". Barnett died in March 2024. The suppression of engineering voice before disasters is a recurrent pattern across Boeing, Challenger, Theranos, Volkswagen, Wells Fargo, and Meta — not an isolated pathology but a systematic organizational failure mode.

---

CrowdStrike (July 2024): Normal Accidents in Modern Infrastructure

On July 19, 2024, a faulty content update from cybersecurity firm CrowdStrike caused 8.5 million Windows devices to crash globally in what became the largest IT outage in history. The technical cause was a field-count mismatch in a rapid-response content update: the IPC Template Type defined 21 input fields, but the sensor code provided only 20. The Content Validator component failed to detect this mismatch, allowing the faulty configuration to pass integrity checks and be distributed globally as a single release event.

CrowdStrike Falcon runs as a kernel-level driver with full system access — a structural single point of failure. When a third-party component executes at kernel privilege on system startup without isolation boundaries, one bug propagates instantly across all systems running the software. The outage exemplifies what Charles Perrow's Normal Accident Theory identifies as the inevitable consequence of interactive complexity and tight coupling: the validator interacted with sensor code field definitions in a way not anticipated by the system's designers, and tight coupling — kernel-level mandatory execution — meant there was no isolation boundary to contain the failure.

The absence of graduated deployment was equally critical. Standard safety deployment pipelines — internal dogfood, canary testing to a small monitored subset, controlled rollout, feedback loops — exist precisely to catch defects before full release. CrowdStrike distributed the faulty update to its entire global installed base simultaneously, bypassing all of these containment mechanisms. The result was a failure whose blast radius was maximized by the deployment architecture itself.

---

Theoretical Frameworks

Normal Accident Theory: Failure as System Property

Charles Perrow's Normal Accidents (1984) argues that complex, tightly-coupled sociotechnical systems will inevitably experience catastrophic failures regardless of organizational safeguards. A system exhibits interactive complexity when its components interact in ways that are unexpected, invisible, or incomprehensible even to expert operators. Tight coupling means components are critically interdependent with minimal recovery time between one component's failure and its propagation to others. When both properties are present, accidents are not aberrations but built into the system's design — they are "normal."

Perrow's most counterintuitive insight is the safety paradox: adding safety measures to an already complex system increases complexity, potentially opening new failure categories rather than closing old ones. This challenges the standard engineering assumption that marginal safety improvements always reduce risk.

High Reliability Organizations: Failure as Preventable

Karlene Roberts and colleagues at UC Berkeley, studying nuclear aircraft carriers, air traffic control systems, and nuclear power plants, identified organizations that achieved sustained near-zero-failure performance over decades in precisely the complex and hazardous conditions where Perrow predicted failure. HRO theory defines key organizational characteristics: strategic prioritization of safety, operational redundancy, decentralized decision-making, continuous simulation training, and safety cultures generating broad vigilance.

The empirical anchor is significant: the US Navy's nuclear-powered fleet has accumulated over 6,200 reactor-years across 526 nuclear reactor cores without a single radiological incident over 50+ years of continuous operation — in systems that are both interactively complex and tightly coupled.

Karl Weick and Kathleen Sutcliffe's 2001 "Managing the Unexpected" formalized five hallmarks of HRO mindfulness: preoccupation with failure, reluctance to simplify, sensitivity to operations, commitment to resilience, and deference to expertise. The fifth — deference to expertise — describes the dynamic migration of decision authority toward personnel with relevant knowledge during emergencies, regardless of hierarchical rank. This structural flexibility distinguishes HROs from rigidly centralized organizations where authority fails to track expertise under pressure.

The Theoretical Tension

NAT and HRO make fundamentally incompatible predictions about the same systems: NAT predicts inevitable catastrophic failure; HRO predicts that specific organizational practices can sustain near-zero failure. This cannot be resolved by applying both frameworks simultaneously — they make mutually exclusive causal claims about what is possible.

One proposed resolution is temporal: NAT addresses long-run inevitability while HRO addresses short-to-medium-run prevention capacity. Under this reading the frameworks may be complementary: HRO practices can defer and reduce failures without eliminating them over infinite time horizons in sufficiently complex systems.

Both frameworks also have non-falsifiability problems: NAT can retrospectively explain any accident as predicted complexity-coupling interaction, while explaining accident-free periods by redefining system boundaries; HRO can attribute any high-reliability outcome to its organizational characteristics while using those same characteristics to explain failures when they occur. Neither framework has systematic empirical comparison across a controlled set of systems.

Fig 1

---

The Swiss Cheese Model and STAMP

James Reason's Swiss Cheese Model represents accidents as the result of multiple defense layers — each with "holes" representing latent weaknesses — aligning so that a failure trajectory passes through all of them. The model has four levels: organizational influences, unsafe supervision, preconditions for unsafe acts, and the unsafe act itself. It rejects single-cause attribution and has been widely adopted across aviation, healthcare, and industrial safety.

Nancy Leveson's STAMP (System-Theoretic Accident Model and Processes) departs from Swiss Cheese in a fundamental way. Rather than treating accidents as failures passing through weakened defenses, STAMP treats accidents as control problems: breakdowns in hierarchical safety control structures where safety constraints were violated. STAMP emphasizes that accidents can occur when all components are individually functioning correctly — the failure lies in dysfunctional interactions and inadequate enforcement of constraints, not in component breakdown. The investigative question shifts from "what failed?" to "how was control of safety lost?"

In complex systems, linear causation models (fault trees, Five Whys) are inadequate. They assume a single root cause at the end of a linear chain, an assumption appropriate only for simple electromechanical systems. The Therac-25 had no single root cause. The 737 MAX had no single root cause. Each was produced by an interaction of technical, organizational, regulatory, and commercial factors that linear models cannot represent.

---

Human Error Versus Systemic Causation

The most consequential interpretive choice in accident investigation is whether to treat human error as an explanation or as a symptom. The "person approach" focuses on errant individuals and treats errors as moral failings; the system approach recognizes that most errors are consequences of the system within which people work — its design, procedures, information environment, and organizational pressures.

James Reason's foundational distinction between active failures and latent conditions formalizes this. Active failures are the unsafe acts of operators in direct contact with the system — the proximate trigger. Latent conditions are organizational, managerial, and design defects that exist dormantly until activated by some local trigger. When a Therac-25 operator typed quickly enough to trigger a race condition, the "error" was the operator's action; the causes were the absent synchronization primitives, the removed hardware interlocks, the cryptic error messages, the single-keystroke resume, and the organizational failure to investigate prior incident reports.

Sidney Dekker's "new view" of human error extends this to investigate how organizational pressures, knowledge asymmetries, and system design constraints shaped decision-making at the time of the incident — not as viewed retrospectively through outcome knowledge, but as experienced prospectively by people who did not yet know what would happen.

Hindsight Bias

Hindsight bias is the systematic distortion in post-accident investigation by which knowing the outcome makes the outcome seem obviously predictable. Fischhoff's 1975 research demonstrated that outcome knowledge irreversibly alters memory representations, causing investigators to misremember the decision-making environment as it appeared to those involved at the time. In accident investigations, this leads to overconfident attribution of blame for decisions that were locally reasonable under the information available — while masking the systemic factors that made those local decisions consequential.

Just Culture

Just Culture, developed by Sidney Dekker, operationalizes the human-error-vs-systemic distinction for organizational practice. It rejects both pure blame culture and pure no-blame culture, instead distinguishing between human mistakes (systemic, managed by system redesign), at-risk behaviors (poor choices amenable to coaching), and reckless actions (deliberate disregard warranting individual accountability). The investigation question shifts from "who caused the problem?" to "what went wrong?" Just Culture creates the psychological safety required for people to report near-misses and anomalies without fear of punishment — the data source on which all other learning depends.

Blameless postmortem practice originated in aviation and healthcare — where punishing mistakes was recognized as driving problems underground — and has since been adopted by software SRE practice at Google, PagerDuty, and elsewhere as standard incident management.

---

Normalization of Deviance

Diana Vaughan's sociological research on the 1986 Challenger disaster produced the concept of normalization of deviance: the organizational process by which deviations from safety protocols become culturally normalized over time when they do not produce immediate catastrophic consequences. NASA engineers at Morton Thiokol identified that O-rings posed structural risk in cold temperatures. Over multiple launches that returned safely despite this known flaw, the deviation became accepted as normal risk. When launch temperature fell to 36°F — well below the identified 53°F threshold — production pressure led managers to override engineer warnings and proceed. Roger Boisjoly's memos documenting the O-ring risk dating from July 1985 were on record. He was excluded from the launch decision.

The same pattern of safety culture erosion through production and cost pressure appears in the 737 MAX: Boeing's key performance metric was meeting tight development deadlines, which displaced quality and safety as organizational priorities through accumulated decisions favoring schedule compliance. This is not a technical problem. It is an organizational incentive problem — and it generalizes across Challenger, Columbia, the 737 MAX, Theranos, Volkswagen's defeat devices, and Wells Fargo's account fraud.

---

Regulatory Capture in Certification

Regulatory capture — the systematic shaping of regulatory agencies to serve industry interests rather than public welfare — is a structural risk wherever regulators depend on industry for information, personnel, and technical expertise. George Stigler's 1971 theory models capture as rational: industries face concentrated, immediate losses from stringent regulation; the public faces diffuse, deferred benefits from that same regulation. This asymmetry systematically tilts regulatory outcomes toward industry interests.

The 737 MAX certification failure is a textbook case. Boeing's ODA engineers were financially employed by Boeing but legally responsible for certifying Boeing designs. Personnel moved between Boeing and FAA positions, reducing regulatory independence through networks of mutual interest and career incentive. Boeing withheld critical information — the existence of MCAS — from FAA certification documentation, exploiting the information asymmetry endemic to delegation models. The FAA's own technical advisory board found that undue pressure had been exerted on ODA unit members.

Bernstein's life-cycle theory of regulatory agencies predicts that this capture is not exceptional but temporally predictable: agencies drift toward industry interests over years of consultation and revolving-door personnel movement, until a disaster temporarily resets public pressure and forces legislative intervention. The 737 MAX crashes triggered the reset. Public Law 116-260 mandated FAA recentralization of certification authority and made exertion of undue pressure on ODA unit members unlawful.

---

Recent Failure Modes: Autonomous Systems

Contemporary autonomous systems introduce failure modes that differ structurally from traditional software failures. Autonomous vehicle perception systems exhibit brittleness under distributional shift: when real-world environmental conditions deviate from training data, perception models malfunction in unpredictable ways. Documented failures include misidentification of articulated-bus components as separate obstacles, lidar misinterpretation of exhaust fumes as solid objects, and failures to detect pedestrians in rain, fog, or urban canyon conditions where GPS signals are blocked.

Sensor fusion failures arise when multiple sensor modalities provide contradictory or ambiguous information and fusion algorithms must decide under uncertainty. Radar exhibits angular ambiguities from finite spatial sampling that are indistinguishable from genuine obstacles without independent confirmation from other sensors. The integration of multimodal sensor failures into a unified perception model under real-time constraints remains an open research problem; current systems lack robust algorithms for detecting and recovering from sensor-specific failures in real-time.

The CrowdStrike outage also illustrates how the architecture of update distribution is itself a safety decision. Simultaneous global rollout of kernel-level software without phased deployment is a design choice that maximizes both the speed of a successful update and the blast radius of a defective one.

---

Controversies and Debates

Root Cause Analysis Oversimplification

The Five Whys method — the most widely used root cause analysis technique — suffers from a fundamental oversimplification problem: different teams using the method on the same problem produce different causes, suggesting it captures the first plausible narrative rather than systemic root causation. The five-iteration depth limit is arbitrary. The method enables unconscious steering toward convenient explanations, particularly when investigators lack access to complete technical evidence or when the methodological pressure is toward narrative closure.

Investigator skill matters more than the method itself: empirical studies show that trained facilitators produce significantly better problem-solving outcomes than untrained teams using the same technique. This reveals a critical limitation: RCA as a method is permeable to investigator bias, expertise, and incentive. Effective root cause analysis of complex systems requires causal models adequate to non-linear interaction — which the Five Whys is not.

Safety-II and Learning from Success

Erik Hollnagel's Safety-II framework extends the safety science agenda beyond failure analysis. Where Safety-I asks how to reduce failures and accidents, Safety-II asks how systems succeed most of the time and how understanding that success can strengthen adaptive capacity. Safety-II emphasizes the adaptive mechanisms workers and systems deploy to handle unexpected challenges — the everyday variability in work-as-done versus work-as-imagined. Learning from this variability, rather than only from failure modes, is proposed as a more complete basis for safety improvement.

Non-Falsifiability

The central empirical challenge for both NAT and HRO theory is that both frameworks can explain any outcome after the fact. An accident confirms NAT's prediction of inevitability; an accident-free period can be interpreted by NAT as an artifact of system boundary definition or timing rather than as disconfirming evidence. Conversely, HRO can attribute any organization's reliability to its identified characteristics and attribute any failure to their absence. The absence of prospective tests — predictions made before outcomes are known — is a persistent methodological weakness in safety science theory.

---

Disaster Justice and Differential Safety

The preceding sections focus on technical and organizational failure. A distinct body of scholarship asks a different question: who decides which risks are acceptable, and who bears the costs when those decisions are wrong?

Differential Safety Standards

Multinational corporations systematically apply lower safety standards in subsidiaries operating in developing countries than in their home-country operations. The International Labour Organization documented this pattern explicitly, and the 1984 Bhopal disaster — at Union Carbide's Indian plant, which operated under minimal safety standards through active cost-cutting while US operations maintained higher standards — exemplifies the mechanism at scale. After Bhopal, leading corporations publicly committed to uniform global safety standards, but enforcement mechanisms remain contested.

This pattern reflects broader global risk distribution: rich countries expose poor countries to greater systemic risk than the converse, and workers in industrializing countries face substantially greater hazard exposure than those in developed countries. Hazardous industries concentrate in the Global South, where regulatory capacity is weaker and the political power of affected communities is lower.

Social Vulnerability

Social vulnerability is not a natural characteristic of populations but a product of social, political, and economic processes that create unsafe conditions for marginalized and lower-income groups. The urban poor in the Global South face disproportionate disaster risk as rapid urbanization and climate change increase both hazard frequency and intensity — while technical solutions are promoted as primary risk reduction without adequate attention to the socioeconomic determinants of vulnerability.

Intersectionality provides a critical analytical lens: multiple overlapping social dimensions — race, gender, class, age, disability, migration status — interact to produce differential vulnerability and differential recovery outcomes. Housing tenure (ownership versus renting) is one concrete example of how structural inequality embeds into disaster vulnerability: renters face greater hazard exposure, lower insurance coverage, and greater barriers to recovery, and tenure correlates strongly with race, income, and age.

National recovery from disasters is substantially mediated by governance capacity, wealth distribution, and social welfare provisioning — not by disaster magnitude alone. The same event produces different distributional consequences depending on institutional context.

Disaster Justice as Framework

Disaster justice addresses fairness in governance of catastrophic hazards stemming from anthropogenic interventions, distinguishing itself from environmental justice by its focus on governance processes, the political nature of disaster responses, and the moral obligation to recognize and empower affected communities. Postcolonial disaster scholarship challenges dominant narratives that treat disasters in the Global South as inevitable development phases or technical failures, grounding disaster analysis instead in historical structural inequalities and contemporary transnational power relations.

---

Key Figures

Charles Perrow (1925–2019): Yale sociologist whose Normal Accidents (1984) established the interactive complexity / tight coupling framework and challenged the preventability assumption underlying most safety engineering.

Karlene Roberts (UC Berkeley): Founder of HRO theory, whose empirical research on naval aircraft carriers, air traffic control, and nuclear power plants identified the organizational characteristics sustaining sustained near-zero failure in high-hazard systems.

Karl Weick and Kathleen Sutcliffe: Organizational theorists whose five hallmarks of HRO mindfulness — preoccupation with failure, reluctance to simplify, sensitivity to operations, commitment to resilience, deference to expertise — operationalized HRO culture for broader application.

Nancy Leveson (MIT): Safety engineer who investigated the Therac-25, developed STAMP and STPA, and authored Engineering a Safer World (2011), establishing systems-theoretic approaches to accident causation as an alternative to component-failure and human-error models.

James Reason (Manchester): Developer of the Swiss Cheese Model and the active-failure / latent-condition distinction, foundational to organizational accident causation theory.

Sidney Dekker (Griffith): Developer of the "new view" of human error, Just Culture, and Safety Differently frameworks, which shift accident investigation from blame attribution to systemic reconstruction.

Diane Vaughan (Columbia): Sociologist whose study of the Challenger disaster produced normalization of deviance, the canonical concept for understanding how organizations come to treat known risks as acceptable through incremental acclimation.

---

Legacy

The disasters surveyed here did not merely kill people. They reshaped regulatory frameworks (IEC 62304, Public Law 116-260), transformed professional practice (blameless postmortems, safety management systems), and shifted theoretical understanding of what system failure means. They demonstrated that adding individual competent engineers to a poorly structured organization does not produce safe systems; that regulators who depend on the regulated for information cannot independently verify what they are not told; and that safety culture cannot coexist with organizational incentives that punish dissent.

They also demonstrated that the costs of these failures are not shared equitably. The 346 people who died in Lion Air JT610 and Ethiopian Airlines ET302 included many from communities with less political leverage to demand accountability than Boeing's home market. The populations most exposed to industrial hazards are those least positioned to resist corporate decisions about where to locate risk. A complete account of engineering disasters requires not only better technical models and better organizational culture, but also confrontation with the structural conditions that determine who bears the consequences when complex systems fail.