Designing Safety-Critical Software Systems

Lead Summary

Safety-critical software systems are those whose failure can cause death, serious injury, or catastrophic loss — aircraft flight management, automotive braking, nuclear reactor control, insulin pumps, railway interlocking. Unlike ordinary software, they operate within a formal engineering discipline that defines how much failure is acceptable (through integrity levels), what kinds of failure are possible (through structured hazard analysis), and how to argue that acceptable risk has been achieved (through safety assurance cases and formal verification).

The discipline has a well-established international standards architecture anchored by IEC 61508, the foundational cross-industry functional safety standard, from which domain-specific standards for automotive (ISO 26262), avionics (DO-178C), medical devices (IEC 62304), and rail (EN 50128) all derive. That architecture is now under stress: modern systems increasingly embed machine learning components whose data-driven, non-deterministic behavior directly contradicts the assumptions on which every current standard was built.

---

The Standards Landscape: A Family of Integrity Levels

IEC 61508: The Root Standard

IEC 61508 is technology-neutral and application-independent. It applies fail-safe and fault-tolerance principles to electrical, electronic, and programmable electronic systems across manufacturing, transportation, energy, and beyond. Its fundamental claim is that any safety-related system must work correctly or fail in a predictable, safe way.

The standard introduced Safety Integrity Levels (SILs) — a four-tier scale from SIL 1 (lowest risk reduction) to SIL 4 (highest). Quantification is built in: a key metric is the safe failure fraction (SFF), the percentage of all failures that result in a safe state. IEC 61508 permits SIL 1 claims for hardware with zero fault tolerance if SFF reaches 60% or above. At the top end, SIL 4 requires the most rigorous combination of fault-tolerance architecture, diagnostic coverage, and quantitative failure-rate targets.

Domain-Specific Standards

ISO 26262, DO-178C, and EN 50128/50129 all share the same core lifecycle model and traceability requirements as IEC 61508, while adapting to domain-specific terminology and regulatory requirements. A cross-domain picture:

Domain	Standard	Integrity Scale
General / industrial	IEC 61508	SIL 1–4
Automotive E/E	ISO 26262	ASIL A–D (+ QM)
Avionics software	DO-178C	DAL A–E
Medical device software	IEC 62304	Class A/B/C (Edition 1); Level I/II (Edition 2, 2026)
Railway	EN 50128 / EN 50129	SIL 1–4

One important gap: there is currently no formal cross-certification path between IEC 61508 and ISO 26262. Supplier components certified under IEC 61508 cannot be automatically reused in an ISO 26262 project without re-evaluation.

What ASIL Actually Applies To

A persistent misconception is that entire systems or products receive an ASIL rating. ASIL is assigned only to safety functions — specific functions whose failure could result in a hazardous event. Convenience features, user interface displays, and non-critical sensors carry no ASIL and follow ordinary product development processes. Only safety-identified functions must undergo hazard analysis and the full ISO 26262 lifecycle.

Equally important: you cannot achieve a higher ASIL by combining multiple lower-ASIL subsystems. Integrating two ASIL B subsystems does not produce an ASIL C system. The integrated system's ASIL is determined independently through its own hazard analysis of the hazards the integrated system introduces.

---

The Safety Lifecycle

Both IEC 61508 and ISO 26262 mandate a V-model structure that spans six continuous phases: concept, development, production, operation, service, and decommissioning. Safety is not a gate event — it is a property maintained across the entire span, including after deployment.

The V-model maps every specification phase on the left (concept → requirements → architecture → implementation) to a corresponding verification phase on the right (unit test → integration test → system validation). Each phase generates gating work products that must be reviewed before proceeding, enforcing structured progression and preventing specification gaps.

Teams that embed functional safety discipline from the start face lower rework and achieve compliance naturally as an output of engineering work. Teams that defer it find rework and evidence-gathering costs compound with every release cycle.

Traceability must be maintained as an ongoing operational activity, not a final documentation task. Every modification to a safety-related system — including over-the-air updates — must be analyzed for safety impact, traced within the existing traceability structure, and re-verified to the appropriate integrity level.

---

Hazard Analysis and Risk Assessment

HARA: The Gateway to SIL/ASIL

Hazard and Risk Analysis (HARA) is the process that opens the safety lifecycle. IEC 61508 structures the first five lifecycle phases as an analysis stage, moving sequentially from defining the Equipment Under Control, through hazard identification, hazard analysis, and risk assessment, to specifying safety functions and their SIL targets.

In ISO 26262's HARA, ASIL assignment is determined through three parameters: Severity (S0–S3), Exposure (E0–E4), and Controllability (C0–C3). The combination of these three values maps to an ASIL rating:

• Severity ranges from S0 (no injury) to S3 (fatal)
• Exposure from E0 (incredible) to E4 (high probability of hazardous situation)
• Controllability from C0 (easily controllable) to C3 (uncontrollable or difficult to control)

Each hazard assigned an ASIL generates a corresponding safety goal, which drives decomposition into Functional Safety Requirements (FSRs) and Technical Safety Requirements (TSRs). This hierarchical flow — hazard → ASIL → safety goal → FSR → TSR — is the operative mechanism through which risk classification becomes engineering obligations that trace through every design, code review, test, and verification activity.

The Four Principal Methods

No single hazard analysis method covers all hazard categories. In practice, multiple methods are combined to achieve comprehensive coverage:

FMEA (Failure Mode and Effects Analysis) enumerates individual component failure modes and their consequences. It operates bottom-up: what can this component do wrong, and what happens if it does? Design FMEA (DFMEA) focuses on component-level failures; System FMEA (SFMEA) broadens to subsystem-to-system interactions. FMEA is extensively used in aerospace, nuclear, and defense — and is deeply embedded in regulatory requirements in those domains. Its limitation: it enumerates single failure modes and does not natively capture interaction-driven hazards or software-specific hazards.

FTA (Fault Tree Analysis) works top-down: start from an undesired top event and decompose it into combinations of lower-level failures using Boolean logic gates. FTA develops fault propagation pathways and provides quantitative probability ranking of root causes. It is standard practice in nuclear, aerospace, oil and gas, medical devices, and pharmaceutical manufacturing — and is often mandatory for safety licensing and certification. Tools such as CAFTA, RiskSpectrum, and SAPHIRE are widely used. FTA's blind spot: it can only analyze failure combinations that are modeled in the tree — unknown interaction scenarios outside the model's scope are invisible.

HAZOP (Hazard and Operability Study) applies structured guide words (MORE, LESS, NONE, REVERSE, OTHER THAN, etc.) to system parameters to identify deviations from design intent. An experienced independent facilitator leads a diverse team of 8–10 members, integrating process engineers, vendor representatives, equipment designers, and safety managers. HAZOP excels at finding hazards you hadn't anticipated — filling "unknown unknowns" that FTA cannot address. Node selection — which system elements to analyze — is critical to completeness. HAZOP is most commonly used in the planning phase and in process industry applications.

STPA (System-Theoretic Process Analysis) is grounded in the STAMP accident model developed by Nancy Leveson at MIT. STPA treats safety as a control problem rather than a chain of component failures. Under STAMP, accidents result when a controller's process model is incorrect, feedback is unavailable or corrupted, or when control actions are unsafe — not only when components break. Step 1 identifies losses, hazards, and unsafe control actions (UCAs); Step 2 traces causal scenarios that lead to each UCA through process model flaws and feedback failures.

In practical case studies comparing STPA and FMEA on software-intensive systems, STPA identifies component-interaction, software-specific, and human-interaction hazards that FMEA overlooks — particularly data-flow inconsistencies and incomplete feedback scenarios. SAE J3307 (2025) provides cross-industry STPA guidance.

Fig 1

Multi-Method Integration

Using multiple methods provides multi-dimensional understanding: HAZOP focuses on process deviations, FMEA examines equipment reliability, FTA computes how failure combinations produce top events. The structured workflow is: HAZOP first (identify what can go wrong), then FMEA or FTA (document failure chains and quantify), with STPA providing systemic control-theoretic coverage throughout. This integration is more defensible and comprehensive than any single method.

---

ASIL Architecture Constraints

ASIL ratings translate into concrete hardware architecture obligations. ASIL D imposes the most stringent constraints:

• Diagnostic coverage ≥ 90% for hardware fault detection (vs. 80% for ASIL C, 60% for ASIL B)
• Explicit evaluation of latent faults and dual-point faults
• Probabilistic Metric for Hardware Failure (PMHF) thresholds: ASIL D Class 1 parts must achieve failure rate below 1E-10 per hour (MTTF above 1.14 million years)

ASIL A is qualitative only — no explicit quantitative PMHF target, reflecting the lower risk profile. For ASIL D software verification, Modified Condition/Decision Coverage (MC/DC) and formal methods are required alongside structural testing.

---

Fail-Safe and Redundant Design

The Core Principle

A fail-safe is a design feature that, in the event of a failure, inherently responds in a way that causes minimal or no harm to equipment, environment, or people. This definition is consistent across electrical, mechanical, aerospace, automotive, and nuclear domains.

A fail-safe system must have a predetermined safe state that it transitions to upon failure detection, with adequate time between detection and loss of safety function to complete the transition safely. Safe states vary by context:

• Nuclear plant: controlled shutdown
• Elevator: stop and open doors
• Medical infusion pump: cease operation
• Train control: close signals to stop

The transition may be abrupt (immediate shutdown) or gradual (graceful degradation to reduced functionality), but the system must reach the safe state before harm can occur.

Not every system can be fail-safe. Aircraft in flight, life-support systems, and other systems that cannot safely shut down must instead employ a fail-operational approach — maintaining continued safe operation during failures through redundancy and fault tolerance rather than transitioning to a stopped state.

Redundancy: Power and Limits

Redundancy is the primary mechanism for achieving fail-safe behavior. Dual-modular redundancy employs two identical processing units with a comparator monitoring both outputs and triggering safe shutdown on discrepancy. Triple modular redundancy enables voting-based error correction. These approaches apply consistently across nuclear, aviation, automotive, and industrial systems.

But redundancy introduces significant penalties: increased complexity, higher weight and power consumption, greater manufacturing and maintenance costs, and difficulty in detecting which redundant path has failed. A failed component may fail by generating incorrect data rather than failing silently, creating additional challenges.

More fundamentally: common-cause failures can defeat redundancy entirely. Common-cause failures occur when a single event disables multiple supposedly independent redundant systems simultaneously — no amount of redundancy can reduce total failure probability below the common-cause failure probability. Redundancy cannot reduce the individual component failure rate by more than approximately one order of magnitude, because common-cause failures typically account for one-tenth of all failures.

Historical examples are stark: the Ariane 5 launch failure resulted from both the duty and backup Inertial Reference Systems failing due to the same software overflow bug. Apollo 13's oxygen tank explosion cascaded to destroy the second tank.

---

Safety Assurance Cases and GSN

The Burden of Proof

In safety assurance, the system developer bears the affirmative burden of proving safety. A safety case is a structured argument — supported by evidence — that a system is acceptably safe for a defined application in a defined operating environment. Regulatory bodies must be satisfied by the case presented before granting certification.

A safety case is not simply a collection of documents. It is an explicit argument structure linking top-level safety claims through intermediate sub-claims to specific evidence artifacts (test results, formal proofs, simulation data, operational experience). The argument structure shows why the evidence supports the claim, not merely that it exists.

Goal Structuring Notation (GSN)

Goal Structuring Notation (GSN) is the most widely adopted graphical language for expressing safety cases. Its visual vocabulary:

• Goals (rectangles): safety claims
• Strategies (parallelograms): argumentation steps
• Evidence/Solutions (ovals): evidence artifacts
• Context (rounded rectangles): assumptions and scope
• Justifications (pentagons): explanations of why approaches are acceptable

GSN was standardized as GSN Community Standard version 3 (May 2021) and is referenced in ISO 15026-2:2022. GSN's modular extension addresses scalability: complex systems can decompose safety cases into separate but interlinked argument modules that can be developed in parallel and integrated without a single monolithic case document.

Critically: GSN notation does not enforce the logical validity of arguments. It is a communication tool that makes claim-argument-evidence structures auditable and reviewable — not a guarantee of their correctness.

Formal Methods in Assurance

Formal methods provide the strongest form of verification evidence: mathematical proof of safety properties. DO-333 (the avionics formal methods supplement) recognizes three principal families: theorem proving, model checking, and abstract interpretation.

Frama-C (for C) and SPARK (for Ada) are the primary industrial toolsets with documented DO-178C/DO-333 certification pathways. AWS has applied TLA+ to production systems including S3 as a practical industry example outside aviation.

But formal methods face severe scalability limits. Model checking confronts state-space explosion: as system complexity grows, the number of possible states increases exponentially, making exhaustive verification infeasible without careful abstraction. In practice, formal methods apply to critical subsystems rather than entire software stacks.

Evidence Portfolios, Not Single Proofs

Regulatory frameworks require mixed portfolios of formal and informal evidence. ISO 26262 mandates formal methods as available techniques at software architectural and unit design levels, while requiring that a structured safety case compile objective evidence from multiple sources: verification results, test reports, simulation evidence, and formal analyses. Common Criteria EAL 5 and above require formal verification but do not exclude empirical testing.

The Decoupling Risk

A documented pattern shows that formal assurance cases can be complete and certified while the systems they document later fail catastrophically. Aviation incidents have been attributed in part to incomplete or flawed reasoning within formally approved safety cases. Assurance case approval provides institutional confidence — and liability protection — without necessarily providing corresponding safety evidence. This risk drives the development of Assurance 2.0 frameworks that systematically seek defeaters (counterarguments) to challenge case assumptions, and of dynamic assurance approaches that update evidence as systems evolve.

---

The Machine Learning Challenge

Why Existing Standards Fall Short

Every current functional safety standard was designed for deterministic, rule-based systems with traceable decision logic and exhaustive testability. Machine learning components violate all three assumptions:

1. ML components cannot be specified with the precision required by functional safety standards because their functionality is learned from data rather than explicitly defined. IEEE, ISO 26262, DO-178C, and EN 50128 all require precise specification of functional safety requirements — a requirement that is systematically infeasible for ML.

2. The V-model breaks down for ML: source code reviews and coverage testing enforced by ISO 26262 and DO-178C are not applicable to deep neural networks. 59.2% of organizations report that V&V of ML-based features is particularly difficult due to inapplicability of traditional approaches.

3. Neural networks are opaque "black boxes" — their internal decision mechanisms are not interpretable to humans, sitting in direct tension with certification requirements for demonstrable, transparent reasoning.

4. Fault avoidance — the classical dependability strategy of preventing errors through rigorous design — is infeasible as the primary strategy for ML because ML internal operation is developed experimentally and is opaque. Fault tolerance and failure management become the necessary strategies instead.

Distributional Shift: The Deployment Failure Mode

ML models deployed in safety-critical systems encounter test-time inputs that diverge from training distribution (distributional shift), leading to erroneous predictions and dangerous failures. In autonomous driving, systems must issue warnings and hand over control upon detecting unusual scenes. In healthcare, distributional shift between training cohorts and deployment populations undermines model reliability. Detecting and responding to distributional shift — through out-of-distribution detection, uncertainty quantification, and runtime monitoring — constitutes a critical robustness requirement for safe deployment.

The Six Certification Pillars for ML

ML assurance literature organizes certification as six interdependent pillars: robustness (handling perturbations and distribution shift), uncertainty quantification (principled treatment of epistemic and aleatoric uncertainty), explainability (human-interpretable justifications), verification (formal or empirical property verification), safe reinforcement learning (constraints during agent training), and direct certification (frameworks enabling models themselves to be certified).

Runtime Monitoring as a Substitute for Offline Verification

Because ML components cannot be fully formally verified offline, runtime monitoring frameworks are positioned at critical decision points to observe system inputs, internal states, and outputs, detecting distribution shifts, out-of-distribution inputs, and model performance degradation during operational deployment. The Simplex architecture — pairing an untrusted ML controller with a trusted baseline controller that can reassert control when the ML component's outputs exceed verified safety envelopes — is one structural pattern for safe deployment.

AMLAS and Emerging Standards

The AMLAS framework (Assurance of Machine Learning for Autonomous Systems) provides a six-stage lifecycle guidance that integrates safety assurance into ML development, from safety scoping through model deployment, including explicit safety argument patterns for each stage. It extends to reinforcement learning components.

At the standards level:

• ISO/IEC 5469:2024 (published January 2024) provides a domain-agnostic functional safety standard for AI, explicitly connecting the AI lifecycle (ISO/IEC 5338) with the functional safety lifecycle (IEC 61508).
• ISO/PAS 8800 addresses AI in automotive development with V-model integration and dataset management requirements.
• ISO/TS 5083:2025 addresses automated driving systems at SAE Levels 3–4.
• ISO 26262 Edition 3 (~2027) is expected to include AI/ML guidance.
• The EU AI Act classifies AI systems as high-risk when used as safety components in EU-regulated products, requiring third-party conformity assessment from August 2025.

---

Recurring Failure Patterns

The Component-Reliability Trap

In complex systems, catastrophic accidents can occur even when all components are individually highly reliable, and even when no single component has failed. Leveson's critique of component-failure thinking: automobiles, chemical plants, and aircraft can have highly reliable components yet produce highly destructive accidents through unsafe interactions among those elements. STAMP and CAST treat safety as a control problem, shifting investigation from "what failed" to "how was control lost."

STAMP explicitly includes organizational, management, and human decision-making factors as essential components of the hierarchical control structure — not external variables. The Boeing 737 MAX MCAS failures exemplify this: single AOA sensor reliance for a critical control function was defensible only if you disregarded that single point of failure, a design choice driven by cost and schedule rather than technical necessity.

The Therac-25 radiation overdoses similarly drove regulatory reform: the FDA revised its medical device software guidelines and IEC 62304 was subsequently created, introducing formal development and validation requirements that have since become baseline for the field.

Compliance as Safety Theater

Compliance with functional safety standards is necessary but not sufficient for actual product safety. Organizations can achieve standards compliance through documentation and process adherence while failing to detect and prevent real safety defects. Safety Differently researchers have documented that incident-free records and procedural compliance can mask growing safety risks through incident suppression — incidents and fatalities can follow years of injury-free performance.

The practical implication: the most successful functional safety programs are those where compliance becomes invisible to developers — a natural output of good engineering rather than a parallel bureaucracy. This requires integrated tooling where safety requirements, design artifacts, test cases, and risk items live in connected systems so evidence chains are maintained as work happens.

The ASIL Inheritance Fallacy

ASIL cannot be directly inherited across different applications or products. An ASIL D component in one vehicle application must be re-evaluated through hazard analysis when integrated into a different application. The widespread misconception that a component's ASIL rating is intrinsic and portable leads to misaligned safety goals and compromised safety cases. In practice, the effort to adapt and qualify a reused component is often equivalent to re-developing it.

Adding Safety Measures Can Increase Risk

Perrow's Normal Accident Theory argues that adding additional warning systems and safeguards to complex systems paradoxically increases system complexity and may create new categories of accidents. When a system already exhibits interactive complexity and tight coupling, new safety components introduce additional failure modes, new interaction pathways, and novel states that operators must navigate. This paradox supports the STAMP approach of analyzing system-level control rather than bolting on additional protective layers.

---

Controversies and Tensions

Agile vs. safety standards. Agile's emphasis on working software over comprehensive documentation directly conflicts with the extensive documentation requirements of IEC 61508, DO-178C, and ISO 26262. Fast iteration cycles clash with the need for thorough safety reviews and formal approval processes. Practical implementations show the conflict can be resolved through process tailoring and automation rather than being irreconcilable — but the tension is real and ongoing.

ML certification paradigm shift. There is broad academic and industry consensus that existing standards do not adequately address ML components. Proposed AI-SIL extensions to ISO 26262 (assessing task complexity via input entropy and output non-determinism) lack empirical validation on deployed systems. The standards community is actively working to close this gap, but deployment of ML in safety-critical systems currently outpaces regulatory frameworks.

Formal methods adoption barriers. Formal methods face skill gaps, cost-justification challenges, and lack of consensus on DO-333 application. State-space explosion limits applicability to subsystems. Tool qualification costs under DO-330 can be prohibitive. Common Criteria EAL 6 and above require formal verification, but industry resistance to formal methods at high assurance levels is documented.

Safety vs. SOTIF. ISO 26262 addresses failures of components — not failures of intended function. When all components work correctly but the system makes a wrong decision (e.g., a perception algorithm misclassifies a stationary object), ISO 26262 offers no applicable safety assurance mechanisms. ISO 21448 (SOTIF, 2022) was created to fill this gap — but the two standards together still do not provide a complete framework for fully autonomous systems operating in open-world environments.