Chaos Engineering Ch 17 · Let's Get Cyber-Physical

Chaos Engineering: System Resiliency in Practice · Ch 17 · Let's Get Cyber-Physical · 2020

What is this chapter about?

In earlier chapters, the things we deliberately broke were websites and apps: the worst case was a small slice of people who couldn't load a video. But software now drives cars, flies aircraft, moves factory robot arms and runs medical devices. When a failure can hurt someone, can you still break things on purpose — and if so, how? That is this chapter.

A strange fact first

The engineers who build cars and planes were "professionally studying how things break" decades before internet companies. Their classic method: sit down as a team, name every part one by one, and ask "How can it break? What happens then? How bad is it?" — and write it all into a big spreadsheet. The chapter's author argues that this old method and chaos engineering are natural partners who have barely been introduced.

Why the old world was hard

The spreadsheet runs on imagination. People can only write down the failures they can think of. Real accidents often come from two parts that are each fine on their own but go wrong together.
Software doesn't wear out. A metal part ages, so you can estimate "how often it breaks". A software bug was there from the day it was written, so that estimate doesn't work.
You can't just try it on the road. A website mistake can be rolled back; a car mistake on the highway cannot.

How it works, intuitively

Turn each row of the spreadsheet into a real little exam. If the sheet says "if the location signal drops, the car switches to another way of estimating its position", then actually cut the signal and watch whether it does that, and fast enough.
Try it in a simulator first, then on a practice track. Like learning to drive: practice on a computer, then on the driving school's closed course, and only then on real streets. Failure experiments climb toward the real world one step at a time.
Watch out for "looking changes it". These systems are very sensitive to timing; the little tools you add to observe them slow them down and change their behavior — like a patient whose blood pressure rises because they are nervous about being measured. The fix is to build those observation tools into the product from the start, so what you test is what you ship.

What it brings / how to choose

The old method makes sure you've thought of the ways things can break; the new method proves your responses actually work. The results go back into the spreadsheet, which becomes a living document that keeps up with the system. One honest cost: however realistic a simulator is, it isn't the real world — and the closer you get to reality, the more each test costs and the more safeguards it needs.

Remember it in one line

Failures you imagine must be confirmed by experiment; for systems that can hurt people, experiments start in the simulator and on the practice track.

Want to see how FMEA scores risk, why software failures can't be given a probability, which level — simulation, bench or field — an experiment belongs on, and how to deal with the probe effect? → Switch to Deep

In one sentence

You think chaos engineering is something internet companies invented — in fact safety-critical industries have been systematically "imagining failure" for decades. This chapter argues the two complement each other: functional safety's FMEA thinks failures through on paper, and chaos engineering confirms that thinking by experiment. In a physical world where failures hurt people, experiments must start in simulation and on test benches, and must face the probe effect — the fact that observing a system changes it.

Glossary

Cyber-physical system (CPS): a system whose software senses the physical world through sensors and acts on it through actuators (motors, brakes, valves) — cars, drones, industrial control, medical devices.
Functional safety: an engineering discipline whose goal is that when part of a system fails, the whole still avoids unacceptable harm to people.
Safe state: the harmless state a system should fall back to after a fault, such as pulling over, landing, or closing a valve.
Random vs systematic failure: the first is hardware ageing or sporadic damage, with an estimable failure rate; the second is a defect built into the design or specification that recurs every time the same conditions occur.
Fault tree analysis (FTA): start from one top-level accident and work downward, drawing which combinations of faults could cause it.
Hardware-in-the-loop (HIL): testing a real controller wired to a simulated vehicle or environment, so the controller believes it is in the real machine.
Real-time system: a system whose results must not only be correct but arrive before a deadline; late means wrong.
Blast radius: the maximum scope an experiment can affect if it goes wrong.

Where it sits

Chapter 17 of the book, in Part V "Evolution", written by Nathan Aschbacher. It follows Ch16 Continuous Verification — chaos engineering's end state in the web world — and pushes the practice outward into software that controls physical devices. It leads into Ch18, which revisits experiment design through the lens of Human and Organizational Performance.

What problem this chapter solves

A modern car often carries around a hundred electronic control units (ECUs), with code size commonly cited on the order of a hundred million lines; a drone's flight controller recomputes its attitude hundreds of times per second. Software is taking over control once done by mechanics and humans, and the unit of its failure is no longer "percentage of failed requests" but injuries.

These industries already have a mature practice — functional safety, governed by standards such as IEC 61508 (generic), ISO 26262 (automotive) and DO-178C (airborne software). Its core is hazard analysis before design, and analysis done thoroughly before building. The problems:

Analysis relies on imagination. The more complex and interconnected the system, the more failures come from interactions between components (the theme of the book's Ch1), and checklist-style analysis can't keep up.
Software doesn't obey probability. Hardware can be described as "fails a few times per billion hours"; software defects are systematic, with no credible failure rate.
The web playbook doesn't transfer. "Experiment in production and limit blast radius with small traffic slices" assumes mistakes can be rolled back; a brake failure can't be.

The chapter asks what chaos engineering can bring to functional safety — and how functional safety must constrain chaos engineering.

Core ideas, explained

1. Functional safety: first ask "how can it fail, and what then?"

Functional safety works top-down. First identify hazards (say, "unintended acceleration"), assess severity, exposure and controllability, and derive safety goals and an integrity level (in automotive, ASIL, from A to D, increasingly strict). Then design a safety mechanism for each goal, and specify which safe state to fall back to on a fault.

The key cut is random vs systematic failure. Hardware components fail randomly, so redundancy can be calculated from failure rates. Software doesn't wear out; its defects exist from the moment they are written and recur under the same inputs and timing. So the standards don't ask for software probabilities; they backstop with process rigor — the higher the level, the more reviews, test coverage and tool qualification required. It also means whether software is safe can only be answered with evidence, not calculation — exactly where experiments come in.

2. FMEA: rehearsing failure on paper

Failure Mode and Effects Analysis (FMEA) is one of functional safety's workhorse tools, and runs the opposite way from a fault tree: bottom-up. For each component, ask four things: how can it fail (the failure mode)? What happens locally? What happens at the system level? What existing means detect or mitigate it?

Each row then gets three 1–10 scores: severity (S), occurrence (O) and detection (D, higher when harder to detect), multiplied into a risk priority number RPN = S × O × D (1 to 1000). In plain words: failures that are severe, frequent and hard to notice get fixed first.

FMEA lists each failure mode of each component and ranks by RPN; it is good at single-point failures but structurally blind to interaction failures where nothing is broken on its own.

The author's verdict on FMEA is two-sided. Its value: it forces a team to think about failure systematically and produces a list of "how we believe the system can fail and how we plan to respond". Its limits:

It covers only what engineers can imagine. An FMEA is capped by the quality of the participants' mental models.
One fault at a time. Multiple simultaneous faults and timing interactions between components have no place in the table's structure.
Occurrence is a guess for software. The O score should rest on failure rates, which software doesn't credibly have.
Written once, then archived. The system changes, the sheet doesn't, and its conclusions quietly expire.

3. Chaos engineering as the next step beyond FMEA

The chapter's central argument: every row of an FMEA is, at heart, an unverified hypothesis. "GPS lost → the system switches to inertial navigation and alerts the driver within the specified time" is a steady-state hypothesis that can be tested by experiment — exactly the input format of chaos engineering.

The two then form a loop. FMEA supplies failure modes worth testing and their expected mitigations; chaos experiments actually inject the fault and observe whether the mitigation works, how quickly, and whether knock-on effects appear that the sheet never listed; the results flow back into the FMEA — correcting severity and detection scores, adding newly discovered failure modes. FMEA turns from a one-off document into a living one continuously calibrated by experiment, and chaos engineering turns from "randomly hunting weaknesses" into targeted experiments guided by safety analysis.

FMEA supplies hypotheses; experiments verify them up a ladder of simulation → bench → track → field; results flow back so the FMEA stays a living document.

This isn't an arbitrary graft: fault injection is already an accepted technique in safety engineering, and ISO 26262 lists fault injection testing among its recommended verification methods. The difference is the stance — traditionally it is often used as a confirmatory test (prove the designed mechanism triggers), while chaos engineering takes an exploratory stance (discover behavior you didn't anticipate).

4. The probe effect: observing the system changes it

Most CPS are real-time systems: a control loop must complete every few milliseconds, so timing is part of correctness. That creates a problem the web world rarely meets — the probe effect: code added to observe or inject faults consumes CPU and shifts scheduling and timing, changing the behavior of the system under test. Add one log line and a race condition may vanish; remove it and the bug returns. What your experiment verified may no longer be the system you ship.

In a real-time system, temporary observation or injection code steals time slices and shifts timing; a common answer is to keep a lightweight probe resident in the shipped build and budget for its cost.

The responses share one idea: don't let the "test build" and the "shipped build" diverge. Design observation and fault-injection hooks as a permanent part of the product — off by default, but shipped and certified with it; prefer low-overhead or hardware-assisted tracing; and measure the probe's own cost and include it in timing analysis.

Key trade-offs and choices

The core trade-off for chaos experiments in CPS is experimental realism vs risk of harm. The web world measures blast radius in percentage of traffic; here it must be measured in "who could get hurt in the worst case".

Which level an experiment runs at (magnitudes illustrative)

Level	Realism	Cost per run	Catches	Cost / blind spot
Pure simulation / SIL	Low–medium	Tiny; thousands of runs in parallel	Logic and state-machine bugs, combinatorial scenarios	Model-to-reality gap; unrealistic timing
HIL bench	Medium–high	Medium; few benches	Real controller timing, drivers, bus issues	Environment still simulated; benches are expensive
Closed track	High	High; people, vehicle, venue every time	Sensor noise, mechanical and environmental interaction	Limited scenarios; still needs a safety driver and independent e-stop
Controlled field / fleet	Highest	Very high	The real long tail	Harm may be irreversible; consider only with an independently guaranteed safe state

Three ways to "think about failure"

Method	Direction	Strong at	Blind spot
FMEA	Bottom-up: component → effect	Listing single-point failures exhaustively, prioritizing	Interaction and multi-fault failures; software occurrence is guessed
FTA	Top-down: accident → cause combinations	All paths to one serious accident	Only the top events chosen up front; still imagination
Chaos experiment	Empirical: inject → observe	Proving mitigations work, exposing unknown behavior	Only conditions that can be reproduced safely; subject to the probe effect

Rule of thumb: use FMEA / FTA to decide what to test, and experiments to decide what to believe; start at the cheapest, safest level, and move up only when the next level has abort and safe-state guarantees independent of the system under test.

Why it matters / in real systems

Autonomous driving, drones, robots, medical devices (IEC 62304) and industrial control are all becoming software-defined, and more backend engineers will write code that sits inside a control loop. When an architecture review asks "how do you know this fallback works?", this chapter's answer is: the FMEA says it works, a bench experiment proved it, and the result has been written back into the sheet.

Industry evidence

PX4 open-source autopilot: fault injection as a built-in flight-stack command. Its failure command is documented as "Inject failures into system.", with the example "Test the GPS failsafe by stopping GPS", and can inject off / stuck / garbage / delayed failures into GPS, battery, motors and more — while staying disabled by default behind the SYS_FAILURE_EN parameter: the capability is resident, but must be explicitly enabled. PX4 Autopilot · failure command source (github.com/PX4/PX4-Autopilot)
NASA JPL Mars Pathfinder (1997): resident tooling saved the mission. After landing, the spacecraft kept resetting itself. The team ran the ground replica with full tracing enabled, reproduced the failure and traced it to priority inversion; the fix used debugging facilities kept in the flight software to change a parameter remotely and enable priority inheritance. A classic case of observation and intervention capability living in the shipped build. Text source: public account of the incident by JPL flight-software lead Glenn Reeves, 1997
Counter-example · Uber self-driving test vehicle crash (2018). The US National Transportation Safety Board's 2019 report found that the system detected the pedestrian about 5.6 seconds before impact but repeatedly misclassified her; the vehicle's factory automatic emergency braking was disabled in autonomous mode, leaving a distracted safety operator as the fallback; and it criticized the company's safety risk assessment and safety culture. A public-road "experiment" without an independent abort guarantee is exactly what this chapter warns against. Text source: NTSB accident report HAR-19/03, 2019

Pitfalls and debates

"Chaos engineering = random destruction in production." In CPS it's the opposite: experiments are guided by safety analysis, start in simulation and climb level by level. The principles (steady-state hypothesis, minimize blast radius) stay; the execution environment changes.
"Once the FMEA is done, we're safe." An FMEA is a list of hypotheses, not evidence; uncalibrated by experiment, it records the team's beliefs.
"Passes in simulation = passes in reality." There is a model gap between simulation and reality; sensor noise, mechanical wear and environmental extremes may be missing.
"Software reliability can be estimated like hardware." Software defects are systematic; forcing a failure rate onto them yields false precision.
Ignoring the probe effect. Timing conclusions drawn from temporary instrumentation may not hold for the shipped build.
Limits of its time. The book came out in 2020; since then ISO 21448 (SOTIF, full edition 2022) addresses hazards that arise with no fault at all — when sensors or algorithms simply aren't capable enough. Fault injection doesn't reach those; scenario-based testing has to fill the gap.

Key takeaways

① In CPS the cost of failure is physical harm; chaos engineering's execution must change, but its principles don't.

② Functional safety has long studied failure systematically: hazard analysis, safety goals, safe states, integrity levels.

③ Hardware fails randomly and can be given probabilities; software fails systematically and can only be answered with evidence.

④ FMEA lists failure modes bottom-up and ranks them by RPN = S × O × D; its blind spots are interaction and multi-fault failures.

⑤ Central argument: every FMEA row is a hypothesis to verify; chaos experiments confirm it and the results make it a living document.

⑥ Experiments climb simulation → HIL → closed track → controlled field, each level needing an independent abort guarantee.

⑦ Probe effect: observation changes a real-time system's timing; keep lightweight probes resident in the shipped build and budget their cost.

⑧ Evidence: PX4 builds in fault injection; Mars Pathfinder was debugged with resident tooling; the Uber crash is the counter-example of missing independent fallback.