Book Deep Read · Chaos Engineering · Chapter 13
Chaos Engineering: System Resiliency in Practice · Ch 13 · The ROI of Chaos Engineering · 2020
Imagine you run a small team at a video-streaming company whose job is to break the system on purpose, a little, to find weak spots. At year end your boss asks: we spent all this money — what did we get? The honest answer is: "Nothing happened." And that is the hardest result in the world to prove.
This chapter tackles a very practical question: how do you show non-technical people that prevention is worth paying for?
You'd expect that as the system gets sturdier, outages keep falling, so comparing outage counts proves the point. In reality, outages often drop for a few months — and then creep back.
Not because the practice stopped working, but because once people feel safe, they speed up: they ship features more often and change code more boldly. Like a driver whose brakes just got fixed and who, without noticing, drives a bit faster. The safety margin you gained gets traded for speed.
The author's insight: what a chaos exercise really produces is a more accurate map in engineers' heads of how the system works. If the real product is people learning, then borrow the old way of grading a training course — the way you'd judge a driving school, in four steps:
The higher the step, the closer to the money your boss cares about — and the harder to pin down. But the lower three are easy to record, so you can stack up evidence step by step.
There is also a bonus people often miss: an exercise pulls teams that rarely talk into the same room. They get to know each other and learn whom to call — priceless when a real outage hits.
One honest cost: the top step can only ever be estimated; you will never get an exact bill.
Don't use "a few fewer outages" as your report card — speed will eat it. Treat chaos as training and collect evidence up four steps: liked it, learned, changed, results.
Want to see why outage counts rebound, what to measure at each level, and how to pitch it to leadership? → Switch to the deep read
You think chaos engineering pays off as "fewer incidents," but incident count is the worst report card: the counterfactual is invisible, the sample is tiny, and the organization re-spends the margin it gained on speed. The sturdier approach is to treat chaos engineering as training and measure it level by level with the Kirkpatrick model: reaction, learning, behavior, results.
Chapter 13 opens Part IV, "Business Factors," and is written by Casey Rosenthal (former manager of Netflix's Chaos Engineering team and one of the book's two lead authors). Part III covered how people learn from experiments; this chapter translates that same insight into the language of management. Next come Ch14 on open science and Ch15 on the maturity model.
A dedicated chaos team needs headcount, a platform, and slices of production traffic for experiments, and every year it must answer "is it worth it?" at budget time. Unlike a new feature, its output is a non-event: success means nothing happened.
Without a credible measure, the consequences are concrete: either the team is cut at the first budget squeeze, or it is pushed into claiming "incidents down 30% this quarter" — and the moment incidents rebound next quarter, the same number is used to prove it doesn't work. A bad metric hurts more than no metric.
The intuitive math: 10 incidents a month before chaos, 7 after, so 3 saved × cost per incident = ROI. The book points out at least three cracks:
The third point does not say chaos is useless; it says the benefit changed form: at the same incident level, the organization now moves faster. In safety science, Rasmussen's "dynamic safety model" describes exactly this drift — economic and efficiency pressure keeps pushing a system toward its safety boundary. Judging chaos by incident count means looking only at the balance after the gain was spent.
The chapter's pivot rests on a view that runs through the whole book: the real product of a chaos experiment is an updated mental model — discovering that the system does not work the way you thought. If the product is learning, you can borrow the most established framework for evaluating learning: the four-level model Donald Kirkpatrick proposed for corporate training in the 1950s. The training industry has used it for decades, and its strength is that it doesn't demand a dollar figure in one step; it stacks evidence level by level.
The beauty of the framework is that it breaks an unanswerable question into answerable ones: you can't say how much loss you avoided, but you can say (numbers illustrative) "40 experiments this quarter overturned 12 expectations and led to 9 fixes, 3 of which closed single points of failure with no fallback."
The book also calls out a kind of return that is easy to miss: everything outside the experiment itself. Preparing a game day often brings teams that rarely interact into one room — for the first time they align their understanding of the system and learn whom to call when things break. Arguing over what to inject exposes missing docs and knowledge silos; experiments often stumble on problems unrelated to the hypothesis (an alert nobody receives). These gains are hard to price, but they pay out in how well people coordinate during the next real incident.
No measure is perfect; the trade-off is between ease of measurement, credibility, and resistance to gaming.
| Measure | Ease | Persuasiveness | Main cost / trap |
|---|---|---|---|
| Incident count before/after | High, data exists | Looks strong, is weak | No counterfactual, tiny sample; once speed eats the gain it "proves" chaos is useless |
| Bugs found × estimated loss | Medium | Medium, nice number | False precision: "loss it could have caused" is all assumptions; counting bugs invites gaming |
| L1 Reaction | Very high, one survey | Weak | Satisfied ≠ useful; only a threshold signal |
| L2 Learning | Medium, needs written expectations | Medium | Depends on people honestly recording "I was wrong" — needs a blameless culture |
| L3 Behavior | Medium-high, tie to tickets | Fairly strong | Many fixes ≠ right fixes; separate "done because of the experiment" from "was planned anyway" |
| L4 Results | Low | Strongest (if attributable) | Same counterfactual and rebound problems; only stands up as the end of an L2/L3 chain |
Rule of thumb: talk L2/L3 to engineering, talk the "L3 → L4 chain" to leadership, and never claim credit with L4 or incident counts alone.
The chapter's value reaches beyond chaos engineering: all preventive work shares the same dilemma — security, testing, tech-debt cleanup, capacity headroom all look like "nothing happened" when done well. Kirkpatrick-style layered evidence is a general way for such work to survive budget reviews. Asked in an interview or architecture review "how do you prove reliability investment pays off?", a good answer doesn't quote an incident-reduction percentage; it lays out the learning → behavior change → results chain and points out up front that incident counts rebound as teams speed up.
Industry evidence
① Chaos engineering's output is a non-event, the hardest thing to prove to the business.
② Before/after incident counts are unreliable: invisible counterfactual, too few severe incidents.
③ The benefit is ephemeral: once sturdier, the organization speeds up, spends the margin, and incidents rebound.
④ A rebound isn't failure; the gain became delivery speed — Google's error budget institutionalizes this.
⑤ Change the frame: experiments produce updated mental models, so evaluate them like training.
⑥ Kirkpatrick's four levels: reaction, learning, behavior, results — higher is more valuable and harder to attribute.
⑦ Hold up L4 with an L2/L3 evidence chain; don't bet on incident counts, and don't give falsely precise dollar figures.
⑧ Collateral ROI: game days connect teams and expose knowledge silos, paying out in the next real incident.