Overview of Principles

In one sentence

You think chaos engineering is "breaking things in production." The words this chapter wants to replace are precisely "breaking things": it is experimentation, not testing — a test can only confirm something you already knew, while an experiment produces new knowledge. The five advanced principles (hypothesize about steady state, vary real-world events, run in production, automate to run continuously, minimize blast radius) do exactly one job between them: rewrite "I think it holds up" into a claim that can be falsified on the spot, then try to falsify it at the smallest possible cost. Falsify it and you have a real problem to fix; fail to falsify it and all you have is confidence about that one run, under those conditions.

Glossary

Where it sits

Chapter 3 of 21, closing Part I, and the methodological spine of the whole book: Ch1 established that a complex system's behaviour cannot be derived, Ch2 established that the safety direction has no natural feedback so the signal has to be manufactured, and this chapter gives the rules for manufacturing it. Downstream, Ch4–Ch8 are Slack, Google, Microsoft, LinkedIn and Capital One bending these five principles to their own constraints; Ch16 ("continuous verification") is where principle four ends up. In the real world it is aimed at any team that already has monitoring and rollback but whose "drills" are still checklist inspections.

The problem this chapter solves

Ch1 and Ch2 leave one hole: given that you cannot derive the behaviour and must manufacture a safety signal, what counts as manufacturing it correctly?

Leave that hole and the practice degenerates into one of two shapes. One is monkeys with hammers: kill a few instances, glance at a dashboard, declare the system resilient. The other is sneakier — the drill as theatre: schedule a quarterly exercise, then write a "drill successful" report afterwards. Because the criterion is read off the graphs after the fact, any outcome can be narrated as success.

Both have the same root cause: nobody wrote "I think it holds up" down as a claim that could be knocked over. That is what this chapter fixes — the minimum set of constraints that makes "causing failures" actually produce knowledge. What happens if you skip it: the team starts treating "we do chaos engineering" as evidence of reliability, which is worse than not doing it at all. It adds no real resilience and removes your wariness.

The core, properly explained

Nail the definition first: experiment, not test

The definition this chapter adopts (and the one the whole field uses) is: chaos engineering is the discipline of experimenting on a system in order to build confidence in the system's capability to withstand turbulent conditions in production. Every word draws a line, and the hardest line is "experimenting."

A test asserts a known property: input A, assert output B, two possible outcomes. It produces no new knowledge — it only tells you whether the property you had already thought of still holds. An experiment samples a question whose answer you do not know, and its output is new information.

Concretely. "When the downstream returns 503, the gateway should return 502 rather than fall over" is a test: you know the right answer in advance, write a unit test. "When the downstream p99 (the 99th slowest of a hundred requests) goes from 80ms to 2s, does checkout success stay above 99.9%?" is an experiment: the answer depends on connection-pool size, retry policy, timeout cascades and cache hit rate as they happen to combine at that moment — and that combination is exactly Ch1's "nobody holds it in their head." You cannot write that assertion, not out of laziness, but because you do not know what to assert.

From this the chapter derives a companion stance: chaos engineering prefers verification (looking only at output at the system boundary) over validation (analysing the internal parts and building a mental model that explains them). Ch1 already showed the latter is unachievable for complex systems, so the discipline bets everything on the former: you can ask "is the customer still okay?" without understanding the whole call chain — which means a team that just inherited a system does not have to read it first before experimenting on it.

Principle 1: hypothesize about steady state, falsifiably

Steady state is the measurable, stable output of a system that is working. The chapter is picky about it: it must be a business metric at the boundary, close to the customer, not an internal attribute.

The canonical example is Netflix's SPS (stream starts per second). It is good in three ways: it maps directly onto "someone is successfully watching"; its curve is extremely stable day over day, so the same hour is basically predictable; and if any link in the chain genuinely breaks, it drops.

The counterexamples are internal metrics: CPU utilization, one service's error rate, queue depth. Not that you shouldn't watch them — of course you do while debugging — but their signal-to-noise is bad: CPU alone can swing tens of percent at minute resolution, which buries the signal in noise.

The standard form of the hypothesis is: under condition X, steady state Y stays within range Z. None of the three can be dropped, least of all Z — the criterion has to be written down in advance: how far a deviation, for how long, counts as falsification. An experiment judged by reading the graphs afterwards is unfalsifiable, and degenerates into the theatre above.

Principle 2: vary real-world events — the hard part is ranking, not injecting

This principle asks you to inject events that genuinely occur in the real world, ranked by potential frequency times potential impact. Two common narrowings need correcting.

The first is thinking it means killing machines. The real list is far longer: network latency and packet loss, a downstream returning malformed or empty responses, expired certificates, clock skew, a full disk, a dependency that is wholly unavailable, retry storms, a bad config push, the loss of one availability zone — plus non-technical events. Google's DiRT really does rehearse "the key engineer is unreachable" (Ch5).

The second is thinking it means failures. A promotion that multiplies traffic by five, an autoscaling event, an ordinary deploy — all real-world events, and all frequent starting points for incidents.

Why ranking is harder than injecting: the failure space combines explosively, while injection mechanisms have not been scarce for years. The chapter's answer is to lean on the real world's own distribution — your incident archive, your alert history and your dependency topology are a ready-made prior. Systematic ranking is the whole subject of Ch6 (Microsoft); Ch12 goes further and computes injection points from data lineage instead of guessing.

Principles 3 and 5: run in production, with the blast radius locked down

These two have to be read together, because they hedge each other.

Why it has to be production. The system you want confidence in is production; everything else is a stand-in. Staging differs in at least six ways: traffic shape and concurrency, data volume and skew, config drift, the versions and capacity of real dependencies, cache hit rate, and pool watermarks. Concretely: with 1/100 of production traffic the connection pool never saturates — and the core mechanism of a retry storm is exactly "pool full → queueing → timeouts → retries." That chain cannot be reproduced in staging, not because the environment is sloppy, but because the mechanism fundamentally needs that order of magnitude.

This is the most contested of the five, and the chapter itself concedes it has to be approached gradually: Slack (Ch4) writes a design document, prepares an abort plan and runs it with humans in the loop; Capital One (Ch8) starts outside production under compliance constraints.

How the blast radius is locked down. The core technique is a canary-style comparison: pull a very small slice of production traffic (order of one percent), split it into a control group and an experimental group, inject only into the experimental group, and compare the two groups' steady-state metric. Four companion measures, none optional — run at low traffic, change one variable at a time, preset an automated abort, and keep one big switch anyone is allowed to hit.

The honest cost: the blast radius can never be zero. Do it in production and some real users are caught in it; squeezing that down to something ethically acceptable is the subject of Ch7 (LinkedIn).

Principle 4: automate it to run continuously — because confidence decays

A hand-run experiment buys you one-off confidence. But the system changes daily (that is Ch2's drift): a timeout parameter merged today, a dependency added yesterday, either can void last month's conclusion. Confidence has a half-life, so this principle asks you to turn experiments into standing infrastructure.

Two things need automating, and the second is the one people skip: execution (when to inject, what to inject, how to split traffic) and the verdict (how the two groups' metrics are compared, what counts as significant deviation, when to abort). Automate only execution and leave the verdict to someone watching a dashboard, and you bottleneck on the human at scale — and in the moment they are staring at the dashboard, the blast radius is growing.

Netflix's own trajectory is the best footnote to this principle: from Chaos Monkey (terminate production instances at random — crude, but running every day) to ChAP (Chaos Automation Platform: pick the experiment, split the two groups, judge and abort, all automatically). The direction is unambiguous — from "a scheduled exercise" to "a standing gate in the pipeline," which is Ch16's continuous verification.

What it is not: not breaking things, and not antifragility

It is not "breaking things in production." Breaking is the means, information is the product, and fixing is the point. The more accurate phrasing is "fixing things in production": the experiment cashes a latent incident in at a time and a size you chose, trading one unhurried working day for one ambush at three in the morning.

It is not antifragility. Taleb's antifragility proposes adding stressors to make a system stronger, usually by adding redundancy and machinery — and adding things means adding complexity, a bill Ch1 and Ch2 already itemized. Chaos engineering runs the other way: it adds nothing, it only surfaces the weaknesses that are already there. And it does not make the system stronger by itself; it produces information, and whether, how and whether-it-is-worth-it to change anything is still a human decision. Treating fault injection itself as the strengthening mechanism is the most expensive misreading of this discipline.

Key trade-offs and choices

All of principle 1's difficulty lives in "which metric":

Two criteria only: at the boundary, and it drops when things truly break. Most teams have two or three metrics that pass both.

The tension between principles 3 and 5 resolves differently at different maturities. Four common stages:

Which stage fits can be decided quite concretely:

• No observability or one-click rollback yet, or a brand-new team. Do not start at "run in production" — principle 3 is the destination, not the entrance. Do one thing: pick a genuine steady-state metric and watch it. Without it, the other four principles have nothing to stand on.
• A platform team building this for the whole company. Invest first in percentage routing, per-group metrics and automated abort — not in a pile of injection mechanisms. Injection has never been the bottleneck.
• Heavily regulated industries (clearing and settlement, healthcare, vehicles). Principle 3 has a hard edge: settled money is irreversible. Starting outside production and turning experiment results into the evidence auditors want is Ch8's route; where the line falls in the physical world is Ch17's subject.

Why it matters / in real systems

The most directly usable thing here is one sentence that can change the outcome of a review: "that has never happened to us" is not evidence; "we deliberately caused it last week and steady state never moved" is. Asked in an interview how you ensure reliability, "write tests, add monitoring" is the pass mark; "tests guard known properties, and the unknown remainder is handled by experiments — define the steady-state metric, write a falsifiable hypothesis, run it in production at a small percentage, automate the verdict and the abort" is a different level.

Pitfalls and disputes

• Picking an internal metric as steady state. The experiment then either always passes or always fails — both mean no signal.
• Not writing the criterion down in advance. Read off the graphs afterwards, the hypothesis is unfalsifiable: whatever happens can be written up as "drill successful."
• Inverting the order: chasing "run in production" before observability and abort exist. The most misread line in the chapter. Going to production without a steady-state metric and an abort switch means using real users as your monitoring.
• Treating chaos engineering as a replacement for testing. Tests are the regression net over known properties; experiments probe the unknown. Cutting the former to fund the latter is a net loss.
• The antifragility misreading. Believing the injection itself strengthens the system. It only produces information; strengthening takes a human change — information left sitting in a report is the same as no experiment.
• One passing experiment is not proof of reliability. A Popperian asymmetry: experiments can falsify, never verify. A hypothesis that survived is valid only for that one combination of conditions. The common slide is "quarterly drills all green → into the report → treated as a reliability conclusion."
• The limits of the principles themselves. They do not tell you which event to inject (Ch6 and Ch12 answer that), nor how to quantify steady state well enough to judge automatically (Ch16). And of the five, only "minimize blast radius" reduces to a number; the other four come down to judgement — Ch15's chaos maturity model patches this, but it is still qualitative buckets, not a gauge.

Takeaways