BOOK DEEP-READ · CHAOS ENGINEERING · CH 5
Chaos Engineering: System Resiliency in Practice · Ch 5 · Jason Cahoon (Google) · 2020
In the previous chapter, Slack wrote a script and rehearsed one old system. This chapter swaps the stage for an entire company — it is about the company-wide drill Google runs internally, called DiRT.
The surprising part: machines are not the only thing being broken. Documents, permissions, communication channels and people are all on the list too.
A building runs a fire drill. Everyone assumes the point is whether the extinguishers are charged — but somebody inspects those every year, and they're usually fine.
What actually traps people on the real day is different: someone piled boxes against the fire door; the person who blows the whistle is on holiday and nobody knows who to follow; the evacuation map is posted on the floor that's burning.
Nobody inspects any of that, because none of it looks like equipment and none of it is on a checklist. That's exactly what DiRT rehearses.
Large companies usually do redundancy well at the machine level: two or three data centres, replicated data, traffic that can be shifted. So a comfortable conclusion settles in: we'd survive.
That conclusion has never been tested end to end. What actually stalls a real recovery is: the only person who really knows how to do the failover is on holiday, and his method was never written down; the emergency runbook lives on the intranet, and the intranet went down with everything else; everyone is waiting for someone with the authority to say go.
None of these is a technical defect. They're all steps nobody ever tested. Machine backups have an owner; backups for people and processes have none.
The payoff is an honest list: you finally know where your recovery breaks, and most of the breaks aren't in the machines. The cheapest tier costs almost nothing — a table, a facilitator, and one sentence: "go find that runbook right now" — a small team can run it today. One honest cost: a company-wide exercise is enormously expensive to coordinate, happens a few times a year at most, and covers only a handful of failure modes.
Your machines will probably hold; your processes, your documents and the operations only one person knows how to do probably won't — so a drill has to break people and paper along with the servers, under one rule agreed in advance: a real incident stops everything.
Want the mechanisms, the numbers and the diagrams? → Switch to the deep read
You assume a disaster drill exists to verify that your technical redundancy is sufficient. A decade-plus of DiRT at Google points the other way: the redundancy is mostly fine — the real single points of failure sit in process, permissions, documentation and people, and none of those four appear in any automated test or any SLO. So DiRT widens the boundary of "the system under test" from services to the whole company's ability to recover, treating even "the key engineer is unreachable" as an injectable failure — while a set of rules of engagement keeps shaking the company from breaking it.
Chapter 5 of 21, the second entry in Part II ("the principles in practice"). Same posture as the previous chapter, a different order of magnitude: Slack pushed one legacy system into a human-in-the-loop drill; Ch5 scales that method to company size and formally pulls non-technical targets into scope. Further on, Ch7 is LinkedIn's ethical constraints on blast radius and Ch8 is Capital One's incremental path under financial regulation. In the real world this chapter addresses organisations that already have multi-site redundancy and a written DR / BCP document, and have never actually tested either.
On a typical large service the technical redundancy looks complete: three or more regions, replicated data, traffic shiftable per region, RPO near zero, RTO stated as 15 minutes. But those numbers are component-level promises, and stacking them up does not add up to "the company can recover from a disaster".
What's missing is the layer in between. In a genuine regional incident the recovery sequence looks like this: someone decides this is an incident → it gets a severity → someone with authority approves the failover → the runbook is located → a string of commands that has never run in production gets executed → support and downstream partners are notified. Exactly one step on that chain is technical; the rest is process, permissions, documents and people — and not one of them has ever been tested.
What happens if you don't fix this? Three very typical failures:
So the question is: how do you verify the recovery capability of the whole company, not just the redundancy of each component — without manufacturing a real incident in the process?
This is the diagram to take away. Decompose one recovery and you get, in order: technical components → process and permissions → documents and tooling → people. Engineering investment and automated testing cover essentially only the first layer, while the probability of breaking on the real day increases as you go down.
The reason is no mystery: layer one has owners, monitoring and CI; the other three have no owner — nobody unit-tests an approval chain, and nobody checks weekly whether the commands in a runbook still work. Almost all of DiRT's value lives in those three layers: it is currently the only way to score them at all.
This is the sharpest failure in layer three, and static review almost never catches it: on the architecture diagram the recovery tool and the system being recovered are two separate boxes, and nobody notices they ultimately land in the same failure domain.
The canonical shape: region A goes dark → follow the runbook → the runbook is on the intranet wiki → the wiki runs only in region A. Four steps, loop closed, stuck. There are many variants: the SaaS your incident channel depends on, the SSO you must pass to log in, the control plane that pushes config — even data centre badge readers.
The test is a single question, and you can ask it of every runbook you own: is everything that recovering X depends on — documents, tools, credentials, comms, people — outside X's failure domain? If you can't answer, you need a drill, not a review. The fix is an out-of-band path at every link: an offline copy of the runbook (an exported PDF counts), a second comms channel that doesn't ride the main stack, independent emergency credentials.
Scale a drill to company size and you scale the risk with it. The guardrail is a set of rules published in advance that every team must follow; four matter most:
There's an implicit fifth rule: a comms protocol — when to announce, to whom, and what prefix marks "this is a drill". Without it, your exercise is read as a real incident by another team and triggers a chain of real escalations.
Not every disaster can be injected for real: "a major earthquake hits the city with headquarters" can only be played on paper. So there's a ladder from fictional to real: tabletop → role-play (internally, Wheel of Misfortune, with a senior engineer as Dungeon Master) → controlled real injection → company-wide multi-day exercise.
Higher is more real, but the paper tiers carry a fatal trap: participants say what they would do rather than doing it. "I'd go look up that runbook." "I'd get X to approve." Those answers sound perfect and hide the entirety of layers three and four.
Hence the most actionable test in the chapter: whenever an answer starts with "I'd go…", make them do it now. Open that runbook. Make that call. Run that command. The facilitator can also push the other way — the senior engineer summoned to save the day is ruled to be "stuck on a subway train with no signal". That turns the drill from "we know what to do" into "we just did it".
The last link is the one most often skipped. The requirement: every finding becomes an action item with a named owner, and once fixed, the same test is re-run. Without the re-run you have a conclusion, not a regression check.
And a cultural requirement: "the drill passed" is not the goal; learning something is. An exercise where everything passes is a design failure — it means you only shook the parts you were already confident about. The political precondition lives here too: the moment findings are used to assign blame, nobody will write pessimistic assumptions into the next plan (the subject of Ch18).
The first table turns the realism ladder into a choice: which tier fits the failure scenario in front of you. What matters is being clear about what each tier cannot catch.
| Form | Catches | Misses | Cost / cadence | When to use |
|---|---|---|---|---|
| Tabletop | breaks in the decision chain, whether a runbook exists at all, who decides | all real behaviour — you get the "should" answer | an hour, any time | scenarios you can't inject (quake, city-wide power loss) |
| Role-play (Wheel of Misfortune) | debugging approach, incident command, whether docs and tools actually open | system behaviour at real scale: retry storms, cache collapse | sustainable weekly | routine training; the best value per hour for layers 3 and 4 |
| Controlled real injection | behaviour at real scale, whether alerts fire, whether recovery is safe | cross-team coordination and escalation — scope too small to trigger it | one window plus a room of people | the first verification of a single system (Ch4's method) |
| Company-wide exercise | the real result of all four layers in series: tech, process, docs, people | rare failure combinations — low cadence, narrow coverage | very costly to coordinate; a few times a year | testing whether a company-level DR / BCP promise is real |
Realism rises with cost. The common mistake is running only the two ends — a box-ticking tabletop, or one annual mega-exercise. The two middle tiers pay best.
The second table gives the method and the cost per layer under test.
| Layer | Typical single point | How to test it | Cost / difficulty |
|---|---|---|---|
| 1 · Technical components | a dependency called redundant that quietly became a single point | automatic injection, run continuously (Ch3, principle four) | needs an orchestration platform and observability first |
| 2 · Process & permissions | a single approver; a rota inside the outage domain | don't tell the approver — time how long escalation takes | exposes organisational problems; most political resistance |
| 3 · Documents & tooling | dead runbook links, stale commands, channels in the same domain | open it on the spot; never accept "I'd go look" | near-zero cost, yet skipped most often |
| 4 · People | tacit knowledge with a bus factor of 1 | "disable" that person in the drill; no answers allowed | easily read as personal; be explicit that the system is the target |
Concretely:
This chapter gives you three sentences that can change the outcome of a DR review. One: RPO / RTO only promise layer one; company recovery is all four layers in series. Two: nothing that recovering X depends on may live inside X's failure domain. Three: whenever an answer starts with "I'd go…", make them do it now.
Asked in an interview how you do disaster testing, "we run a quarterly failover drill" is the component layer. "Process, runbooks and key people are also under test; we agree in advance that real incidents win and that every test has a revert plan; findings are tracked to a re-run" is another level — it shows you know there are four layers.
Evidence from industry
1 · The core claim: technical redundancy is mostly fine, and the real single points sit in process, permissions, documents and people — so the thing under test must widen from components to the whole company's ability to recover.
2 · The four-layer model: technical components → process and permissions → documents and tooling → people. Investment and test coverage cluster in layer one; the chance of breaking rises as you go down. RPO / RTO only promise layer one.
3 · The sharpest class of finding is a circular dependency in the recovery path: the runbook on the intranet wiki, the wiki in the dead region. Static review misses it; only removing the region exposes it. The test: is everything recovering X depends on outside X's failure domain? The fix is an out-of-band path at every link.
4 · Test human-shaped SPOFs directly: "disable" the person who knows most and forbid them to answer — Google's own wording says exactly this.
5 · Four rules of engagement: real incidents outrank drills, a mandatory revert plan, a stated impact expectation, cross-functional review with VP-level sign-off for high risk — plus a comms protocol, or your drill gets read as a real incident.
6 · The realism ladder: tabletop → role-play (Wheel of Misfortune) → controlled real injection → company-wide exercise. Don't run only the two ends; the middle two pay best.
7 · The most actionable line: whenever an answer starts with "I'd go…", make them do it now — the entire value of a paper exercise hangs on that move.
8 · Findings need owners and a re-run of the same test; "everything passed" is a design failure; and the moment drills feed blame, they degrade into theatre.