BOOKS DEEP-READ · SRE · CH 33
Site Reliability Engineering · Ch 33 · Google · 2016
You have flown on a plane, maybe been in an operating room, and you may live not far from a power station. These fields share one thing: one bad day costs lives. They have spent a century working out how not to have one, and turned it into a craft with rules. This chapter does something simple: it lays those practices on the table and asks which ones software can copy, and which it cannot.
Lifeguards at a pool rotate positions every ten or fifteen minutes. Not because they are tired — they are sitting still. They rotate because of a very plain finding: stare at the same patch of water long enough and you start looking without seeing. And a drowning person usually doesn't shout or splash; they just go quietly under.
Notice the shape of that fix. It does not ask lifeguards to concentrate harder. It treats "attention will drift" as a design condition and builds the roster around it.
Software is a young field, and the price of being young is having no elders. Every rule in flying and in power generation has a real casualty behind it, and those lessons were written into procedures the whole industry must follow — you get the lesson without having to crash yourself.
Software doesn't work that way: every company crashes on its own, then treats the crash as confidential. The same hole has been dug thousands of times in thousands of companies.
Gather the practices up and four things are left.
One, rehearse the disaster until it is routine. Submarine crews drill constantly, and the drills are deliberately made harder than the real thing; when the real day comes, the body already knows what to do before the mind catches up.
Two, after something breaks, ask only what conditions made it possible — never who did it. This isn't generosity, it's arithmetic: the only source of truth is the mouth of the person who was there, and the moment you start punishing people for telling the truth, that mouth closes. Aviation went furthest: reports go to a third party unconnected to the regulator, so the regulator cannot get hold of them to punish anyone.
Three, give the repetitive work to machines, but deliberately keep some of it by hand. This is the counterintuitive one: when automation is too good, people slowly stop being able to do the job — and the rare situations machines can't handle are exactly the ones that need a practised pair of hands. Aviation has paid dearly for this lesson.
Four, decide by data and by rules written in advance, not by who is most senior. The middle of an incident is no time to invent a process; who has the final say must be settled before the bad day.
Software has been copying all along, and most people don't notice they're copying: on-call, drills, postmortems, incident commanders — software invented none of them. In return, software holds two cards nobody else has: it can usually be undone within minutes, and it usually doesn't kill anyone. Those two cards are the only reason software dares use a tool that explicitly permits some failure.
And the honest cost: because it can always be fixed straight away, the industry has picked up the habit of shipping first and thinking later — and once software starts driving steering wheels and infusion pumps, that habit stops being good enough.
Fields where mistakes kill don't get there by asking people to be more careful. They treat "attention will drift and people will err" as a design condition: rehearse disaster in peacetime, chase conditions rather than people, hand repetition to machines while deliberately keeping some by hand, and decide by rules written in advance. Software can copy all of that — but not "budget some failure", which only works because failure here is reversible and not fatal.
Want the mechanisms, the five-industry comparison and the diagrams? → Switch to the deep read
The claim of this chapter is that almost nothing in SRE was invented by software. The author interviewed engineers who came to Google from nuclear power, commercial aviation, railway signalling, telecoms, manufacturing, finance and even pool lifeguarding, and pulled out four themes that recur across all of them — preparedness and disaster testing, postmortem culture, automation, and structured decision making. The genuinely valuable part is the reverse conclusion: software's two unique conditions are that failure is usually reversible and usually not fatal, which is the only reason it can use a tool like the error budget that explicitly permits failure — and that one thing no other industry can ever copy.
1 − SLO slice of permitted failure (Ch3); ship while it lasts, freeze when it's spent.99.999% availability — about 5 minutes of downtime a year, the traditional telecoms benchmark.This is the first chapter of Part V, "Conclusions", and the only chapter in the book that looks away from Google: the preceding thirty-odd chapters describe how we do it, this one asks how everyone else does it and what makes us think we're right. It follows Part IV (SRE as an organization) and leads into Ch34. Its real-world counterparts are the things other fields codified decades ago: functional-safety standards (IEC 61508, ISO 26262), aviation's crew resource management and confidential reporting, medicine's surgical checklists.
An uncomfortable question: is SRE a set of general engineering principles, or a local solution one company improvised in one era?
You cannot settle that from inside Google. With a sample of one company, any "this works well for us" might just be the surviving practices congratulating themselves. There is only one way to check — go and look at industries whose stakes are far higher and who have been betting for a century: if they independently grew the same practices, those practices are more likely to be principles; if they do the opposite, you should ask what makes you the exception.
The gap in stakes is worth stating plainly. A global outage of a web service affects hundreds of millions of people, burns a lot of money and makes headlines — but almost never kills anyone. One mishandled airliner kills hundreds. One reactor accident is measured in decades and hundreds of square kilometres. The same word, "reliability", is not the same quantity on both sides.
What happens if you skip this? The industry keeps using "we're different" as a permanent exemption — and as code starts controlling steering wheels, infusion pumps and power grids, that excuse is expiring fast.
An organizational focus on safety. In nuclear power and railway signalling, nothing is switched on until a safety case has been filed — a positive argument for why this thing will not hurt anyone, reviewed by a party independent of whoever built it. The sharp part is which way the burden of proof runs: not "you may ship because nobody found a problem" but "you may not ship because you haven't argued it is safe." Software's nearest equivalent is the production readiness review (Ch27), but our default direction is still the former.
An obsession with detail. Operating discipline on a nuclear submarine is beyond what software imagines: every valve movement logged, handover walked through item by item, orders repeated back. Behind it sits one simple mechanism — what wasn't written down didn't happen. In software: incident timelines, change logs, on-call handover notes. Their value isn't that you can look them up later, it's that writing them forces you to think the thing through once.
Putting human failure into the roster. The lifeguard rotation is the prettiest example in the chapter: it doesn't ask for more focus, it concedes that attention decays and pays for it with rotation. The second rule is just as valuable: if you suspect someone is in trouble, go in immediately, and there is no penalty for being wrong — the cost of a false positive is deliberately driven to zero to push false negatives towards zero. The software version is "escalate early, wake people early", and it sits in real tension with Ch6's "alerts must not be noisy": automated alerting should suppress false positives, human judgement should encourage them. Not a contradiction — the first spends an attention budget, the second spends one moment of mild embarrassment.
Rehearsing disaster. Drills in high-stakes fields share two traits: they are frequent, and they are deliberately harder than reality.
What deserves copying from DiRT is not "cut a datacenter" but its rules of engagement. There is also a variant few dare run — cut the person too: temporarily "disable" the veteran who carries the undocumented knowledge and see whether the team still runs. It is the only way to find out whether that knowledge really made it into the docs.
Blameless postmortems are already consensus in software (Ch15). What this chapter adds is how far other industries institutionalized it.
Aviation's move was to take the collection channel out of the regulator's hands: pilots report their own errors to a third party with no enforcement role, and the de-identified reports are used only for analysis and publication. That is far harder than "our team culture is forgiving" — it relies on nobody's goodwill, because the path from "telling the truth" to "being punished" has been cut structurally.
Medicine supplies the counter-example: a long-running culture of blame meant adverse events were systematically under-reported, and under-reporting means you don't even know how many there are, so improvement has nowhere to start.
Software is clearly behind here, and not inside companies — between them. An aviation accident report is read across the whole industry; software postmortems mostly stop at the company boundary. This is the most glaring gap the chapter exposes.
The lesson from the manufacturing side is straightforward: standardize repetitive actions, then automate them, and watch process variation statistically — the same thing Ch5 says about toil.
But the chapter offers a strongly counterintuitive lesson in the other direction: the nuclear navy deliberately does not automate everything it could. The first reason is hands: an operator who hasn't done it personally in years won't do it well on the day it matters. The second is subtler — situational awareness: someone who operates by hand carries a live mental model of the system's state, while someone watching an automated dashboard often notices only once the deviation is large.
Aviation paid for this in public. Autoflight made routine cruise extraordinarily safe, but created a structural problem: automation disengages exactly when it can't cope, and the situations it can't cope with are the hardest to fly — so the hardest task lands on the least practised hands. Air France 447 (2009, 228 dead) went that way: iced pitot tubes disengaged the autopilot and the crew held the nose up through the stall warning. This case isn't from the book; it is the chapter's claim corroborated in a public accident report (French BEA final report, 2012).
So the test is clear: automation should eat what is frequent and known, and should not monopolize what is rare and requires judgement. The latter needs people in the loop and opportunities to practise — which loops straight back to theme one.
Decisions in high-stakes fields share three traits: driven by data rather than instinct, recorded so they can be reviewed, and escalation paths settled in advance. Manufacturing's define–measure–analyse–improve–control loop is essentially a way of turning "making the call" from a personal talent into auditable steps.
Software's two best counterparts appear earlier in the book without being named as the same species: SLOs turn "can this ship?" into something you can compute (Ch3, Ch4), and the incident command system fixes "who decides" before anything breaks. What they share is moving judgement out of the incident and into the time before it — the people in the middle of it have neither normal adrenaline nor normal information.
The anti-pattern is everywhere in software: treating a senior engineer's instinct as the sole basis for a decision. It is fast and accurate in familiar territory, but instinct grows on historical samples, and rare situations are exactly where those samples are thinnest — instinct fails most easily at the moment you lean on it hardest.
This chapter is not telling you to import nuclear procedure into software. The real question is: where do you sit on the two axes of cost-of-failure and rate-of-change? Different positions leave you entirely different tools.
Table 1 · Five industries' reliability recipes: the one thing that transfers, the one that doesn't
| Field | Main instruments | Transfers to software | Doesn't transfer |
|---|---|---|---|
| Nuclear | Safety case + layered independent defences + frequent drills | Reversed burden of proof: no argument for safety, no go | A design you can freeze for decades |
| Aviation | Checklists, crew resource management, third-party confidential reporting | Separating the reporting channel from enforcement | Licensing and mandatory industry-wide procedure |
| Medicine | Checklists, review (long held back by a blame culture) | A sheet of paper can save lives | Every "system" is different |
| Telecom · Rail | Formal verification, the 99.999% benchmark, redundancy | Critical paths are worth formal methods | Protocols stable for decades are what make that verification affordable |
| Web services | SLOs, error budgets, canaries, fast rollback | — | Its two cards: reversible + non-fatal |
The second trade-off: when should you follow procedure, and when should you improvise? High-stakes fields lean overwhelmingly on the former and software cannot copy that outright — a real share of software failures are unprecedented, and following procedure through something unprecedented only gets it wrong more tidily.
Table 2 · Procedure vs improvisation: split by "have we seen this before", not by importance
| Situation | Which one | Why | Cost |
|---|---|---|---|
| Known failure with a playbook | Strict procedure | Judgement at 3am is worse than written steps | Playbooks expire; procedure dulls thinking |
| Unprecedented failure | Roles by procedure, actions improvised | No steps to follow; but "who decides" must be fixed | Heavily dependent on who is on call |
| Frequent low-risk change | Give it to automation | Frequent and known is the machine's home turf | People lose their hands on that path |
| Rare high-risk operation (failover, restore) | Procedure + two-person check + drills | Rare means nobody has the hands for it | Drills cost, and may genuinely break production |
The empirical value here is that the provenance of software's proudest practices is all traceable. The incident commander comes from firefighting and emergency response; the release checklist's ancestor is the pre-flight checklist; the institutional prototype of the blameless postmortem is in aviation; Chaos Monkey and DiRT are software implementations of the old rule that you rehearse disaster. The practical payoff: when you're stuck on one of these, go and read how that industry got past it.
1.5% to 0.8% and complications from 11% to 7%. One sheet of paper, essentially free. This is the strongest outside evidence that checklists are not bureaucracy.Haynes et al., New England Journal of Medicine, 2009 (no verifiable link retrieved on this run, so the citation is text only)1 · The core claim: almost nothing in SRE was invented by software; five unrelated high-stakes industries converge independently on the same four themes, which is what makes them look like principles rather than habits.
2 · The four themes: preparedness and disaster testing, blameless postmortems, automation, structured decision making — the first carries the most weight.
3 · Preparedness at heart means treating "attention drifts, people err" as a design condition: lifeguards pay for attention decay with rotation, and drive false negatives near zero by making false positives free.
4 · Nuclear and rail reverse the burden of proof: no argument for safety, no go. Software's production readiness review is a weaker version of the same move.
5 · Two rules of engagement are the ones to copy: real emergencies outrank drills, and every drill ships with a rollback plan; the bolder variant is to cut out the person carrying the undocumented knowledge.
6 · Blamelessness institutionalized means separating the reporting channel from enforcement (aviation's third-party confidential reports), which is much harder than team culture; software's biggest gap is that nothing is shared between companies.
7 · The automation test: eat what is frequent and known, don't monopolize what is rare and needs judgement; structured decision making is moving judgement out of the incident and into the time before it (SLOs, fixed incident roles).
8 · The one thing that doesn't transfer is the error budget — it presupposes reversible, non-fatal failure. Software is moving fast into a new quadrant of high cost and high rate of change, which was nearly empty when this chapter was written.