BOOKS DEEP-READ · SRE · CH 7
Site Reliability Engineering · Ch 7 · Google · 2016
Auto-updates on your phone, the badge reader at the office — that's automation. Google has a group of engineers called SREs whose job is to keep the site up, and the machines they look after number in the hundreds of thousands. Clicking through them one at a time is not an option. This chapter is the story of how, over more than a decade, they moved from "a human works through a checklist" to "the system takes care of itself" — and what they broke along the way.
You probably assume the biggest win from automation is saving time. Google's answer: no. Saving time comes last. First place goes to doing it exactly the same way every time. Ask a hundred people to do one task by hand and you get a hundred slightly different results; run a script a hundred times and you get the same result each time (even if it's the same mistake). At scale, "identical" is worth far more than "fast" — because the most expensive line item in an outage is rarely "the code was wrong", it's "this box differs from that box and nobody knows how".
Bringing up a new Google datacenter once meant an SRE working through a checklist of hundreds of steps over several weeks: install this, configure that, verify the other. A checklist is written for humans, and humans skip steps, read them differently, get tired. Worse, when it was done, nobody could state what "correct" actually looked like — all that existed was a checklist that goes stale and the memory of a few veterans.
Two moves. First, turn the checking into a program: stop asking a veteran "is this one configured right?" and instead write many small checks that verify, one fact at a time, that the thing that should be there is there — and say precisely which one isn't. Second, and this is the turn, turn checking into fixing: the program already knows what it should look like, so let it put things right.
That relies on one property, idempotence: whatever state you're in now, one run lands you in the intended state, and ten runs land you in the same place as one. Like a steering wheel that self-centers — you don't need to know how far off it is; let go and it returns. Only with that property can automation be run repeatedly, at any moment, without fear.
Following that road up is a staircase: hands → someone writes a script → the script becomes a shared tool → the tool moves inside the system and ships with it → and finally, the system notices its own problem, fixes it, and nobody gets woken. Google's cluster system Borg — the one Kubernetes learned from — stands on the top step: you say "I want five copies of this version", and it watches the rest.
One honest line: moving humans out of the operation also moves them out of understanding it. On the day the automation itself goes wrong, whoever is on call may have neither the reflexes nor a manual.
Automation isn't a labor-saving tool, it's an amplifier: it multiplies what you got right and what you got wrong, and it does so much faster than a person could. So the thing worth chasing isn't "more scripts", it's "a system that looks after itself without bolt-on scripts at all".
Want the five-level ladder, idempotent reconciliation and the diagrams? → switch to the deep read
This chapter isn't about which scripts to write; it traces a trajectory: the end state of automation isn't "a human hands the operation to a script", it's "a system autonomous enough that no bolt-on automation is needed". It ranks the reasons to automate (saving time comes last), lays out a five-level ladder, records Google's path from handwritten checklists → idempotent reconciliation → service-owned admin APIs → Borg's declarative model, and issues one cool-headed warning: automation is a force multiplier, and multiplying force does not by itself improve the accuracy with which that force is aimed.
This chapter belongs to Part II, "Principles". It follows Chapter 5 (Eliminating Toil, which establishes why to automate) and Chapter 6 (Monitoring, which establishes how you know to act), and sets up Chapter 8 (Release Engineering, automation applied to the delivery path). In today's world it maps onto Kubernetes controllers, the idempotent execution of Terraform and Ansible, and every GitOps platform — this chapter is their shared intellectual prototype.
"Manual operations don't survive scale" is easy to say, but what exactly fails? The chapter names three collapses. Consistency: people cannot repeat work identically; at dozens of machines that's fine, at tens of thousands "every box is slightly different" becomes the most expensive debt you carry into an incident. Repeatability: a checklist is natural language, so different readers produce different outcomes — and the checklist itself goes stale with nobody knowing which step stopped mattering. Auditability: manual work leaves no structured trace, so the postmortem runs on memory.
More worth copying down is how the chapter ranks the reasons to automate — putting the most commonly cited one last:
The chapter's opening judgement deserves a wall poster: automation is a force multiplier, not a panacea — and multiplying force does not naturally improve the accuracy with which that force is applied. Two counterintuitive corollaries follow. Automate a bad process and what you get is a bad process that runs faster. So the question to ask first isn't "can this step be scripted?" but "should this step exist at all?" — and Chapter 5's answer is delete, then simplify, then automate.
Which means the real subject here is not how to write automation, but who owns it and which layer it lives in.
The chapter takes a single task — failing a database master over from one datacenter to another — and lays out five levels:
1. No automation: a DBA moves the master by hand. 2. External, system-specific: an SRE keeps a failover script in their home directory. 3. External, generic: that capability is folded into a shared tool everyone uses, of which the database is just one supported case. 4. Internal, system-specific: the database carries failover itself, shipped and tested with it. 5. No automation needed: the database notices the problem, fails itself over, and nobody is paged.
The point of the ladder is not the degree of automation but the distance between the code and the system it describes. Levels 2 and 3 are the most common and the most fragile: they live outside the system, so every change to the system breaks them — and the people changing the system have no idea they exist.
Early cluster turnup was a checklist of hundreds of steps plus manual verification. SRE's first move was not a one-click turnup but checking: they extended Python's unit test framework so that each test asserts one fact about real production (does this service's config point at the right place? is the thing it depends on ready?), and chained the tests by dependency — if a prerequisite fails, the tests behind it don't run, so the error names the first broken link instead of dumping a screen of red that hides it.
Why build the checks before the actions? Because you cannot automate a state you cannot yet judge right or wrong. Checking before fixing is the first criterion on this road, and it's the step most often skipped today when people write an operator or a health sweep.
With checks in place, the next move is to let the program put right what it found: pair each check with a repair action, and require that action to be idempotent — from whatever state, one run lands in the desired state, and N runs are equivalent to one.
It looks like one less manual touch; it is in fact the hinge of the whole chapter. Correctness stops being defined as "the right steps ran in the right order" and becomes "does the current state equal the desired state". An imperative script must know which step it reached, so dying halfway makes resumption hard; an idempotent reconcile action ignores history and is safe to rerun at any time — which is precisely the kernel of the Kubernetes control loop and of terraform apply.
The next problem isn't technical. SREs write the automation, developers change the system, and the split hardens into specialization: the SRE becomes "the only person who can drive this machine", while the development team knows nothing about how its own service gets brought up. The automation then rots continuously — every small change to the service snaps the bolt-on, and repairing it starts to feel like archaeology.
The chapter's fix is to move ownership back: have each service expose its own administrative API, so that cluster turnup degenerates into calling each service's own interface in turn. Your service, your responsibility for whether it can be brought up automatically — and the automation now evolves and is tested along with the service. That is the step from level 3 to level 4, and the ancestor of today's "you build it, you run it" and the service-owned operator.
The final jump comes from the platform. Borg abstracts a pile of machines into a warehouse-scale computer: you no longer say "install B on machine A", you declare "five replicas of this version, this much CPU and memory each", and Borg handles placement, monitoring and rescheduling after failure. At this layer level 5 stops being an aspiration — self-healing becomes the platform's default behavior rather than somebody's script.
The cost lives here too: the platform absorbs complexity, it does not abolish it. You buy "operations got simpler" and you pay with "what you must understand when it breaks got deeper" — "why won't the scheduler place my job" is far harder to reason about than "which machine missed the install".
The ladder is not "higher is better" — it's "pick by rate of change and by scale". An operation you run twice a year against three machines is usually a losing case for automation.
| Level | Typical form | When to pick it | What it costs |
|---|---|---|---|
| 1 Manual | human follows a checklist | one-off, rare (1–2×/year), under a dozen machines | inconsistent, unauditable; collapses the moment scale grows |
| 2 External, specific | a script in someone's home directory | emergencies, proving an idea out | privatized knowledge; rots on the first service change |
| 3 External, generic | shared failover / deploy tooling | a class of systems with strong commonality, reused across teams | furthest from the system; doesn't hear about changes, fails silently |
| 4 Internal to the system | service-owned admin API / operator | long-lived services that change often | the service team must build and test the ops interface |
| 5 None needed | declarative platform + self-healing | when a platform exists to lean on (Borg / Kubernetes) | the platform becomes new complexity and a new single point; debugging is harder |
The second table is the same question from another angle: how do you intend to define "correct"?
| Imperative script | Idempotent reconcile (declarative) | |
|---|---|---|
| Correctness is | each step succeeded, in order | current state == desired state |
| Dies halfway | hard to resume; a human works out where | just run it again |
| Config drift | goes undetected | corrected on every reconcile pass |
| Writing it | intuitive, cheap | you must be able to judge state first; costly up front |
| Fits | genuinely one-off actions, data migrations | maintaining a desired state over time |
There's one more criterion the chapter doesn't write as a formula but that you can take straight to a decision: the value of automation ≈ rate of change × scope affected × cost per manual run − build cost − maintenance cost. The term most often omitted is maintenance, and it is exactly what kills level-3 automation in bulk.
This chapter is the intellectual prototype of a whole class of today's infrastructure. Kubernetes controllers are idempotent reconciliation industrialized: each controller keeps comparing desired against current and closes the gap — and Kubernetes is the public successor to the Borg experience. Terraform and Ansible sell the same property: apply the same description repeatedly and it converges to the same place.
The warning holds up just as well in practice: automation acts fast and wide, so when it aims at the wrong target the damage arrives at automation speed. Which is why in modern practice staged rollout + a bounded blast radius + one-click rollback aren't optional extras — they're the seatbelt that ships with the automation.
① One line: the end state of automation isn't "a human hands the operation to a script", it's "a system autonomous enough not to need bolt-on automation".
② Automation is a force multiplier, not a panacea — multiplying force doesn't aim it better; automating a bad process yields a faster bad process.
③ The ranking is counterintuitive: consistency > a platform > faster repair > faster action > saving time, with time saved dead last.
④ Five levels: manual → external specific → external generic → internal to the system → none needed; the test isn't degree of automation but how far the code sits from the system.
⑤ The route is check first, fix second: you can't automate a state you can't judge; chain the checks by dependency so the error names the first broken link.
⑥ The hinge is idempotence: correctness moves from "which steps ran" to "current == desired", so anything is safe to rerun — the kernel of Kubernetes controllers and terraform apply.
⑦ Ownership decides lifespan: automation bolted outside a system rots silently; put the admin interface inside the service and it keeps up with change.
⑧ Borg made self-healing a platform default, at the cost of absorbing complexity and raising the bar for debugging.
⑨ Automation lets failure scale too: a human wipes few disks in a day, automation wipes a whole batch in minutes — limits, confirmation and rollback are the matching seatbelt.
⑩ The decision rule: value ≈ rate of change × scope × cost per manual run − (build + maintenance); maintenance is the term people forget.