BOOK DEEP READ · SRE · CHAPTER 9
Site Reliability Engineering · Ch 9 · Max Luebbe · Google · 2016
The apps on your phone get bigger and slower with every update. That is not your imagination. Software has a natural drift: it only ever gets more complicated, because nobody is paid to make it simpler. Chapter 9 of Google's SRE book is about fighting that drift, and its position fits in one line: the biggest reason systems break is not failing hardware or heavy traffic — it is that nobody can understand them anymore.
Think of a system as a house someone has been extending for ten years. Year one, it was pleasant. Then came a partition wall, a hidden cable run, a spare socket left in "just in case", a backup water pump that never runs. Ten years on it is the same house, but changing a lightbulb starts with a guess: where does this wire go? Will this wall hold?
Software drifts faster than houses, because extensions are free. Adding a feature earns praise; deleting old code earns nothing and carries the risk that someone, somewhere, still depends on it. So everyone adds and nobody subtracts.
Three reasons. First, people are attached to what they wrote — deleting it feels like admitting months of wasted work. Second, the "we might need it someday" paths never run, so on the day you finally need one, it has usually been broken for years and nobody knew. Third, changing a hundred things and shipping them together: when something goes wrong afterwards, all hundred are suspects.
① Separate "this is genuinely hard" from "we made it hard". Delivering a page to your phone quickly and correctly is hard in itself; you cannot escape it. But the three layers of wrapping added to work around one tool's quirks are difficulty we manufactured — that part can go, and should.
② Code is a liability, not an asset. Every extra line is one more thing somebody must read, test, and possibly be woken up at 3 a.m. to reason about. So on a project that is already finished, the best change is often a negative one — delete more than you add. Deleting is safe: version control keeps a copy, and you can fetch it back if you ever really need it.
③ Ship a little at a time. Release a hundred changes at once and you cannot tell which one hurt; release one and it is obvious what to undo. Frequent releases are not about speed — they are about having few suspects when something breaks.
In 2012 a US trading firm rolled out a new version and missed one of its eight servers. That server still carried a piece of code retired eight or nine years earlier and never deleted, which the new version woke back up. Forty-five minutes later, more than four hundred million dollars were gone. Deleting what no longer runs is not tidiness — it is defusing.
Your reliability ceiling is how much you can still understand: tell the difficulty the problem forces on you apart from the difficulty you added, and peel the second kind away; treat code as debt rather than achievement, so you dare to delete; change one thing at a time so you can tell who did it. One honest cost: subtraction is nearly always harder in the moment and wins no credit, and it pays off only at some outage six months later — so it has to be explicitly valued, because willpower alone will not sustain it.
Want the criteria, the comparison tables and the diagrams? → switch to the deep read
Chapter 9 turns a slogan into working engineering criteria: reliability is capped by complexity, so managing reliability is largely managing complexity. It hands you four levers — separate essential from accidental complexity, treat code as a liability (the "negative lines of code" metric), keep APIs minimal and systems modular, and release in small batches. The counterintuitive lesson: in software, "boring" is high praise — you do not want an interesting system, you want a predictable one you can still read.
Written by Max Luebbe (Google). This is the closing chapter of Part II, "Principles": it follows Ch 8 on release engineering (how to move changes safely) and leads into Part III, "Practices" (monitoring, on-call, troubleshooting). It pairs with Ch 5 on toil — toil is the bill complexity sends to humans. In the real world it maps onto technical-debt work, dead-code cleanup, API review, flag retirement, and the perennial argument about whether to split a service one more time.
Software systems are inherently dynamic and unstable: only a system that never changes can be perfectly stable, and such a system has stopped producing value. So teams add every day — and complexity has an ugly property: it gets added in and rarely taken out. The reason is incentives, not technology. Adding a feature is visible work; deleting code earns no credit and carries the risk that someone still depends on it.
Scale amplifies the imbalance. Google's published figures for 2016: over 2 billion lines in a single repository and roughly 40,000 commits per working day (Potvin & Levenberg, CACM 2016) — at that size nobody understands the system by reading it, and without deliberate subtraction complexity compounds.
Not fixing it costs you three things. Failure rate: the SRE book's introduction puts roughly 70% of outages down to changes in a live system, and the more complex the system, the less predictable each change is. Response speed: on-call cannot build a mental model, so troubleshooting degrades from reasoning to guessing. Silent risk: the old paths and fallback logic that never run go unverified year after year, and get woken up at the worst possible moment.
The chapter opens by naming the tension: software can only be perfectly stable if it never changes, and software that never changes has no business value. So SRE never asks for "stop changing things". It asks that the complexity each change brings in becomes visible and negotiable — what does this feature buy? How many extra execution paths, how many more components that must now evolve together?
The chapter leaves room for exploration: put experiments in a sandbox, bounded in scope, in time and in blast radius. Try things in immature ways, but do not leave the prototype inside a load-bearing wall.
"Boring" is an insult almost everywhere else and high praise in software: you do not want your program to be creative or surprising, you want it to stick to the script and predictably accomplish the business goal. Every "interesting" behaviour is, by definition, one you did not anticipate.
Which complexity earns its keep? The chapter borrows Fred Brooks' distinction: essential complexity is inherent in the problem definition and cannot be removed; accidental complexity comes from how we implemented it and can be engineered away. The book's own example is concrete: serving web pages quickly and correctly is essential complexity; if the team writes that server in Java and then spends half its energy fighting garbage-collection pauses, that part is accidental.
Hence two SRE duties, one forward-facing and one backward-facing: push back when complexity is being introduced into a system you are responsible for (make the proposal state its benefit instead of waving it through), and go find and remove accidental complexity in the systems you already run. Note the calibration: the goal is not fewer features, it is the same features with a more boring implementation.
The chapter uses a blunt heading for the human obstacle: engineers get attached to code they wrote, and deleting it feels like conceding that months were wasted. The chapter asks for different bookkeeping: code is not an asset, it is a liability — every line must be read, tested and understood by whoever is paged at 3 a.m., while its value comes only from what it is doing right now. On that ledger, code that no longer does anything is pure liability.
So why the hesitation? Fear of needing it later. The chapter's answer is direct: version control makes deletion reversible — pull it back out of history if you ever truly need it — whereas every piece of dead code left in a production binary is an irreversible risk. That is not fastidiousness; it has an extremely expensive precedent (see Knight Capital below).
Following from that, the chapter offers a pointed metric: on a project that is already complete, the ideal change is a negative one. The reasoning is hard to argue with — every line added or changed is an opportunity to introduce a defect; a smaller project is easier to understand, easier to test and usually carries fewer bugs. "I wrote 3,000 lines this week" is worth far less celebration than "I deleted 3,000 lines and lost no functionality".
Note that it is a metric, not a target to be scored on. Its real use is a moment of self-calibration — before you commit, ask whether this requirement could be met by deleting something instead.
The chapter quotes Antoine de Saint-Exupéry as a design rule: perfection is attained not when there is nothing more to add, but when there is nothing left to take away. For APIs that means: fewer methods and fewer arguments make an interface harder to misuse, and leave you the budget to make the survivors excellent — documentation, tests and compatibility promises all scale linearly with surface area. A library exposing 40 methods will never have all 40 well tested or indefinitely compatible.
Modularity is the same principle at system scale: clear boundaries and loose coupling are what keep a change local. The payoff lands directly on reliability — only a change that is local can be rolled back locally. The chapter extends this to data formats: Google uses protocol buffers precisely because they are designed for backward and forward compatibility (old code reads new data, new code reads old data), letting programs and data formats evolve independently.
The last lever is operational: simple releases beat complicated releases. The argument is one sentence — it is far easier to measure the impact of one change than the impact of a batch. Ship 100 unrelated changes together, watch latency degrade afterwards, and working out which one did it is expensive: usually you can only bisect by rolling back and re-testing.
Put numbers on it: bisecting 100 changes takes about log₂100 ≈ 7 rounds of "roll half back, observe, split again", and each round costs tens of minutes from deploy to stable metrics — half a day gone, with users eating the degradation throughout. Ship one change and the suspect set is 1; the rollback is a single step. The chapter likens this to gradient descent: take a small step, see whether it improved or degraded things, decide the next one — take steps too large and you no longer know which way you are going.
Table 1 · Essential vs accidental complexity: how to tell, what to do
| Essential | Accidental | |
|---|---|---|
| Comes from | the problem definition itself | the language, tools and history we picked |
| The one question | Swap in a different implementation — is it still there? | Swap in a different implementation — does it disappear? |
| The book's example | serving pages fast and correctly | the tuning and workarounds bred by GC pauses |
| What to do | own it: document, test, wrap it in a good abstraction | peel it: delete dead paths, drop shims, pick a better tool |
| Cost of misjudging | "simplify" it away → you cut a real requirement, and the outage returns in another shape | "own" it forever → you carry it for life, and it keeps growing |
Table 2 · Four everyday battlegrounds: the temptation, the simple move, the honest cost
| Battleground | Temptation | What this chapter leans toward | Honest cost / when to invert |
|---|---|---|---|
| Add a flag | "add a switch, run both ways" | a flag is temporary scaffolding: set its removal date when you add it, delete the branch once rollout completes | removal takes discipline; flags earn their keep during rollout and for risky features you may need to kill fast — but give them an expiry |
| Build a generic abstraction | "there'll be a second and third caller later" | hardcode first, abstract when the second caller actually shows up; keep the interface small, the implementation thick | you refactor once when caller two arrives; once an interface has been promised externally, abstracting late costs more |
| Write a fallback | "a backup path makes it more reliable" | spend the effort making the primary path more reliable; if a backup stays, it must be exercised routinely | you give up a theoretical safety net; simple degradations — read-only mode, serving cache — are still worth it, provided they run daily |
| Split one more service | "splitting decouples it and lets us ship independently" | ask whether the split follows an essential boundary or an org chart; a split you cannot roll back independently is not decoupling | it trades code complexity for network and operational complexity (timeouts, retries, orchestration); with genuinely clashing release cadences, splitting is right |
Table 3 · Release batch size (same total volume of change)
| 50–100 changes per release | 1–5 changes per release | |
|---|---|---|
| Suspects when it breaks | 50–100 | 1–5 |
| Diagnosis | bisect ≈ log₂n ≈ 6–7 rounds, tens of minutes each | look at the last one — 0 rounds |
| Rollback granularity | the whole batch, taking 99 innocent changes with it | exactly the offending change |
| Fixed cost per release | amortised, so it looks cheaper | multiplied 10–100×; automation must drive it near zero |
| Precondition | the fallback when releasing is expensive and gated by humans | depends on Ch 8: hermetic builds plus an automated pipeline |
Table 4 · Evidence to collect before deleting (the chapter urges deletion; deletion still needs proof)
| Method | What it proves | Blind spot |
|---|---|---|
| Static reference analysis | nothing in the repository calls it | reflection, config-driven dispatch and cross-language or cross-repo calls stay invisible |
| Runtime sampling / coverage | it was not executed during the observation window | a short window misses quarterly jobs and disaster-recovery paths |
| Turn it off, then delete | nobody complained while it was off, so nobody uses it | needs a flag and a waiting period, and must be reversible instantly |
| Announcement + migration window | external callers had time to move | only works for callers you know about — a public interface always has users you don't |
The chapter's claims have public evidence on both sides: Meta built a system to automate code deletion, Amazon publicly argues against writing fallbacks, and Knight Capital demonstrated with four hundred million dollars what keeping code that no longer runs can cost. Day to day it turns into three questions — in design review, "what does this layer buy?"; in code review, "could deleting something satisfy this requirement?"; in release review, "if this batch breaks, how do I attribute it?"
① In one line: reliability is capped by complexity, so managing reliability is managing complexity — half of SRE's job is deliberate subtraction.
② Only a system that never changes can be perfectly stable, and that system has no value — the aim is not frozen change but change whose complexity cost is visible and negotiable; sandbox experiments with bounded time and blast radius.
③ In software "boring" is praise; separate essential complexity (still there after you swap implementations) from accidental (introduced by the implementation, removable), and do the two SRE duties: block the new, delete the old.
④ Code is a liability, not an asset: code that no longer executes is liability only. Version control makes deletion reversible; dead code left in production is not — Knight Capital proved it in 45 minutes and $460M+.
⑤ "Negative lines of code" is a good metric: on a finished project the ideal change removes more than it adds — but it is a compass, and it breaks the day it becomes a KPI.
⑥ Minimal APIs: perfection is what's left when nothing more can be taken away. Modularity keeps changes local and therefore rollbacks local, and data formats need the same evolvability (why Google uses protocol buffers).
⑦ Small batches buy attribution: 100 changes at once means seven rounds of bisecting; one at a time means a suspect set of 1. It only works if releasing is automated and nearly free (Ch 8).
⑧ Delete with evidence: static analysis, runtime sampling, turn-off-then-remove, migration notices — blind deletion is an outage source too (Hyrum's law).