BOOKS DEEP-READ · GOOGLE SRE · CH 18
Site Reliability Engineering · Ch 18 · Google · 2016
When you check out on Black Friday and the site doesn't fall over, something unglamorous made that possible: months earlier, somebody worked out how many machines to buy and which buildings to put them in. That job is called capacity planning. Chapter 18 of Google's SRE book is about how they handed that chore to a program.
The whole book is about running production systems well. This one chapter suddenly changes register: it's about the operations team writing a real software product. That sounds like a distraction — why should the fire brigade open a factory? Because some chores take a week by hand and are stale the moment you finish. Only by turning one into software does that week of effort accumulate into anything.
The old way was an enormous spreadsheet: this service gets this much in that building, cell by cell, then a round of meetings to sign it off. The trouble is that the sheet records conclusions, not reasons. Change one assumption — growth came in faster, a batch of machines is two months late, a launch slipped — and the whole sheet has to be filled in again. The previous week's work is gone. Tired people start padding their numbers, because too many is safer than too few. That's how the money burns.
The idea is plain. When you brief a builder, "three lights in the living room" is worse than "the living room must be bright enough, the switch goes by the door, and the whole job stays under budget." The first one has to be rewritten the moment the house changes; with the second, the house can change and not a word of your brief moves — the plan recalculates itself.
Google did exactly that. Instead of handing over "50 machines in building A," they wrote down the conditions: this service must absorb the traffic we expect, it must survive losing a whole building and still keep two spare copies, it must sit near its users, and the total bill should be as small as possible. Then a program that is good at puzzles takes over: it treats every condition as a rule that must hold at once, and picks the cheapest arrangement inside those rules. When an assumption changes, you re-run it — a new plan in minutes, and nobody fills in a sheet.
One honest caveat: if you leave a condition out, the program will not warn you. It will simply give you the cheapest answer in the gap you left — forget to say "keep spare copies" and it will keep none. So the human work doesn't disappear. It moves from filling in cells to thinking clearly about what you actually require.
Rather than recomputing "how many machines" by hand, write down why that many and let a program do the arithmetic. Answers go stale; conditions don't. And the reason the ops team should write this software themselves is that only the people who use it daily know how to state the conditions.
Want the mechanisms, the notation and the diagrams? → Switch to the deep read
You'd think SRE writes software to do less work. Really it's so that operations finally has an output that compounds. This is the one chapter in the book that isn't about running systems but about an ops team building a product: using Google's internal capacity-planning tool Auxon as its case study, it argues for replacing hand-picked resource numbers with intent-based capacity planning — engineers declare constraints and goals, and a solver computes the allocation.
This chapter sits in the middle of Part III, "Practices": after Ch17 on testing for reliability, before the frontend and datacenter load-balancing chapters. Structurally it is the odd one out — everything around it is about holding production up, and only this one is about how an SRE team ships software. What it really interlocks with is Ch5 on eliminating toil and the 50% cap from Ch1: this chapter answers "what is the other 50% actually for?"
Traditional capacity planning is a four-step loop: gather demand forecasts → build an allocation plan → review and sign off → deploy and configure the resources. Its artifact is a large table of concrete numbers: "service Foo gets 50 cores in cluster X, 30 in Y."
The trouble is that this table stores the conclusion, not the reasoning. The moment any input moves — the forecast is revised, a machine delivery slips, a dependency gets heavier, a launch shifts a quarter — the whole table has to be recomputed, and recomputing it means spreadsheets plus a round of cross-team alignment: days to weeks per iteration. The book's word for this is brittle. It isn't that the process is slow; it's that every change costs a full restart.
Two consequences follow. The plan is stale as soon as it's finished, so teams pad for the worst case and burn money on idle machines; and the workload grows linearly with the number of services — order-of-magnitude, one mid-size service across a few dozen clusters with a dozen dependencies is already hundreds to thousands of cells, times every service. That is the definition of toil, word for word.
The central concept is intent: the requirements and constraints that make a plan valid, rather than the numbers those requirements produce. A traditional plan ships the binary and discards the source; intent-based planning ships the source, and the binary can be recompiled at any time. The book uses a ladder to show that "intent" is a spectrum, not a switch:
| Rung | What you hand the system | What it must figure out | Cost / risk |
|---|---|---|---|
| 1. Exact resources | "Foo gets 50 cores each in clusters X, Y, Z" | Nothing — just execute | Most brittle: one cluster or forecast change voids it |
| 2. Relaxed placement | "Foo gets 50 cores across any 3 clusters in region A" | Which clusters are in A, and what's free | The quantity is still hand-picked |
| 3. Declare the goal (the book's sweet spot) | "Let Foo absorb region A's forecast demand at N+2" | Demand-to-resource ratios, the dependency graph, redundancy semantics | A wrong ratio propagates fleet-wide |
| 4. Declare reliability | "Run Foo at 99.999% availability" | A full mapping from reliability to resources | That mapping is still immature — the book calls this out of reach |
The judgement is refreshingly concrete: rung 3 is the sweet spot — abstract enough for a machine to take over, specific enough to actually solve. Rung 4 sounds best, but too many links are missing between "five nines" and "this many cores, placed here," and the chapter explicitly marks it as not yet achievable. That is the most portable lesson here: raise intent to the "demand plus redundancy" level and stop; don't chase a one-sentence answer to everything.
"Declare your intent" is empty advice on its own, so the chapter decomposes it into three mandatory inputs. Miss one and the machine can't turn:
Auxon is the capacity-planning tool Google's SREs wrote for themselves, and it is the chapter's single case study. Its shape is a compile-and-solve pipeline:
minimize Σ price·alloc subject to inequalities like "usage in each cluster ≤ supply" and "replicas ≥ demand + 2," with alloc forced to whole numbers because you can't run 2.7 replicas. In plain words: write every hard requirement as an inequality, attach a "cheaper is better" objective, and let the solver search an enormous space for the best arrangement.Two properties explain the design. First, a changed input only needs a re-run; nobody recomputes by hand, and an iteration drops from weeks to minutes. Second, when nothing satisfies the constraints, the solver can point at which one is binding — far more useful than a spreadsheet that won't close, because it turns "we're short" into a negotiable fact like "N+2 in region A is incompatible with current supply."
The honest cost: MIP is NP-hard, and at scale you cannot guarantee a globally optimal answer in acceptable time, so in practice you accept approximations and decompose the problem by region. The chapter doesn't dodge this — capacity planning is decision-making on top of an uncertain forecast anyway, and a fast approximation beats an exact answer that has gone stale.
The chapter's other half is organizational: this kind of software should come from the SRE team itself. Not because nobody else will, but for three structural reasons — SREs are both the developers and the users, which is the shortest feedback loop there is; they hold production domain knowledge no external vendor can obtain (real dependencies, real failure modes, real procurement constraints); and engineering output breaks the link between team value and headcount.
It also solves a people problem: a genuine software output gives SREs a growth path that doesn't require switching roles. But the chapter attaches a hard constraint — don't lower your standards. This software makes resource decisions for production, so it must be built to product quality (design docs, tests, named owners and a real support commitment), not as "just an internal tool."
Pin down the confusion first: autoscaling is not capacity planning. Autoscaling redistributes machines you already have, on a scale of minutes. Capacity planning decides on a scale of quarters whether to buy, how much, and where to put it. The first cannot conjure machines that haven't been delivered; the second cannot track your lunchtime peak. Two different timescales; neither substitutes for the other.
| Your situation | Verdict | Why | Honest cost |
|---|---|---|---|
| Few services, few placements (under a dozen cells) | Don't build — a spreadsheet is fine | Few enough cells that one person holds them all; a solver costs more than it saves | You'll be a step late when it grows; leave a migration path |
| Cloud-only, elastic, capacity never scarce | Don't build — autoscale plus a cost dashboard | Remove the "can't buy it" constraint and the main value disappears | The bill absorbs your overprovisioning; treat cost separately |
| Dozens of services × regions, real procurement lead times | Worth building | Cell count explodes and assumptions move constantly — the solver's home turf | Product-grade investment: design, tests, on-call |
| Scarce resources (GPU / TPU quota, private links) | Worth building | Scarcity forces explicit priorities, or contention is settled by whoever shouts loudest | Someone must own the priorities; tech can't fix politics |
| No product-grade engineering capacity | Not yet | A half-built tool making production decisions is worse than the manual sheet | You keep the toil, but it stays legible |
| Hand-built plan | Reactive autoscaling | Intent-based planning | |
|---|---|---|---|
| Timescale | Quarters | Seconds to minutes | Quarters, recomputable anytime |
| Cost of a changed input | Days to weeks | N/A | One re-run |
| Handles lead times? | Yes, by memory | No | Yes — supply is an explicit input |
| Main failure mode | Stale plans, habitual padding | Nothing left to scale into | Omitted constraint gets optimized away |
The idea is everywhere today under other names. Kubernetes' declarative API (you write desired state, controllers converge on it), Terraform's desired-state model, every cloud's reserved-instance planner — all of them are "hand over conditions, not steps." Auxon's distinction is that it writes the objective function down explicitly as "minimize total cost" and genuinely puts a mathematical solver behind it.
In an interview the use is concrete: asked "how do you do capacity planning," ask back which rung of the ladder they mean — "last year's peak times 1.5" and "declare N+2 and let the system compute" come from completely different levels of organizational maturity. Then explain why autoscaling is not capacity planning, and you're most of the way there.
1. The one chapter about an ops team shipping a product, told through Auxon, a capacity-planning tool.
2. The old disease: traditional planning ships numbers instead of reasons, so any changed assumption means starting over — brittle, and linear in the number of services, which is the definition of toil.
3. The idea: intent-based capacity planning — declare requirements and constraints, let a solver compute the allocation. Answers go stale; conditions don't.
4. Four rungs: named clusters and cores → region and total → demand plus N+2 (the sweet spot) → a bare reliability target (judged out of reach).
5. Intent needs three things: the dependency graph, the demand-to-resource ratio, and priorities. Without priorities, "infeasible" is all you ever get back.
6. Auxon's shape: five inputs → a config language compiles constraints → mixed-integer programming minimizes cost subject to them → an allocation plan; infeasibility names the binding constraint.
7. The distinction to keep: autoscaling ≠ capacity planning, five orders of magnitude apart in timescale.
8. Why SRE writes it: developer and user at once, exclusive production knowledge — but don't lower your standards; it decides things for production.
9. The biggest trap: a solver optimizes only what you wrote, so an omitted constraint is a constraint given away; the exchange rate is the quiet single point of error.
10. The echo today: Kubernetes' declarative API, Terraform's desired state, cloud reservation planners, and the GPU-quota era — same idea, different resource.