CHAPTER DEEPREAD · SRE · CH 17
Site Reliability Engineering · Ch 17 · Google · 2016
The apps on your phone update almost weekly, and the services behind them change dozens of times a day. The odd thing is how rarely an update takes everything down. Chapter 17 of Google's SRE book is about exactly that: change is unavoidable, so how do you keep change from breaking things? The answer is a whole discipline of tasting before serving.
Think of shipping a new version as a restaurant putting a new dish on the menu. You changed the recipe, so you don't send it straight to a full dining room. First you taste it yourself in the kitchen. Then you run the whole thing end to end — plating, timing, service. Then you serve it to one table by the window and watch how they react. Only then does it go out to everyone. And regardless of new dishes, every day you send someone in to order a plate pretending to be a customer, just to confirm the kitchen is still alive. Those steps map onto the kinds of checks this chapter describes.
Not changing code is the safest thing you can do, and also the end of your business. The real difficulty is that things break not because someone got a line wrong, but because of the things that only exist in production — real user traffic, real configuration, the real number of machines. Everything tastes fine in the kitchen and falls apart in the dining room. Worse, many teams have exactly one move after shipping: stare at the dashboards and hope, letting users find the problem first.
The most counterintuitive line here is this: the value of testing isn't finding a problem, it's deciding at which step the problem gets found. If a dish is too salty and the kitchen catches it, the cost is one remade plate. If the customer catches it, the cost is an apology, a refund, and your reputation. Software works the same way. A bug stopped by an automated check before release is nearly free to fix. A bug that leaks into production is a 3 a.m. page, an investigation, a rollback, and a postmortem — there goes your night.
So the lesson isn't "write more tests." It's put each gate where the cost is lowest, and be honest about what each gate misses — the kitchen can't taste the crowding in the dining room, and one table by the window doesn't speak for every customer.
A small system you can undo in a minute can afford looser gates and lean on "pull it back the moment it looks wrong." A large system that takes hours to recover has to move its gates earlier and stop bad changes before they ship. The honest cost: more gates means slower releases — and a check that passes sometimes and fails other times is worse than no check at all, because it teaches everyone to ignore red lights.
Testing isn't about proving nothing is broken. It's about making sure breakage is discovered at the cheapest possible place. Taste before serving, serve one table first, and keep sending in a fake customer — three gates, each covering a different stretch, none of them optional.
Want the mechanisms, diagrams and real systems? → Switch to the deep read
You think testing exists to find bugs. This chapter makes a sharper claim: testing is how an SRE quantifies confidence, and what it really changes is not the number of bugs but the cost of fixing them. The chapter opens by saying that one key responsibility of Site Reliability Engineers is to quantify confidence in the systems they maintain — and that reliability is a function of mean time to failure (MTTF) and mean time to repair (MTTR). The economics of testing land squarely on the MTTR side: a bug blocked before it ships has a repair time of roughly zero.
This chapter belongs to Part III, "Practices," right after the incident-handling cluster that includes Ch 15 on postmortem culture. It builds on Ch 3's error budgets and Ch 8's release engineering, and it sets up Ch 19–22 on load balancing and overload, and Ch 27 on launch coordination. In today's terms it maps onto one pipeline: CI → canary release → black-box probing, now marketed as progressive delivery and synthetic monitoring.
SREs get asked one question every single day: "can this version go out?" Past reliability can be settled by looking at monitoring history; future reliability can only be predicted. And prediction requires that the system hasn't changed — but systems change constantly. The chapter's answer: testing is the mechanism you use to demonstrate specific areas of equivalence when changes occur. Every test that passes both before and after a change removes one piece of uncertainty.
What happens if you don't solve it? Both extremes are common. One is fear of shipping: every release pulls the whole team in for manual checks, cadence falls to once a month, features pile up into one enormous change, and failures get harder to localize. The other is shipping blind: push, watch dashboards, let users be your probes. A production bug costs you alert → diagnose → roll back → postmortem, easily hours; the same bug caught by CI costs one red build and one commit.
One formulation in the chapter is worth copying down: when a system-level test applied to a subsystem detects exactly the same problem that monitoring would detect, that test lets the push be blocked so the bug never reaches production — repairing such "zero MTTR" bugs by blocking a push is both quick and convenient.
That line redefines testing from a quality activity into an economic one. You are not eliminating bugs, which is impossible. You are choosing where they get discovered, and the location sets the price.
The first family runs against an undeployed binary on constructed, known data. Three layers:
Regression testing carries one highly actionable rule: when you find a bug, first write a test that reproduces it reliably, then fix it. That rule also answers "where do I start on a legacy project with no tests?" — you don't backfill full coverage; you accumulate cases one fixed bug at a time.
The second family runs against an already deployed system, because some things exist only in production.
plan and drift detection are today's version of the same idea.On canaries the book makes a point that is routinely missed: a canary test isn't really a test; rather, it's structured user acceptance. Configuration and stress tests confirm a specific condition over deterministic software; a canary is more ad hoc — it exposes the code under test only to less predictable live production traffic, so it isn't perfect and doesn't always catch newly introduced faults. Treat it as your last line of insurance, not as proof that the change is good.
Lay the pieces out on a two-by-two and one cell is conspicuously empty. Traditional testing = known input × undeployed binary. Monitoring = unknown real traffic × deployed binary. So who owns known input × deployed binary? Probers.
The chapter names the cause of the gap precisely: the release test probably wrapped the integrated server with a frontend and a fake backend, while the probe test wraps the release binary with a load-balancing frontend and a separate, genuinely scalable persistent backend. In other words, the configuration actually running in production is one that was never tested as a whole. Google fills that cell with a tool called Prober, which runs a protocol check against a target and reports success or failure; it can validate the response payload itself — the HTML contents of an HTTP response, say — and even extract values and export them as time series.
Probers have costs too. They issue real requests against production, so you must handle write side effects (usually dedicated test accounts and disposable data), quota consumption, and the question of who is at fault when the prober itself dies — its alerts have to distinguish "the service is broken" from "the prober is broken."
① The tools SREs write are software too. Draining traffic from a cluster, restarting in bulk, reclaiming data, adjusting quota — these have a larger blast radius than most product code and frequently have no tests at all. The chapter offers a crisp rule: how thoroughly a tool is tested determines how often it is appropriate to use that tool against production. An untested tool runs only in a window with a human watching; a well-tested one has earned automation.
② How do you test disaster recovery tooling? By nature it only runs on the worst day, so you can't wait for a real disaster to validate it. The chapter's move is elegant: many disaster recovery tools can be carefully designed to operate offline and compute a checkpoint state equivalent to cleanly stopping the service, then push that checkpoint to be loadable by existing non-disaster validation tools. Turn a disaster into a snapshot you can validate offline — the single most portable trick in the chapter.
③ Speed is availability. The longer tests take, the longer the lag between check-in and feedback, and the more engineers route around them. Flakiness is the subtler poison: what an engineer actually wants to know is whether their code has an unanticipated race condition, and a test that flips red and green on its own buries that signal in noise. An intermittently failing test is worse than no test — it trains the whole team to ignore red.
All three are tied together by one control loop: as MTBF rises in response to better testing, developers are encouraged to release features faster; new bugs push release velocity back down. In practice that loop is the error budget — keep shipping while the SLO has room, stop and reinvest in testing when the budget is spent.
Table 1 · Seven checks: what each catches, what each misses
| Check | Runs where | Catches | Misses / cost |
|---|---|---|---|
| Unit | undeployed, milliseconds | logic errors in one function or class | blind to all interaction between components |
| Integration | undeployed, seconds | contract errors once components are assembled | dependencies are mocked, so real downstream behaviour slips through |
| System | undeployed, full system | end-to-end behaviour, perf drift, old bugs returning | slow; still fake data and fake topology |
| Configuration | production, continuous | live config diverging from version control | config that is "correct" but semantically wrong still passes |
| Stress | production or staging | capacity and degradation limits (full disk, overload) | consumes real resources; overdone, it becomes your outage |
| Canary | production, 1% traffic | problems that only real traffic exposes | the book is explicit: not a test, doesn't always catch new faults; costs bake time |
| Prober | production, 24/7 | is the combination that actually runs still alive? | covers only the few explicitly asserted paths; write side effects |
Table 2 · Where to spend: start from your MTTR
| Spend on "stop it before release" | Spend on "pull it back fast" | |
|---|---|---|
| Fits | rollback takes tens of minutes to hours: stateful services, data migrations, schema changes, client / embedded releases | rollback in seconds to minutes: stateless services, feature flags, instantly switchable frontends |
| Main levers | a thicker test pyramid, configuration tests, pre-release stress tests | canary plus automated judgment, feature flags, one-click rollback |
| Cost | release cadence is pinned by test duration; the suite itself needs maintenance | some real users always act as probes; demands mature monitoring and automation |
| Failure mode | tests too slow → bypassed; flaky → ignored | canary signal too weak or bake time too short → it blows up after widening |
These aren't alternatives; they're two sides of one equation. Reliability is a function of MTTF and MTTR, and you can buy it from either side. The actual mistake is funding neither and trusting luck.
Almost every modern delivery practice has its prototype in this chapter: layered CI tests map onto Bazel and GitHub Actions pipelines; configuration tests onto Terraform plan and drift detection; canaries onto Spinnaker, Argo Rollouts and Flagger; probers onto Prometheus blackbox_exporter, Datadog Synthetics and every synthetic-check product. Reading this chapter gives you the manual for why those tools look the way they do — and lets you answer "how do you make a release safe?" starting from MTTR rather than from a list of tool names.
4.2 million individual tests running continuously and about 150 million test executions per day — an average of 35 runs per test per day — with 99% of executions passing. That is the hard evidence behind "speed is availability": at this volume, suite duration and flakiness directly set how fast an engineer gets feedback. A. Memon et al., "Taming Google-Scale Continuous Testing," ICSE-SEIP 2017 ↗30% of Netflix's production canary judgments, averaging 200 judgments a day — confirming the book's framing that a canary is user acceptance, needing statistical judgment rather than an assertion. Netflix Tech Blog, "Automated Canary Analysis at Netflix with Kayenta," Apr 2018 ↗① In one line: testing is how SREs quantify confidence; it changes not the number of bugs but where they are found, and therefore what they cost.
② Core claim: reliability is a function of MTTF and MTTR; a bug blocked before release has near-zero MTTR, and fixing it by blocking a push is quick and convenient.
③ Traditional tests (undeployed + known data): unit → integration → system, with system splitting into smoke, performance and regression.
④ Production tests (deployed): configuration tests (live config vs. version control), stress tests (go find the limits), and canaries.
⑤ The book is explicit: a canary isn't a test, it's structured user acceptance — exposed only to a small slice of unpredictable real traffic, and not always catching new faults.
⑥ The missing cell is the prober: known requests against the deployed system, because the real production combination was never tested as a whole.
⑦ Three problems at scale: SRE tooling needs tests too (test depth decides how often it may touch production); disaster tooling is tested offline (compute a checkpoint equivalent to a clean stop, then validate it); speed is availability (a flaky test is worse than no test).
⑧ The control loop: better tests → higher MTBF → faster releases → new bugs → velocity pushed back. Its practical form is the error budget.
⑨ The judgment to keep: don't ask "are we testing enough?" — ask "which cell will this bug be caught in, and what is that cell's MTTR?"