BOOKS DEEP-READ · SRE · CH 27
Site Reliability Engineering · Ch 27 · Google · 2016
One day an app on your phone quietly grows a new feature; or a huge website puts a brand-new product on its front page. Between "the engineers finished the code" and "hundreds of millions of people are using it" lies the stretch of road where things most often go wrong. This chapter is about how to hand something new to an enormous audience without turning it into an outage.
When a launch goes badly, the cause is usually not a bug in the code. Bugs are what testing catches. What actually hurts is too many people arriving at once — not enough machines, an empty cache, and the team next door whose service you are calling having no idea you were coming. None of this reproduces in a test environment, because a test environment has no real people in it.
Knowing how to ship safely used to live inside a few veterans' heads. Someone who has shipped twenty times knows to request machines two weeks ahead, to warn the teams downstream, to keep a one-click way back. A newcomer doesn't, and falls into the same hole. With one team that is survivable; with several hundred teams, the same hole gets dug several hundred times. Experience that can't be copied is the same as no experience.
The chapter offers four things.
First, keep a small squad whose whole job is helping other people ship. They don't build your product; the role is closer to a wedding planner or a moving company — they didn't cook the food, but they know exactly where each step goes wrong, because they do a hundred of these a year.
Second, a checklist, like the one a pilot reads aloud before takeoff. Every line on it is the lesson of some real past accident. The beauty of a checklist is that it doesn't test your memory — it just asks you to go down the list.
Third, don't open the gates all at once. Let one person in a hundred have it, watch for a day, then one in ten, and only then everyone. Like trying a new dish on one table before printing it in the menu.
Fourth, do a silent dry run first. Copy the real orders over to the new kitchen, let it cook them, then throw the food away — guests still eat what the old kitchen made, but you now know whether the new one panics under a rush.
Shipping stops being a matter of luck and of whether a veteran happens to be around, and becomes something you can follow step by step. There is a counterintuitive twist too: the checklist is supposed to get shorter — once a line is handled by the tools themselves (say, the system now throttles traffic by default), it should be crossed off. The cost is real: all of this takes effort, and a small team that copies it wholesale tends to end up shipping slower, with a checklist everyone signs without reading.
The risk of a launch doesn't live in the code. It lives in how many people arrive in the first minute and in the one dependency nobody told. The fix is to turn the veterans' experience into a checklist that keeps renewing itself, plus "give it to a small slice first" and "run it silently first".
Want the mechanisms, the rollout percentages and the diagrams? → Switch to the deep read
SRE Chapter 27 argues that launching safely is not an individual skill but a process that can be engineered, reused and scaled. Google's answer is a dedicated advisory team — Launch Coordination Engineering (LCE) — plus a launch checklist that keeps metabolizing: every line on it is the epitaph of a real incident, and any question that shared infrastructure can answer by default should be deleted from it.
This is the closing chapter of Part III, "Practices". The twenty-odd chapters before it are about keeping a system reliable once it is running (alerting, incident response, overload, cascading failures, consensus, pipelines, data integrity); this one goes back to the moment a system runs for the first time. It follows Ch25/Ch26 (runtime pipelines and data reliability) and leads into Part IV, "Management". Its real-world descendant is the production readiness review that platform teams run today.
A launch aimed at hundreds of millions of users never involves one team only: you need capacity, three or four internal services need to raise your quota, monitoring has to be configured in advance, a client release has to go out, legal and support have to be ready. Miss any link and the incident is waiting for you in the first hour.
Google's scale makes two things true at once. There are many launches — any scheme that pulls a room full of experts together for each one collapses under its own weight. And a single launch can be violent: a new product placed on the front page goes from zero straight to peak, with no natural warm-up ramp, which invalidates the usual assumption of "ship small, let usage grow".
What happens if you don't solve it? Launch knowledge stays oral tradition, and the same mistake gets reinvented in a few hundred teams. The chapter's job is to move "launch experience" from a property of people to a property of the organization.
The definition is deliberately almost too wide: a launch is any new code that introduces an externally visible change. Why not just big releases? Because the thing that blows up is usually the change nobody considered a launch — an experiment flag that went live, a backend implementation quietly swapped. Narrow the definition and risk leaks out through the gaps in the process.
The owner is LCE: a dedicated team of SRE consultants playing five roles — auditing reliability, acting as liaison between teams that don't know each other, driving the technical side of the launch, serving as gatekeeper who signs off, and educating developers.
The crucial part is that it advises and does not take over: LCE does not change your code or provision your capacity for you. That is not politeness, it is arithmetic — for a team of a dozen or so to cover an entire company's launches, the only scalable posture is to turn knowledge into process rather than insert people into every project. The moment LCE starts doing the work, it becomes the company's launch bottleneck.
The chapter holds a launch process to five standards: lightweight (little burden on developers), robust (clear criteria, not the reviewer's mood that day), thorough (nothing important omitted), scalable (able to cover many launches) and adaptable (a redesign and a new button share one process at different depths).
These five fight each other: more thorough is less lightweight, and the more rigid the rules the less adaptable. The answer is not to split the difference but two moves: tier by risk (high-risk launches get the full review, low-risk ones self-serve), and push questions down into the infrastructure — whatever a framework can handle by default should not be asked of a human again.
The checklist is organized in nine sections: architecture and dependencies, integration, capacity planning, failure modes, client behavior, processes and automation, development process, external dependencies, rollout planning. A few lines, to show what it actually asks:
The most counterintuitive mechanism is here: the checklist has to lose weight. If a question can be answered by shared infrastructure by default (the framework throttles, the client library backs off, the platform requests quota), it should be deleted from the checklist and become default behavior. A grow-only checklist gets longer until everyone ticks boxes with their eyes shut — the moment it stops being a safety net and becomes a bureaucratic ritual. This also gives LCE a real deliverable: driving the whole company to converge on common infrastructure.
Staged rollout: the new version takes a small slice of real traffic first. Google's public CRE writing gives concrete steps — start somewhere between 1% and 10% depending on user base, and for internal binary releases go 1% → 10% → 100%. The 1% step catches gross failures like "90% of responses are errors"; the 10% step catches problems an order of magnitude smaller. Leave at least 24 hours between steps, because traffic is diurnal and a shorter bake never sees peak hours.
Dark launch: mirror real production requests to the new implementation and discard what it returns; users still get the old implementation's response, and you keep only the new one's error rate and latency. The value is getting the shape of real traffic at zero user impact — real requests are always nastier than a load-test script. The costs are explicit: writes cannot be dark-launched (they have real side effects), and downstream load doubles.
Feature flags: split deploy from release. Code ships but stays off; when something breaks you flip a switch (seconds) rather than roll back a binary (minutes, plus restarts). Ramping by percentage or by region also becomes one line of config.
Overload behavior and load tests: push the service past its breaking point before launch and find out whether it degrades gracefully or avalanches. A service that starts rejecting the excess at 120% load and protects the other 100% is healthy; one that at 105% load sees response times climb to 30 seconds and drags its callers down is not.
The four techniques are not interchangeable options; they stop different kinds of risk. The real selection question is "what is the dominant risk of this particular launch?"
Table 1 · Four risk-reducers: what each stops, and what it doesn't
| Technique | Stops | Doesn't stop | Cost |
|---|---|---|---|
| Staged rollout | Regressions in error rate, latency, resource use; real user feedback | Problems that only appear at full scale (hot keys, lock contention, quota ceilings) | Long ramp (≥24h per step); two versions coexist meanwhile |
| Dark launch | Crashes, timeouts and leaks in the new implementation under real traffic shape | Writes; anything about user experience (nobody checks the answers) | Doubles downstream load; needs traffic-mirroring plumbing |
| Feature flags | Time to stop the bleeding — off in seconds, no binary rollback | Data already written wrong; the combinatorial explosion of flags | Flags accumulate as debt; an uncleaned old flag is a time bomb |
| Load / overload tests | Where the breaking point is, and whether overload means degrade or avalanche | Real user behavior distributions; long-tail requests and cold caches | Needs a production-like environment or the result is not credible |
The second trade-off is how heavy the process should be. "Scalable" and "adaptable" cash out as tiering by risk — a dedicated team cannot possibly shepherd every launch end to end.
Table 2 · Three tiers of launch review, and what each costs
| Tier | Fits | You get | You pay |
|---|---|---|---|
| Full coordinated review | Brand-new products, a dozen teams, traffic from zero straight to peak | An outside view that drags every cross-team blind spot into the open | Expensive; once the coordinating team is mandatory it is a company-wide bottleneck |
| Self-serve checklist + spot checks | Big features on existing products, real architectural change | Wide coverage at low cost; teams grow their own launch instincts | With nobody watching, boxes get ticked blind; quality varies by team |
| No review (standard platform) | Routine iteration that stays entirely on the company's frameworks and pipeline | The highest shipping frequency; the platform absorbs the risk by default | Requires infrastructure that really is safe by default; a platform bug is global |
The chapter's position is clear: the goal is to get more and more launches down into the third tier. The checklist is not the destination but a signpost toward infrastructure that is safe by default — which is exactly why deleting items matters as much as adding them.
You don't need an LCE today, but the mechanism has been broken up and absorbed into standard tooling: Spinnaker / Kayenta automated "ramp up and stare at graphs" into automated canary analysis; Argo Rollouts and Flagger wrote the rollout ladder into declarative Kubernetes config; LaunchDarkly and OpenFeature turned feature flags into a product; GitHub's Scientist shrank dark launching into a function-level library. The LCE role itself evolved into the production readiness review platform teams run.
1%–10%, internal releases go 1% → 10% → 100%, and at least 24 hours between stages because traffic is diurnal. Google Cloud Blog, "How release canaries can save your bacon", CRE Life Lessons ↗50 time-series data points per metric for the statistics to mean anything, which often makes a canary analysis several hours long. Google Cloud Blog, "Canary analysis: lessons learned from Google and Waze" ↗use, the new one in try, and the run always returns the old path's result while randomizing the order of the two, comparing results, and swallowing and recording exceptions the new path raises — refactors validated on real production traffic, with users none the wiser. github/scientist README ↗1 · In one line: launching safely is not an individual skill but a process that can be engineered, reused and scaled — here, an advisory team plus a checklist that metabolizes.
2 · "Launch" is defined deliberately wide: any new code introducing an externally visible change; what blows up is usually the change nobody counted as a launch.
3 · LCE's five roles: audit, liaison, drive the technical side, gatekeep and sign off, educate. Advising without taking over is what makes it scale.
4 · Five constraints on the process: lightweight / robust / thorough / scalable / adaptable. They conflict; the answer is tier by risk plus push questions into the infrastructure.
5 · Nine checklist sections: architecture and dependencies, integration, capacity planning, failure modes, client behavior, processes and automation, development process, external dependencies, rollout planning.
6 · The counterintuitive rule: the checklist must lose weight — once infrastructure handles a question by default, delete it; grow-only checklists rot into rubber stamps.
7 · Four techniques, each for a different risk: staged rollout (1% → 10% → 100%, ≥24h per step), dark launch (real traffic, discarded responses), feature flags (deploy/release split, seconds to stop the bleeding), load tests (confirm degradation, not avalanche).
8 · Today: Kayenta / Argo Rollouts / Flagger / LaunchDarkly / Scientist each took a piece, and the LCE role became the platform team's production readiness review.
9 · The honest edges: checklists can't rescue deep architectural problems; a brand-new product has no ramp to give; bad client behavior, once shipped, can't be recalled.