BOOK DEEP-READ · SRE · DIGEST 01
Site Reliability Engineering · Ch 11 / 13 / 14 / 16 · Google · 2016
Behind every app you use, a group of people takes turns watching it. When something breaks at 3 a.m., their phone is the one that rings. Google's SRE book devotes four chapters to exactly that: how to build a rotation that doesn't burn people out, what the first responder should actually do, how a crowd of engineers can fight a fire without tripping over each other, and why every single alert has to be written down. Together they are the least "technical" part of the book — because what they manage is people.
A system goes down. The company sends a room full of its strongest engineers — and the fix takes longer than if it had sent two. Why? Because ten people each open a laptop and each start changing things. A has just rebooted a machine; B was in the middle of loading data onto that machine; C decided it was the network and edited a config; D is emailing the boss a picture of the situation from half an hour ago. All that effort cancels itself out.
Firefighting used to depend on a hero: the person who had been there longest and carried the whole history in their head. That model has three fatal flaws. Heroes get tired, take holidays, and leave. Newcomers can never learn what lives only in someone's head. And nobody knows how many times anyone actually got woken up this month — the pain is quietly absorbed by individuals, never shows up on the company's books, and so the alert that fires every night at two and means nothing lives on for years.
The answer these chapters give is unfashionably simple: copy the fire service. On a real fireground the most experienced person does not hold a hose. They stand outside as the incident commander, responsible only for who is doing what and what happens next. A separate crew holds the hoses — and only that crew is allowed to turn valves, no matter how urgently anyone else wants to reach in. Someone else does nothing but talk to the outside, so families and reporters all get the same current story. And there is a whiteboard on site — where the fire started, which floors are sealed — so anyone arriving late can catch up by reading it instead of interrupting someone to ask.
Two more things matter just as much. The rotation has to be survivable: there must be a cap on how many times one shift can be woken, because every incident deserves a proper sit-down review afterwards, and pile up too many and the reviews become box-ticking. And every alert has to be booked — how often it fired, and how often anyone truly had to get out of bed. Only once it's booked can you see that some alerts rang all quarter and were never real. Once you can see it, you can delete it with a clear conscience.
Firefighting stops depending on one person's memory and becomes a procedure anyone can run: newcomers have a path to learn, veterans can take holidays, and pain turns into numbers you can put on a meeting agenda. The cost is real too: the procedure has overhead, and running the full command structure for a small glitch is pure waste. The craft is knowing when to escalate and when to just fix it where you stand.
Firefighting isn't a race to be fastest — it's a contest in not descending into chaos: the person directing doesn't touch the system, only one crew touches it, only one voice speaks outward, and there's a whiteboard everyone can read. Add a rotation that people survive and alerts booked one by one — and none of that can be done by a machine.
Want the roles, the concrete thresholds and the diagrams? → Switch to the deep read
You think on-call exists to make fixes fast. It actually exists to keep both the system and the people alive for the long run. These four chapters take firefighting apart and rebuild it as a reproducible engineering institution: Ch11 treats on-call as a resource with a capacity ceiling; Ch13 uses three real outages to show that emergencies only get useful countermeasures once you classify them by who lit the fire; Ch14 ports the fire service's Incident Command System (ICS) into the machine room; Ch16 builds tooling that turns every page into bookkeeping you can count — because pain that isn't quantified never gets fixed.
All four live in Part III, "Practices": Ch11 opens it with on-call, Ch13 and Ch14 sit next to each other on response and coordination, and Ch16 follows Ch15 on postmortem culture. Why merge them: they are four stages of one pipeline — build the rotation (11) → respond when it pages (13) → coordinate without colliding (14) → turn the experience into data (16). Each is thin alone; strung together they are the one loop in SRE that cannot be automated. It follows Ch6 on monitoring (where alerts come from) and maps onto any team that runs a 24×7 rotation.
The old world, typically: one person carries the pager with the whole history in their head. Nobody counts how often they get woken, so the pain is absorbed privately — which is how the alert that fires at 2 a.m. every night and means nothing survives three years. When a real incident arrives, a room full of people each investigate and each change things: A reboots a machine, B is loading data onto it, C edits a config on a hunch, D tells the outside world a story that is half an hour stale. Afterwards nobody can say what was changed, the postmortem has no timeline, and the same hole gets stepped in again next year.
What happens if you don't solve it: reliability stalls, because nobody knows what to fix; and the team burns down — senior engineers leave over lost sleep, taking the only copy of the operational knowledge with them.
The most counter-intuitive line in the chapter is that on-call is by design a minority of what an SRE does. Google's hard constraint is that operational work (on-call included) stays under 50% of working time, with a target around 25%; the rest must go into engineering — otherwise the team degrades into pure operations and reliability stops improving forever.
That gives two dials you have to hold at once. Quantity: the rotation needs enough people. The book's rule of thumb is a floor of 8 engineers at a single site, or 6 per site across two sites — below that, per-person shift time can't be brought down and holidays or departures break it. Quality: there must be a cap on how often a shift gets woken. Google's number is at most 2 incidents per 12-hour shift — a proper postmortem takes roughly 6 hours, so two consume the shift's slack; more than that and reviews become perfunctory, which wastes the pain entirely.
Compensation is a rate limiter too: Google pays for on-call in time off or cash but caps it (around 5%–6% of overall salary). That ceiling is deliberate. If nobody can earn meaningfully more by taking more shifts, the incentive that pushes teams into overload is removed at the root.
Then there is feeling safe. Under adrenaline people switch to intuitive, fast, heuristic responses — and intuition under pressure is often wrong. The chapter's remedy is not "don't be stressed" but three concrete resources: clear escalation paths, written incident-handling procedures, and a blameless postmortem culture. The opposite failure is easy to miss: operational underload is also a disease — a quiet system lets responders go stale, which is why Google runs disaster exercises and role-played failure drills to keep the reflexes alive.
The takeaway test: if the average shift is woken more than twice, or the rotation pool is under 6–8 people, the problem is not "everyone push a bit harder" — it is capacity. Add people, cut alerts, or hand the pager back to the development team.
The chapter uses three real Google outages to sort emergencies into three kinds: test-induced (a deliberately destructive test meant to expose hidden dependencies reached further than intended), change-induced (a config push triggered a fleet-wide crash, stopped by a fast rollback in minutes), and process-induced (an automated workflow received the wrong input and swept far more machines than anyone meant into a decommissioning path).
The classification pays off because the countermeasures differ. For change-induced outages, whether you can roll back fast decides almost the entire loss, so stopping the bleeding outranks root-causing. For process-induced ones, what's missing is the safety gate inside the automation that asks "is this scale plausible?". For test-induced ones, what's missing is an abort path built into the test.
Two themes run through the chapter: all problems have solutions — panic is the real enemy; and treat outage history as an asset ("those who cannot remember the past are condemned to repeat it"). Concretely: keep and re-read the history, deliberately ask the uncomfortable "what if…" questions, and encourage proactive destructive testing — because hope is not a strategy.
The takeaway test: before any change or test ships, answer "what is the rollback path and how long does it take to run?". If you can't, it shouldn't ship.
The chapter first dissects an unmanaged incident and names three anti-patterns: sharp focus on the technical problem (nobody lifts their head to the business impact), poor communication (internal and external stories diverge), and freelancing — everyone unilaterally changing the system and cancelling each other out.
The fix is borrowed from US fire service practice, the Incident Command System, whose core is recursive separation of responsibilities: everyone holds a defined role, roles can be subdivided further, and any responsibility not delegated stays with the commander by default. Four roles:
Three pieces of supporting infrastructure: a recognized command post (the book prefers a text channel such as IRC — it survives network trouble better, logs everything for the postmortem, and bots can record automatically), a live incident document (owned by the IC, from a template, so arrivals read instead of interrupting), and explicit handoff (an outgoing IC hands the role over and the incoming one acknowledges out loud).
On when to declare an incident, three tests: do you need to pull in a second team? is the outage already visible to users? is it still unsolved after an hour of concentrated analysis? Any yes, declare. The stance is explicit — declare early: keeping a small incident orderly is easy, retrofitting order onto a sprawling one is not.
The takeaway test: if a second person is changing the system during an incident, your command structure has already failed. Settle who Ops is before continuing the fix.
The chapter's claim is a single sentence: to improve a baseline you must first measure it. Google built two tools. Escalator answers "did a human actually pick this up?" — transparently added to the alert's recipients, it tracks acknowledgement and escalates along a path when nobody answers (primary → secondary → whole team). Outalator answers "how many things are these alerts actually saying?":
cause:network, action:ignored, bug:1234) — record first, generalize later, which is far more realistic than designing the perfect taxonomy up front.The unexpected payoff is behaviour change: once the data is on the table, "this one fired over a hundred times last quarter and never once needed a human" stops being a complaint and becomes an executable proposal to delete it.
The takeaway test: for every alert that can wake someone, you should be able to answer "how often did it fire last quarter, and how often did a human truly have to act?". If you can't, you have not yet earned the right to call your alerts necessary.
Table 1 · Four dials on on-call health, and what turning each the wrong way costs (numbers are Google's figures as given in SRE)
| Dial | Google's setting | Cost of too tight | Cost of too loose |
|---|---|---|---|
| Share of ops work | ceiling 50% of time, target ~25% | nobody does engineering; reliability never improves | team drifts away from production and builds the wrong things |
| Rotation size | ≥8 at one site; ≥6 per site across two | too much shift time each; one departure breaks it | shifts so rare that responders go stale (operational underload) |
| Incidents per shift | ≤2 per 12-hour shift | reviews become perfunctory — two 6-hour postmortems don't fit | — (but drills must make up for lost reflexes) |
| On-call compensation | time off or cash, capped (~5%–6% of salary) | too little breeds resentment | uncapped → people earn by taking shifts, pushing the team into overload |
Table 2 · The same incident, with and without a command structure (Ch14's two walkthroughs)
| Unmanaged incident | Incident under ICS | |
|---|---|---|
| Who may change the system | whoever feels urgent → changes cancel out | Operations only; nobody else reaches in |
| Where the global state lives | in people's heads, mutually inconsistent | in the live incident document; arrivals read it |
| External message | several versions, most of them stale | one Comms voice, updated on a cadence |
| Shift change | people quietly vanish; context is lost | explicit handoff plus explicit acknowledgement |
| Cost | looks free; burns the time on coordination friction | the process has overhead: full ICS for a small glitch is waste |
Table 3 · Keep or kill an alert, judged on Outalator's long-term data
| What the long-term data shows | Verdict | Action |
|---|---|---|
fires often, nearly all action:ignored | pure noise | delete it, or demote to a ticket — stop waking people |
| fires rarely, always needs hands | a healthy page | keep it; add a playbook |
| fires often, always needs hands | something underneath is genuinely broken | don't retune the threshold — fix the cause, or trade error budget for the time |
| barely fires, nobody has practised | operational underload | keep reflexes alive with disaster exercises and role-played drills |
These four chapters are the source text for today's entire incident-management industry. The IC / Ops / Comms split is the default model inside PagerDuty, Atlassian and Incident.io; Escalator's acknowledge-and-escalate semantics are table stakes in every on-call scheduling tool; and Outalator is the direct ancestor of incident grouping and alert-fatigue reporting in modern alerting platforms. Asked in an interview or a design review "how do you run on-call?", answering with a cap on incidents per shift, an actionable rate, and who is the single person allowed to change the system already leaves "we have a group chat and shout in it" far behind.
① The four chapters form one loop: rotation (11) → response (13) → coordination (14) → bookkeeping (16) — the one part of SRE that cannot be automated.
② On-call is meant to be a minority of the work: ops share capped at 50%, targeted near 25%; the rest belongs to engineering.
③ On-call capacity has a hard floor: 8 engineers at one site, or 6 per site across two; at most 2 incidents per shift, because one proper postmortem costs 6 hours.
④ Capping compensation (~5%–6% of salary) is a deliberate rate limiter that removes the incentive to earn by taking more shifts.
⑤ Feeling safe rests on three things: clear escalation paths, written procedures, blameless postmortems. Underload is also a disease, treated with drills.
⑥ Emergencies come in three kinds by who lit the fire: test-, change- and process-induced. For change-induced ones the whole outcome is rollback speed; stop the bleeding before root-causing.
⑦ The four ICS roles: IC holds the picture and not the keyboard, Operations alone changes the system, Comms is the single outward voice, Planning carries the long tail — plus command post, live document, explicit handoff.
⑧ Three tests for declaring an incident: need a second team? users affected already? still unsolved after an hour? Declare early.
⑨ Ch16 in one line: pain that isn't quantified never gets fixed. Escalator tracks whether anyone picked it up; Outalator groups alerts into incidents, tags them, and aggregates over time.
⑩ For every alert that can wake someone, be able to say how often it fired last quarter and how often a human had to act — otherwise you haven't earned the right to call it necessary.