BOOK DEEP-READ · SRE · DIGEST 02
Site Reliability Engineering · Part IV · Ch 28–32 · Google · 2016
The earlier chapters were about machines: how to monitor, how to release, how to fight fires. These five are about people — the small group of engineers everyone calls when something breaks. How you grow them, how you shield them, where you send them, and then the harder question: how do you make fewer places need them at all?
A team is drowning in work. The company adds two people — and it gets slower. What worked instead was sending in one person who said up front, "I am not here to do your work." He sat in on meetings, read old records, and kept asking: "When this alert fires, what do you actually do?" Two months later the team had pulled itself out. Adding hands didn't help. Adding questions did.
Three chronic ailments. Nobody teaches the new hire, so they get dropped onto the ticket queue — a year later they are excellent at answering tickets and still don't know how the system works, so any unfamiliar failure stops them dead. The veterans are interrupted to death: pulled away seven or eight times a day, not one line of real work moves. And the hardest services rest entirely on what a handful of people remember — and that handful does not grow, because hiring is far slower than services multiply.
① Train people like apprentices, not like a conveyor belt. Learn in order, each step resting on the last. First watch the master work — while secretly writing down what you would have done, then compare. Then reverse it: they watch you. You graduate by passing a checklist, not by serving time. What you're training isn't command recall, it's the ability to take things apart.
② Post one gatekeeper; don't leave everyone's door half open. In any given shift exactly one person absorbs every interruption and everyone else is left completely alone. The reason is blunt: climbing back into focus is expensive, and two people each cut in half produce less than one person with a whole clear day.
③ When a team is already stuck in the mud, send a coach, not a labourer. The borrowed person does not answer your tickets: he first builds context, demonstrates once what a good postmortem looks like, and then pulls the team out with questions — "if this alert never fired, would any user notice?" Ask that often enough and they will delete it themselves.
④ The deepest step: stop rescuing services one at a time. Rather than waiting until a service is built and then sending someone to inspect it and produce fifty required changes, put the good practices into the foundations and the prefabricated parts — anything built on top arrives with monitoring, throttling and rollback already in it. The inspection step then disappears.
The order matters: while the fire is burning, do ② first — until interruptions are blocked, nothing else moves; then ③; once the team can breathe, ① ; and only when three or four services are repeating the same pain does ④ make sense — build the foundation too early and you don't yet know which parts to prefabricate. The honest cost: all four run on human judgement and trust, and the moment any of them becomes a form that must be signed, it degrades into a rubber stamp.
To rescue a team crushed by operations, in this order: block the interruptions, bring in a coach who only asks questions, then train people properly, and finally bake the good practices into the foundation. Success is not measured by how many services you took on, but by how many no longer need you.
Want the mechanisms, the tests, the numbers and the diagrams? → Switch to the deep read
You think Part IV is the skippable "management" part. In fact it answers the book's most dangerous question: there will never be enough SREs, so now what? Five chapters, five segments of one answer — Ch28 takes a new hire from zero to on-call instead of using them as a ticket sponge; Ch29 admits attention is a resource that interruptions destroy, and therefore concentrates all interrupts onto one person; Ch30 sends a team in operational overload one embedded SRE who "only asks questions and does not do your work"; Ch31 keeps a team spread across sites sharing one version of reality; Ch32 delivers the endgame: the engagement model evolves from "review a finished service" to "join at design time" to "bake production readiness into the platform so the review itself disappears."
All five chapters are in Part IV, "Management" (Ch28–33); Ch33 has its own page here and Ch34 is the conclusion. They follow Part III's practices — on-call, incident response, postmortems — which say how the work is done, whereas these five say who does it and how to keep them able to. Why merge them: they form one supply-side chain — make people (28) → shield them (29) → rescue them (30) → share reality between them (31) → finally reduce the need for them (32). Break any link and the technical machinery of the earlier chapters stays on paper, because nobody has time for it.
Pin the scenario down: a 6-person SRE team owns 3 services, takes in roughly 40 tickets a week (30 minutes each on average — 20 hours, about half a person) and is paged 5 times per 12-hour shift (the book's healthy ceiling is 2). That puts operational work at around 70% of the team's time, far past the 50% cap. At which point the team does three things that all look reasonable and all deepen the hole: drop the new hire onto the ticket queue; have everyone keep their door half open for "a share" of the interrupts; ask upward for more headcount.
None of the three works. The ticket queue teaches reactive fragments, and a year in the hire still cannot handle a failure they haven't seen. Everyone half-absorbing interrupts means everyone is shredded, and project progress is zero. And getting one person from hired to independently on-call is measured in quarters — far slower than services grow.
What happens if you don't fix it: the team degenerates into a ticket machine and reliability stops improving; senior engineers leave because they are busy every day and nothing advances, taking the only copy of the operational knowledge with them; the service is eventually handed back to development — which cannot hold it either. SRE's bottleneck was never technology. It is the supply of people.
The line worth nailing to the wall: don't use new hires as ticket sponges. "Spend six months on tickets to get familiar" sounds gentle and is systematically miseducating them — the ticket stream is random, reactive and non-cumulative, so what they learn is a pile of "when X, click Y" rather than how the system works. The chapter lays out two contrasting trajectories: chaotic (random, one-off, trivia) versus ordered (sequential, cumulative, aimed at the system itself), and insists on the latter.
Two design principles follow. Systems and services, not trivia: what deserves a new hire's time is "which hops a request makes from the user to the disk", not "which flag that command takes". And targeted project work, not menial work.
Three capabilities are named as the things to cultivate, and all three are precisely what manuals cannot teach and tickets cannot drill: reverse engineering (handed an undocumented system, work out how it runs), statistical and comparative thinking (facing several suspect paths, let data rather than instinct choose which to check first), and improvising from first principles (in an unprecedented failure, reason up from "why does this work at all").
Four concrete practices: disaster role playing (Google calls it the Wheel of Misfortune — a weekly round where one person plays "the system" and sets the puzzle while the others play the on-caller); really breaking things and really fixing them in a controlled environment; documentation as apprenticeship; and the crucial one, shadow on-call: the trainee receives every page in parallel with the primary, writes down independently what they would do, then compares against what the primary actually did. Reversed, there is also the reverse shadow — the trainee handles it while the veteran watches in silence.
The test to take away: ask "do our new hires go on-call because a checklist says so, or because enough time has passed?" No checklist means you are passing ticket volume off as training.
The chapter first splits operational load into three kinds, because the remedies differ completely: pages (urgent, immediate), tickets (human-initiated, queueable) and ongoing operational activities (routine work with no requester behind it, such as pushing a config by hand). Lump them together as "we're busy" and you will never know which to cut.
The key insight is about people: cognitive flow state — focus takes a climb to reach, and one interruption drops you back to the bottom. The real cost is not the ten minutes, it's the climb. From which comes the chapter's most counterintuitive and most copyable rule: polarizing time — within any block, a person is either 100% on interrupts or 100% undisturbed, and never half-open. Two people each cut in half produce strictly less than "one person undisturbed all day plus one person on the door all day".
The supporting practices: appoint an explicit interrupt catcher, the one person on the team allowed to do toil; measure the interrupt volume — "tickets per week × average minutes each" converts straight into person-hours and tells you whether the role is half a person or two. To actually reduce interrupts there are two levers: automate or self-serve the repeating ticket categories, and push the ticket back to its source — if the root cause is a product defect, file a bug rather than fixing it by hand each time (the smoother your manual fix, the less likely the defect ever gets fixed). One last discipline: hand off an interrupt together with its context, or the next person rebuilds it from nothing and the interrupt gets paid for twice.
The test to take away: if nobody can name this week's interrupt catcher, the answer is "everyone is, so nobody is doing project work".
This chapter handles the hardest position: a team has already fallen into operational overload, past the 50% cap, and — this is the crux — it no longer believes this can be fixed. The signature symptom is that alerts have become a to-do list: every one is treated as "an item to handle", so nobody asks any more why it deserves to fire.
The remedy is to send in one outside SRE, and to state plainly: he is not extra hands; his deliverable is a change in this team's judgement. Three phases:
① Get context. Read the postmortems, sit in on the meetings, walk the alerts one by one asking three things: is this actionable? Is it urgent? Does it really need a human? And find the biggest source of stress — usually not a technical problem but the habit of treating every alert as equally urgent.
② Share context. Write one properly good postmortem for the team to see (demonstration beats instruction); and classify the fires — is this missing capacity, or a leaking abstraction that keeps producing the same problem in new costumes? Without the classification you will keep putting out different flames of one fire.
③ Drive change. Start with the basics: does this service have an SLO, and is anyone measuring against it? Without one, "what should we fix" is forever a matter of taste. Then clear the decks (a temporary freeze, borrowed people) so the team has hands free. The two methods that matter most: explain your reasoning, so the team can make the call themselves next time; and ask leading questions — don't hand over the answer, ask "if this alert never fired, would any user notice?" and let the team reach the conclusion. Conclusions they reached themselves are the ones that hold.
The self-restraint matters just as much: the embedded SRE must never become the team's ops bucket. The moment he starts answering their tickets they treat him as temporary reinforcement, the change stops — and when he leaves, everything reverts.
The test to take away: at the end of the embedding, ask the team how many alerts they deleted themselves last month. Some deletions means the judgement really changed; none, with only the backlog cleared, means you hired a two-month temp.
SRE teams are distributed by nature — rotating across time zones, so "who is on call" changes two or three times a day. That creates a purely organizational problem: how does everyone keep the same picture of what shape this service is in right now?
The central institution is the production meeting: held regularly (usually weekly) with a fixed agenda — upcoming production changes, key metrics, outages, paging events, non-paging events, and prior action items. The part most often misread: it is a meeting about the service, not about the team — no status reporting, no personnel. The "non-paging events" line is the one most worth copying: it collects the phenomena that woke nobody but were odd anyway, and that is often a large outage's only early signal.
Inside the team three collaboration roles are named: the tech lead owns technical direction, the manager owns people and priorities, the project manager owns cross-team drive — collapse all three into one person and engineering is always what gets sacrificed first.
The cross-team lesson comes from a real case in the book: several SRE teams each built their own monitoring dashboards, so someone started a unifying framework (Viceroy). The first attempt failed, for reasons that are not technical at all: scraps of time contributed by many teams ("20% time") cannot carry a platform project, and there was no clear ownership. What made the second attempt work was dedicated people, plus a well-defined plugin boundary so each team could plug its own data in. And one harder lesson: win voluntary adoption rather than mandating it — a platform that is forced gets routed around, usually by building yet another private dashboard.
The test to take away: if nobody's primary job is your cross-team platform project, its failure is already written; it just hasn't happened yet.
This is the heaviest chapter in Part IV and the close of the whole chain. Its premise is a cold fact: SRE capacity is scarce, so "which service we take and on what terms" has to be an explicit decision, not a matter of who argues loudest.
The first-generation tool is the PRR (Production Readiness Review), in six phases: engagement (agree the scope) → analysis (review design, monitoring, failure modes; produce the list of required changes) → improvements and refactoring → training (SRE learns the service, documentation is written) → onboarding (SRE goes on-call, ownership transfers) → continuous improvement.
PRR has two structural defects. It is too late: the service is built, often launched, so SRE can only recommend retrofits — and that is when architectural change is most expensive, so half of those fifty items get shelved forever as "too much rewriting". And it does not scale: a PRR is heavyweight while SREs are far fewer than services, so the queue only lengthens.
Hence the second generation: early engagement — bring SRE in during design and implementation, when changing the architecture is an order of magnitude cheaper and SRE can actually shape the design instead of patching afterwards. The cost is a different skill requirement: they have to be able to review a design and argue with its author.
The third generation is the chapter's real conclusion, and the part most often skipped: bake production readiness into frameworks and platforms. Rather than reviewing service after service, provide a production-ready framework — standard monitoring and control surfaces, standard instrumentation, standard configuration and deployment shapes — with the SRE experience encoded in it. The benefits compound: lower operational overhead overall (every service has the same operational shape, so one toolset covers them all), best practices spread by themselves (the framework ships a version and everybody gains), and most importantly the engagement itself gets light or disappears.
The test to take away: count how many identical production-readiness checks you ran in the last year. More than three, and that checklist should become code instead of being re-run by hand.
Table 1 · Four ways to grow a new SRE (the order is the usage: project → role play → shadow)
| Approach | What it builds | Cost / how it fails |
|---|---|---|
| Drop them on tickets | Tool fluency, fast reactions | No model of the system; an unfamiliar failure stops them — basically don't |
| A targeted small project | End-to-end grasp of one real path | Consumes a veteran's review time; pick wrong and it becomes menial work |
| Disaster role playing | Reasoning and escalation habits under pressure | Fictional scenarios, so none of the real system's dirty detail |
| Shadow / reverse shadow | Real judgement on real pages, with an answer key | Occupies a second veteran's attention; a thin rotation never gathers enough samples |
Table 2 · Three generations of engagement: when each one is the right answer
| PRR after the fact | Early engagement | Frameworks and platform | |
|---|---|---|---|
| Moment of entry | Built / already launched | During design and build | Before the first line of code |
| Cost of change | Highest (often shelved) | An order of magnitude lower | Near zero (right by default) |
| Effort vs services | Linear | Linear | Once, reused by all |
| Demands on SRE | Can operate and inspect | Can review a design and argue | Can do productized platform engineering |
| When to use | Inheriting one or two legacy services | A new service is being designed | Three or more services repeat the same pain |
| How it fails | Queue never clears, list gets shelved | SRE becomes a design rubber stamp | No dedicated staff → half-built, routed around |
These five chapters are the shared origin of today's "platform engineering" and "SRE transformation" vocabularies. Ch32's third generation all but predicted internal developer platforms, Kubernetes and service meshes — and predicted their most common death, the platform with no dedicated team that everyone routes around. Ch29's interrupt catcher became the default rotation shape of nearly every infrastructure team. Asked in an interview how your SRE and development teams divide the work, being able to say which generation of the engagement model you are on and which one you are moving to beats "we have an SRE team" by a whole level.
① Part IV answers whether SRE can scale at all: make people (28) → shield them (29) → rescue them (30) → share reality (31) → reduce the need for them (32).
② Train by ordered accumulation, not by ticket wash: systems and services rather than trivia, targeted projects rather than menial work. You graduate on a checklist, via shadow on-call then reverse shadow, plus a weekly round of disaster role playing.
③ The three capabilities to build: reverse engineering, statistical and comparative thinking, improvising from first principles — no manual teaches them, no ticket drills them.
④ Operational load has three kinds: pages / tickets / ongoing operational activities; lump them together as "busy" and you never know what to cut.
⑤ Polarize time: a person is either 100% on interrupts or 100% undisturbed — half-open doors are strictly worse. To reduce interrupts, push them back to the source: automate or self-serve the repeats, file product defects as bugs.
⑥ Operational overload takes a coach, not a labourer: get context → share context → drive change; the methods are explain your reasoning and ask leading questions, and never become the team's ops bucket.
⑦ The production meeting is about the service, not about the team; its "non-paging events" line is often a large outage's only early signal.
⑧ Three generations of engagement: PRR after the fact (too late, doesn't scale) → early engagement (change an order of magnitude cheaper) → frameworks and platform (invest once, reused by all, the review itself disappears).
⑨ Cross-team platform projects nearly always die of the same cause: scraps of time and nobody whose main job it is; and adoption must be won, because a mandate gets routed around.