CHAPTER DEEP READ · SRE · CHAPTER 12
Site Reliability Engineering · Ch 12 · Chris Jones · Google · 2016
A site won't load, an app spins forever, a payment hangs — somebody has to go find the fault. You probably assume that job runs on talent: some people just have the touch and can glance at a log and know. Chapter 12 of Google's SRE book says otherwise: it is not talent, it is a craft, and a craft can be taught. The chapter breaks "how experts find faults" into steps anyone can follow.
Think of an emergency room. A patient comes in, and the doctor's first move is not to work out the diagnosis — it is to stop the bleeding. The cause can wait; the patient cannot. Only once things are stable come the blood tests and the scans (metrics and logs), then the reasoning, then trying a treatment and watching the response, then the write-up.
The most valuable line in the chapter hides in that ordering: step one is not finding the cause, it is stopping the bleeding. Spend forty minutes nailing the culprit and your users hurt for forty minutes — solving the case does not shorten their pain by a second.
It is hard because people guess. The book names the common guesses: staring at a metric that has nothing to do with this; mistaking coincidence for cause — two lines rise together on a dashboard, so one must have caused the other (the more things you monitor, the more pairs of lines happen to look alike); and clinging to whatever caused last time's outage.
① Stopping the bleeding and solving the case are two jobs, and they can run at once. Shift traffic to the machines that still work, switch off a non-essential feature, degrade the slow thing — none of that requires knowing the cause. One condition though: don't wipe the scene. The most common way to stop the bleeding is a restart, and a restart usually burns the only evidence, so the same failure comes back next month.
② Halve it, don't walk it. A request passes seven or eight stages: gateway, frontend, auth, business logic, cache, database, storage. Probe the one in the middle: if the middle looks healthy, the fault is downstream; if the middle is already wrong, it is upstream. Each cut removes half, so seven or eight stages collapse in about three cuts. This is the line between investigating and guessing.
③ A broken system is usually still busy — just busy with the wrong thing. So don't ask the big question "why did it break". Ask three smaller ones: what is it doing? where is the effort going? why there? The book's example: a database cluster slows down; ask where the CPU is going and it turns out to be sorting log entries; ask where inside the sorting and it is matching a badly written pattern against filenames. Four layers separate the symptom from the culprit — unreachable by guessing, inevitable by asking, layer by layer, where the effort goes.
④ The cheapest question: what touched it last? Systems have momentum — a thing that was running fine does not break on its own. Reading the change log usually beats reading the code.
In July 2019 Cloudflare shipped a new firewall rule containing a badly written pattern, which pinned CPU on machines worldwide and knocked out a large slice of the web. The timeline is this chapter in miniature: trouble at 13:42, understood only at 14:02, and the moment they understood it they killed that whole rule set globally with one switch; traffic was back by 14:09. Fixing the rule itself and turning it back on came more than forty minutes later.
Troubleshooting is a craft, not a gift: stop the bleeding before solving the case (without wiping the evidence), cut from the middle instead of walking from the front, ask "what is it busy with, where is the effort going" rather than staring at "why did it break", and always look first at what touched it last. One honest cost: all of this rests on being able to see inside the system — logs, metrics, request paths all have to be paid for in advance. If nobody paid, the prettiest method still leaves you guessing on the day it matters.
Want the full process diagram, comparison tables and real cases? → switch to the deep read
Chapter 12 of the SRE book dismantles a popular myth: troubleshooting is not innate intuition but a craft that can be decomposed and taught. It is the product of two things — a generic hypothetico-deductive process (problem report → triage → examine → diagnose → test/treat → cure) multiplied by concrete knowledge of this particular system. Without the process you guess; without the system knowledge you cannot even form a testable hypothesis. The most counterintuitive lesson is the ordering: step one is not finding the cause, it is stopping the bleeding — "you aren't helping your users if the system dies while you're root-causing."
log₂n steps for n candidates — roughly 7 for 100.Written by Chris Jones. This is the core skills chapter of Part III, "Practices": it follows chapter 6 on monitoring (symptom-based alerts are what deliver the problem to you) and chapter 11 on being on-call (who picks it up), and it sets up chapter 13 on emergency response, chapter 14 on incident management and chapter 15 on postmortem culture. It is also the direct beneficiary of chapter 9 on simplicity — you cannot debug beyond the system you still understand. In practice it maps to playbooks, incident command, observability platform choices, and the most common interview prompt of all: "error rate just jumped to 3% — how do you investigate?"
The chapter opens on a diagnosis of its own: troubleshooting is widely treated as a gift — some have it, some don't. The illusion is understandable, because for people who do it daily the process has become so ingrained it is hard to articulate. The cost of treating it as a gift is concrete: it never gets taught, practised or written down, so the team's ability is tied to a few veterans and leaves when they do.
The second problem is that distributed systems invalidate the old instincts. Single-machine debugging means finding the broken box or process; but when one request fans out to tens or hundreds of RPCs, often no machine is broken at all — one hop got 300 ms slower, then came timeouts, retries and queueing, surfacing at the edge as "the site is slow". You are not looking for a broken part; you are looking for a bad segment on a path.
What happens if you don't solve it? Bill it to the SLO. A monthly availability target of 99.9% allows roughly 43 minutes of unavailability for the whole month (the accounting of chapters 3 and 4). Which means time-to-diagnose is not a soft metric; it is the denominator of availability — for the same incident, 30 minutes versus 5 minutes to locate burns most of the month's error budget in the difference.
The chapter also teaches what not to do first, listing four hallmarks of ineffective troubleshooting: ① looking at symptoms that aren't relevant, or misunderstanding what a metric means, and chasing a wild goose; ② not understanding how to change the system, its inputs or its environment safely, so hypotheses cannot be tested; ③ coming up with wildly improbable theories, or latching on to the cause of a past outage; ④ hunting spurious correlations — coincidences, or pairs that share a third cause. The fourth gets worse automatically with scale: a service exporting a few thousand metrics yields millions of pairs, and some of those curves will match perfectly by chance. One more number from the book's introduction belongs here: roughly 70% of outages are due to changes in a live system — bad news, and also the best first lead you will get.
The chapter defines troubleshooting as an application of the hypothetico-deductive method: given observations about a system and a theory of how it ought to behave, you repeatedly hypothesise possible causes and try to test them. Both inputs are required. You can work with only the generic process, but you will usually be far less efficient than someone who knows the system; with only system knowledge and no process, people follow intuition — and intuition is exactly what breeds the four failure modes above.
This is also why the skill looks innate: a veteran's system knowledge is tacit and their process is internalised, so an onlooker only sees "they spotted it instantly". The chapter's whole job is to put both on the table — the process as Fig. 1, the system knowledge as playbooks and as the closing demand that observability be designed in.
This is the section to memorise. The book says it plainly: "Stopping the bleeding should be your first priority; you aren't helping your users if the system dies while you're root-causing." The goal of triage is defined just as plainly — make the system work as well as it can under the circumstances. The means are emergency measures that need no diagnosis: divert traffic away from a broken cluster, deliberately drop some load to protect the rest, degrade responses to cached or simplified results.
These two jobs should never be serialised: in a mature system a cross-cluster traffic shift is a minutes-long operation, while tracking down an intermittent memory leak can take hours. Making users wait for the second is pure waste.
But the chapter immediately adds the constraint most often forgotten: "An emphasis on rapid triage doesn't preclude taking steps to preserve evidence of what's going wrong, such as logs, to help with subsequent root-cause analysis." The practical sting is that the most common way to stop the bleeding — restart, replace, reprovision — is also the most effective way to destroy evidence: in-memory state, the stack at the time, logs that only appear during the failure, all gone. That is how the same outage returns monthly and is "solved" by a restart every time. The countermeasures are cheap: keep one instance pulled out of traffic but not killed, grab a dump before restarting, archive that window's logs separately.
Examining means looking at each component's own behaviour to judge whether the whole is healthy. The chapter's toolkit has three layers:
The first move is to shrink the search space. On a path built from chained components, do not start at the first one and walk — cut in the middle: if input and output look right at that point, the fault is downstream; if it is already wrong there, the fault is upstream. Each cut removes half, so n stages converge in roughly log₂n cuts — about 3 for 7 hops, about 7 for 100 candidate builds. The same blade works along the time axis: bisect 100 commits to find the one that introduced a regression.
The second blade rests on an observation: a malfunctioning system is often still trying to do something — just not the thing you want. So drop the oversized question "why did it break" and ask three small ones: what is it doing? where are its resources going, or where is its output going? why there? Each answer takes you one layer down. The chapter's example demonstrates the drill cleanly:
The third blade is the cheapest and the most often skipped. The chapter puts it in physical terms: systems have inertia — a working computer system tends to remain in motion until acted upon by an external force, such as a configuration change or a shift in the kind of load being served. So the most recent change is always suspect number one, and a rollback is the fastest hypothesis test you own: if reverting fixes it, that was basically it.
For the blade to be usable there is a precondition: every layer needs change logging — from the binary version, configuration, experiments and flags of the servers handling user traffic, down to the packages installed on individual machines in the cluster. Without that record, "what changed recently" degenerates into oral archaeology in a chat channel.
Once you have a hypothesis you have to test it. The chapter's list of considerations is all scar tissue:
Inverted, the point lands harder: the most expensive thing in a long investigation is usually not running the tests, it is forgetting which ones you already ran.
Proving causality strictly requires a controlled experiment, which production rarely permits, so in practice we often settle for correlation plus a plausible mechanism — the chapter is candid about that. Two things raise your confidence: reproduce it in a test environment, and write the chain down — a postmortem covering what went wrong, how you tracked it down, how you fixed it and how to prevent a repeat (the chapter's own aside: ideally the system is alive at this point).
The chapter also devotes a section to something routinely thrown away: negative results are magic — "we tried X; it is not the cause / it did not get faster". The argument has four parts: ① a negative result is a conclusion, not a failure, and one clean negative can settle the hardest design arguments; ② the tools and methods built for the experiment usually outlive the experiment; ③ if you don't publish, the next person — quite possibly you in three months — walks into the same pit; ④ therefore, do publish negative results. The section targets a real industry bias: nobody wants to write up "we tried this and it didn't work".
Table 1 · Triage vs diagnosis: two jobs, two clocks
| Triage · stop the bleeding | Diagnosis · solve the case | |
|---|---|---|
| Goal | Make the system work as well as it can right now | Find out what caused it |
| Typical moves | Divert traffic / shed load / degrade / roll back | Metrics, logs, traces, profiles, bisection |
| Needs the cause? | No | The cause is its output |
| Timescale | Minutes — a cluster shift is often one command | Tens of minutes to hours, and unpredictable |
| Main cost | May destroy evidence; degraded UX meanwhile | Users keep absorbing the failure throughout |
| How to order them | In parallel, never in series. Stop the bleeding while keeping one instance out of traffic but un-restarted, with dumps and logs captured — then investigate. | |
Table 2 · Four diagnostic blades: when to reach for which
| Technique | Best at | Precondition | Where it fails |
|---|---|---|---|
| Bisection | "One hop on the path is broken"; a regression from some commit | Each hop is independently probeable; the fault reproduces | Intermittent faults — your one probe lands on a good moment |
| What / where / why | Resource exhaustion (CPU, memory, handles, connections) | Profiles and layered timing data exist | A system that is doing nothing at all — not busy wrongly, just stuck |
| What touched it last | Roughly 70% of outages — the ones trailing a change | Change logs at every layer (versions, config, flags, packages) | The other 30%: load-mix shifts, expired certs, hitting a capacity wall, upstream behaviour changes |
| Service-specific tools | State that generic tools simply cannot see | Someone invests in building them ahead of time | Only the author can drive them; not in the playbook means not existing |
Table 3 · Ways of examining: what each answers, and its blind spot
| Instrument | Answers | Steady-state cost | Blind spot |
|---|---|---|---|
| White-box metrics | Which component is unhappy, and since when | Low; can run at full rate permanently | Component granularity only; more metrics, more spurious correlation |
| Text logs | What specifically happened at that moment | Medium; verbose logging drags the system | Anything uninstrumented is dark; hard to aggregate at volume |
| Distributed tracing | Which hop a request's time went to | Low (kept down by sampling) | Low sampling rates miss rare bad requests |
| Profiling | Which function the CPU or memory goes to | Mostly on demand | Shows what is busy, not what is waiting |
Table 4 · Ordering your tests when checking a hypothesis (lowest risk first)
| Kind | Example | Information | Risk / side effects |
|---|---|---|---|
| Read-only observation | Read dashboards, check the change log, open one trace | Medium | Near zero — always exhaust this tier first |
| Off-path reproduction | Replay requests in a test environment, load one isolated replica | High | Low; but it may refuse to reproduce (environment mismatch) |
| Reversible production change | Roll back a version, flip off a flag, retune one machine | Very high (if the rollback fixes it, that was it) | Medium; do it on a small slice of traffic first |
| Invasive action | Enable verbose logging, take a heap dump, restart the primary | High | Changes later results: verbose logging worsens latency, a restart erases the scene |
Nearly every one of the chapter's criteria shows up, positively or negatively, in public incident write-ups. Day to day: on call, ask "can we divert, can we degrade" before "why"; in postmortems, check whether stopping the bleeding also wiped the evidence; in observability choices, first confirm you can split a request's latency by hop; in interviews, the model answer to "error rate jumped to 3%, what now" is exactly this order — mitigate, check recent changes, bisect the path, then drill on where the effort goes.
merge_join calls — which often indicate suboptimal indexing; they then used Dapper to trace individual requests from the frontend reverse proxy through to the app's response, examining the RPCs issued by each server along the way. Methodologically that is Fig. 2: descending from "the whole thing got worse" to "this hop got worse".SRE Book Ch.12 Effective Troubleshooting (free online) ↗pg_dump had been failing silently for a long time because of a version mismatch). Recovery came from a staging snapshot, losing roughly six hours of data. GitLab live-streamed the recovery and published the postmortem — an expensive demonstration of this chapter's "do publish negative results".Postmortem of database outage of January 31 ↗① In one line: troubleshooting is not a gift but a decomposable, teachable craft = a generic hypothetico-deductive process × concrete knowledge of this system. Neither half works alone.
② The process is a loop: problem report → triage → examine → diagnose → test/treat → cure, and a falsified hypothesis sends you back to Examine for another.
③ The ordering to memorise: stop the bleeding first — "you aren't helping your users if the system dies while you're root-causing." Triage and diagnosis run in parallel, never in series.
④ But fast triage does not license destroying evidence: a restart is the best mitigation and the best evidence shredder, so keep one instance out of traffic and un-restarted, and capture dumps and logs first.
⑤ Learn the four hallmarks of ineffective troubleshooting: wrong symptom or misread metric, untestable hypotheses, clinging to last time's cause, and chasing spurious correlations (more metrics, more coincidences).
⑥ Four diagnostic blades: bisection (log₂n cuts — about 3 for 7 hops), what / where / why (Spanner → CPU → log sorting → a backtracking regex, four layers deep), what touched it last (~70% of outages trail a change; rollback is the fastest hypothesis test), and service-specific tools.
⑦ Make alternatives mutually exclusive, order tests by likelihood against risk, watch for side effects of active tests (verbose logging worsens the very latency you are chasing), and write down what you tried — the priciest overhead is forgetting what you already ruled out.
⑧ In practice you often settle for "correlation plus a plausible mechanism"; and negative results are magic: they are conclusions rather than failures, the tools outlive the experiment, and not publishing means someone (probably you, in three months) repeats it — so publish.
⑨ Examining has three layers: metrics say which component, logs say what happened at that moment, traces say which hop the time went to. And the chapter's closing claim is the most valuable of all: how easy an outage is to debug was decided on design day — build observability into every component and keep the interfaces between them clear and observable, and that beats any after-the-fact technique.