BOOK DEEP-READ · SRE · CHAPTER 15
Site Reliability Engineering · Ch 15 · John Lunney & Sue Lueder · Google · 2016
The site went down, and two hours later it was back. Then what? At most companies the answer is "then nothing" — everyone exhales and goes back to work, and three months later the exact same thing happens again. Chapter 15 of Google's SRE book is about that "then what": writing the incident down, explaining it properly, and turning it into a list of changes that must actually get made. The document is called a postmortem.
Aviation has an old rule: after a crash you investigate the aircraft, not the pilot. Not because pilots never err — often one really did press the wrong switch. But what investigators actually want to know is: why does that switch look identical to the one beside it? Why does the manual disagree with what the cockpit displays? Why did the wrong action look right to him at that moment? Answer those and the next pilot won't press it. Conclude "pilot error" and the switch is still there, waiting for the next person.
The most valuable line in the chapter is the engineering version of that idea: you can't "fix" people, but you can fix systems and processes.
It's hard for a very human reason: the moment you start hunting for someone to blame, people stop talking.
And the information you need lives only in that person's head — what they could see at the time, and why the step looked reasonable. No monitoring system can say it for them. But once they know that speaking up gets logged against them, they'll only mention the parts that couldn't be hidden. Next time, they simply won't report the small stuff at all. So your incident log fills with big outages and looks reassuringly calm, while the small problems quietly pile up in the dark.
① Delete the question "who did this?" from the room. The premise is an assumption: everyone involved acted in good faith and did what looked right given the information they had. So the question changes from "who got it wrong" to "why did that choice look right at the time?" This isn't being nice to people; it's a trade. You give up the name, and you get the part of the story only that person knows.
② Agree in advance on what forces a postmortem. How long a user-visible failure has to last, any data loss at all, whether someone on call had to get up at night and intervene by hand, how long the fix took, and — whether monitoring missed it and a user noticed first. Why in advance? Because if you argue about "was this a big one?" after the fact, the answer shifts with who's in the room and who's annoyed that day.
③ The output isn't a document, it's a to-do list. However moving the writing is, without a few lines of "change which system, who does it, by when," it's just an essay. And someone has to actually read it — the book is blunt about this: a postmortem that hasn't been reviewed might as well never have existed.
④ Old postmortems are the best teaching material. Google's approach is rather fun: dig out a postmortem from months or even years ago and read it together — the people who were there, the bystanders, and the new hires. There's also a rehearsal called the "Wheel of Misfortune": take an old postmortem and re-enact it, with newcomers playing the roles of the people who lived it.
The tuition has already been paid, and the postmortem is the only move that converts it into something: don't chase the person, chase "why this looked reasonable at the time," because the information you need is only in their mouth; agree beforehand on what must be written up, make sure someone reviews it, and remember the real product is a set of changes tracked to closed. One honest cost: blamelessness is easy to fake — no blame in the meeting, but the incident still shows up at review time. If people can smell that, none of the rest works.
Want the full process, comparison tables and real public postmortems? → Switch to the deep read
SRE Chapter 15 establishes one rule: the incident has already been paid for, and the postmortem is the only action that converts that cost into an asset. Whether it works turns on one counterintuitive design decision — deleting "who got it wrong" from the process entirely. Not out of kindness, but because the information you most need exists only in the head of the person involved: what they could see, and why that choice looked right at the time. Start hunting for a name and you never get it. The chapter's own line: "You can't 'fix' people, but you can fix systems and processes to better support people making the right choices when designing and maintaining complex systems." And a harder test: a postmortem that hasn't been reviewed might as well never have existed.
Written by John Lunney and Sue Lueder. The chapter sits in Part III, "Practices," right after the incident-handling run: it follows Ch. 12 "Effective Troubleshooting" (how to find the fault), Ch. 13 "Emergency Response" and Ch. 14 "Managing Incidents" (how to organize people during one), and it governs the 48 hours after the incident ends. It is also the repayment channel for Ch. 3 "Embracing Risk" — the slice of error budget you burned is only bought back by the action items a postmortem produces. In the field the names differ (incident review, post-incident review / PIR, correction of errors / COE) but the work is the same; it also maps to an interview question you will almost certainly get: "tell me about a production incident you were part of, and what you changed afterwards."
The chapter's epigraph is its whole economics: "The cost of failure is education." — Devin Carraway. In engineering terms: the incident has already spent the money — users hurt, error budget burned, somebody's weekend gone. The only possible return on that spend is learning something and landing it. Skipping the postmortem is paying tuition and not attending the class.
Bill it to the SLO and it gets concrete: a 99.9% monthly availability target allows roughly 43 minutes of unavailability for the whole month (the Ch. 3/4 arithmetic). One 30-minute incident eats about 70% of that; the same class of incident returning three times in a quarter clears out the year. So "stop it happening again" is not a moral request, it's budget arithmetic.
Three specific ailments are in scope. First, lessons that never land: it's fixed, someone posts "all good" in chat, the lesson lives in two or three heads, one of them changes teams six months later, and the same failure replays. Second, blame-induced silence — the chapter's firmest claim: removing blame gives people the confidence to escalate issues without fear; conversely, an atmosphere of blame risks creating a culture in which incidents and issues get swept under the rug, leaving the organization exposed to greater risk. Third, no criteria: if nobody agreed in advance on what requires a postmortem, the working rule becomes "write one when it's too big to hide," and your incident record ends up containing nothing but P0s.
The chapter's definition is a set of five things: a written record of an incident, covering its impact, the actions taken to mitigate or resolve it, the root cause(s), and the follow-up actions to prevent recurrence. Four of the five look backwards; only the last looks forwards — and the chapter's weight rests on that last one.
The ordering of the three stated goals is deliberate too: (1) ensure the incident is documented; (2) ensure all contributing root causes are well understood; (3) and especially, ensure effective preventive actions are put in place to reduce the likelihood and impact of recurrence. That "especially" is not decoration — it means a beautifully written postmortem with an exact timeline and no action items is, by this chapter's test, a failure. So the fastest way to judge whether a team's postmortem culture is real is not to read the documents but to check whether the action items got closed: each one needs an owner, a sensible priority, and a ticket in the bug tracker. Otherwise you get a lot of "we'll be more careful next time," which changes nothing.
One more test that's easy to miss: the reader of your postmortem is not your manager, it's the on-call engineer six months from now who has never met you. They'll find it at 3 a.m. because they're seeing similar symptoms. Write for them and your timeline and terminology come out right by themselves.
The chapter is explicit: postmortem criteria should be agreed upon and published before incidents occur. Its list of common triggers:
The easiest one to skip is the last: monitoring failed and a human noticed first. It means the user told you before your systems did — and even if the impact was small and it recovered in minutes, the hole in your monitoring is worth a postmortem on its own (echoing Ch. 6). Likewise, "an on-call engineer intervened" implies something sharper: needing a human awake at 3 a.m. typing commands is itself a defect awaiting repair.
The chapter pairs the criteria with a line that offsets the pressure they create: writing a postmortem is not punishment — it is a learning opportunity for the entire company. If a team hears "we need a postmortem" as "someone is about to be punished," then the stricter the criteria the worse the under-reporting. The two must be introduced together.
The chapter's definition is an operational assumption: a blamelessly written postmortem assumes that everyone involved in an incident had good intentions and did the right thing with the information they had. Note the tense — the information they had at the time. That's the pivot of the whole mechanism: whoever reads the postmortem already knows the ending and carries hindsight bias, so the clues look obvious; the person in the moment had none of that.
So the questions have to change. Not "why did you press that button" but "why did pressing that button look correct at the time?"; not "who changed the config" but "what conditions made that change look safe?" The chapter frames the shift as moving from assigning blame to investigating the systematic reasons why an individual or team had incomplete or incorrect information — only then can an effective prevention plan be built.
None of this was invented by software. The chapter says plainly that it originated in the healthcare and avionics industries, where mistakes can be fatal and so every "mistake" is cultivated as a chance to strengthen the system. It condenses into the line worth memorizing: "You can't 'fix' people, but you can fix systems and processes to better support people making the right choices when designing and maintaining complex systems." The chapter also contrasts two ways of writing the same technical judgement (this backend is bad enough that it breaks weekly and should be rewritten): as an exasperated complaint that implies somebody is incompetent, or as a forward-looking case (rewriting it really would stop these pages, the maintenance manual is too long for anyone to fully learn, and future on-callers will thank us) — only the second version is something a colleague can pick up and act on.
The previous section was values; this one is mechanism. The chapter is direct: removing blame gives people the confidence to escalate issues without fear; conversely, an atmosphere of blame risks creating a culture in which incidents and issues get swept under the rug, leaving the organization exposed to greater risk.
Why is that worse than it sounds? Because what it corrupts is not morale, it's data. Every reliability decision rests on knowing what has actually gone wrong: how to spend the error budget, what to fix next quarter, where to add alerts. Once small incidents stop being reported, your incident record holds only the ones that couldn't be hidden — a survivorship-biased sample. You conclude the system is stable, when in truth nobody is talking. Which is why blamelessness has to be built as a process rather than an attitude: attitudes change with people and with performance reviews; processes don't.
The chapter treats postmortems as collaborative from end to end and names the capabilities that matter: real-time collaboration (several people editing at once, while the information is still warm), an open commenting and annotation system (anyone can question any line), and email notifications (stakeholders don't have to go looking). These sound like tool features, but they decide whether a postmortem gets finished before the information evaporates.
Then it's reviewed — by senior engineers who weren't directly involved, checking a specific set of questions: was the key incident data captured for posterity? are the impact assessments complete? was the root cause dug into deeply enough? is the action plan appropriate, and are the resulting bug fixes at an appropriate priority? was the outcome shared with the relevant stakeholders?
And it states one best practice bluntly: no postmortem left unreviewed — an unreviewed postmortem might as well never have existed. The reasoning is practical: writing is private, and review is the step that merges the knowledge into the organization.
The chapter knows process alone isn't enough, and spends a section on how to make postmortems a habit, with practices concrete enough to copy:
One further best practice deserves its own line: visibly reward people for doing the right thing — especially for owning up to their own mistakes. It is the cheapest and most effective fuel a blameless culture has. The chapter also argues the whole thing should be measured rather than felt: track how many get written and how the action items are closing, and ask engineers directly whether they genuinely believe the culture here is blameless — because the one way this fails is everyone agreeing out loud and believing none of it.
The chapter ends on continuous improvement, and the direction is tooling: a common template, and populating the incident data (timelines, monitoring graphs, alert records) automatically wherever possible. The logic is plain — the more friction, the fewer people write; a document that takes three hours to assemble by hand will only ever be written for major outages. Google has also published its template and a complete example postmortem (Appendix D, a service overload incident) that you can take and adapt.
Table 1 · Chasing the person vs chasing the conditions: what each buys you
| Blaming (whose fault) | Blameless (why it looked right) | |
|---|---|---|
| Core question | Who did this, and why did they break process | What conditions made the action look correct at the time |
| What you get | A name plus "I'll be more careful" | Conditions only the responder knew → changeable action items |
| Effect next time | The trap is still there for the next person | The path is closed; that route no longer works |
| Effect on reporting | Small incidents stop being reported; your data becomes survivorship-biased | People escalate issues without fear |
| Apparent upside | Looks "rigorous" and accountable; easy to report upward | Looks like "nobody is responsible"; needs explaining to leadership |
| Real cost | Organizational risk keeps accumulating out of sight | Worthless if performed rather than meant — see §7 on blameless ≠ unaccountable |
Table 2 · When one is mandatory (thresholds are illustrative — the book says "a threshold"; the numbers come from your SLO)
| Trigger | Example threshold | Cost of skipping it |
|---|---|---|
| User-visible downtime / degradation | > 1% of requests for > 5 minutes; or > 10% of the monthly error budget | The budget is nibbled away and you notice only at year end |
| Data loss of any kind | No threshold — write one for a single lost row | Data problems almost always recur, and the second time is bigger |
| On-call intervention | Any manual action: rollback, traffic drain, manual restart | "A human has to type this at 3 a.m." never gets fixed (toil, Ch. 5) |
| Long resolution time | > 60 minutes from alert to recovery | The real MTTR drivers (stale runbooks, missing access, nobody reachable) go uninvestigated |
| Monitoring failure | Found by a user, support or a manual check | The monitoring gap stays open and users keep being your alerting system |
| Any stakeholder asks | No justification needed | The judgement of "is this worth writing up" concentrates in a few hands |
Table 3 · Four ways postmortems fail: how to spot them, how to treat them
| Failure mode | Symptom | Treatment |
|---|---|---|
| The essay | Immaculate timeline, no action items — or only "we should be careful" | Every action item becomes a ticket with an owner and a priority; audit the close rate |
| The manhunt | Named individual failings in the text; tense review meetings | Swap the question template to "why did this look right"; reward people publicly for owning mistakes |
| Big-incidents-only | Only P0s in the record; small problems leave no trace | Fix the triggers in advance (Table 2), especially the monitoring-failure one |
| Written, never read | The doc sits in a folder; nobody can find it six months later | Mandatory review plus broadcast; keep old ones in circulation via reading clubs and role play |
Table 4 · Three kinds of action item — a good postmortem has all three
| Kind | Goal | Typical example | Speed / cost |
|---|---|---|---|
| Prevent recurrence | The same cause can no longer produce the same result | Add a target-cluster health check before pushes; require confirmation for dangerous commands | Slowest, highest value; usually needs code changes |
| Reduce impact | If it happens again, it hurts an order of magnitude less | Degradation switches, rate limits, partitioning the blast radius | Moderate cost; also helps against failures you haven't imagined |
| Detect and recover faster | Push MTTR down | Add a symptom alert, update the runbook, make rollback one command | Cheapest and fastest; usually the items you can finish this week |
If you only have time for one category, do the third. "Prevent recurrence" often waits a quarter for scheduling, while "add an alert, make rollback one command" usually lands within the week — and it helps with the next, completely different incident too.
The chapter lands in a lot of daily places: at on-call handover, check for unclosed action items first; the opening line of an incident review should be "we're not looking for a person, we're looking for conditions"; and in interviews, when asked to describe an incident you caused, the strong answer isn't self-flagellation — it's how big the impact was, why the judgement was reasonable at the time, what you changed afterwards, and whether that change was ever verified. Outside engineering it has a very practical use too: the public postmortem is now the default form of outage communication, and how well it's written directly affects customer trust.
① In one line: the incident has already been paid for, and the postmortem is the only action that converts it into an asset — the epigraph, "the cost of failure is education," is the chapter's whole economics.
② A postmortem is five things: the record, the impact, the mitigating/resolving actions, the root cause(s), and the follow-up actions that prevent recurrence. Of the three stated goals, the chapter singles out the last ("and especially" that effective preventive actions are put in place).
③ Fastest way to tell real from theatre: don't read the documents, check whether the action items closed. No owner, no priority, no ticket means it doesn't exist.
④ Blameless is defined as an assumption: everyone had good intentions and did the right thing with the information they had. So the question becomes "why did this look right at the time." The practice comes from healthcare and avionics, and condenses into the line worth memorizing: "You can't 'fix' people, but you can fix systems and processes."
⑤ The cost of blame is under-reporting. It corrupts not morale but the incident data you make decisions from — a record containing only P0s is a survivorship-biased sample.
⑥ Fix the triggers in advance: user-visible downtime past a threshold, data loss of any kind, on-call intervention, resolution time past a threshold, a monitoring failure, plus "any stakeholder may request one." The last two get skipped most and are worth the most.
⑦ Postmortems are collaborative, and every one must be reviewed by senior engineers who weren't involved against five questions (data captured / impact complete / root cause deep enough / plan and priorities right / shared with stakeholders). An unreviewed postmortem might as well never have existed.
⑧ Practices that turn it into culture: postmortem of the month, an internal postmortem group, reading clubs (often on postmortems years old), and Wheel of Misfortune role play, plus visibly rewarding people who own their mistakes — and measure it, not only by counting documents but by asking engineers whether they truly believe it's blameless. Finally, lower the friction of writing one (templates plus auto-populated incident data): fewer writers means shorter organizational memory.