BOOK DEEP-READ · SRE · CHAPTER 4
Site Reliability Engineering · Ch 4 · Google (Chris Jones, John Wilkes, Niall Murphy, Cody Smith) · 2016
Whenever you order food, hail a ride or open a video app, you carry a private line in your head: what counts as "fine" and what counts as "unacceptable." The companies behind those services need that line written down too — otherwise "how are we doing today?" is settled by whoever talks loudest. Chapter 4 of Google's SRE book is about turning that line into a number everyone accepts.
A takeout shop wants to promise "fast delivery." That sentence is useless until it is split into three separate things: what you measure (minutes from order to doorbell), where you draw the line (95% of orders inside 30 minutes this month), and what you owe if you miss (a coupon). The book calls these SLI, SLO and SLA, and they get blurred together constantly. The trick to telling them apart is simple: ask "what happens if we miss it?" — only the one with a stated penalty is a contract.
The hardest of the three is the first one: what to measure. Plenty of teams have dashboards glowing green while users write one-star reviews. The reason is mundane: they measure "did the server answer," while users care about "did I get what I came for." Google published one such comparison: for the same search service, counting "the server returned a normal response" gave a respectable 99.7%; counting "the user actually saw results within five seconds" left about 80%.
This chapter flips the order: first ask what users care about, then find a way to measure it — rather than surveying the metrics you already collect and picking a flattering one. Two more traps come with it.
First, don't look only at the average. "Average delivery: 25 minutes" can mean half the orders arrive in 15 and half in 35 — and the half that leaves bad reviews has been averaged away. Line the orders up fastest to slowest and look at the miserable tail. Second, write the number out in full. "95% of orders within 30 minutes" isn't enough; you have to say 95% over what period, and which end the clock stops at — when the courier taps "delivered," or when you actually open the door. Leave that unwritten and two departments will each compute their own "we hit it."
Promise 30 minutes, deliver in 12 for years, and that sounds like pure upside. But over time everyone reorganizes their life around 12 minutes and nobody keeps a buffer; the day traffic makes it 28 minutes — still within the promise — lunch collapses for the whole building. An internal Google component that half the company depended on hit exactly this: it was so steady that everyone assumed it could never break, so nobody wrote a fallback. The fix was blunt: when it runs too well, deliberately take it down for a while. The honest cost: almost all the effort sits up front — translating "what users care about" into a number that is both measurable and agreed on is usually harder than watching it afterwards.
Don't open with an argument about how many nines. First work out what users actually care about, turn it into a number that states where it is measured and over what window, watch the worst-off sliver rather than the average, and pick a target few enough that you dare settle an argument with it. Too bad needs fixing — and too good needs people reminded that you can break.
Want the mechanisms, numbers and diagrams? → Switch to the deep read
You think setting an SLO means choosing how many nines. That is the last step, and the cheapest one. The real work comes before it: find out what users care about, translate it into something measurable (an SLI), nail down where it is measured and over what window, then pick a target that is just good enough — and that you would actually cite to win a prioritization argument. An SLA is that target written into a contract with a penalty attached; SRE owns the first two steps and hands the third to business and legal. Choose the wrong indicator and every nine after it is self-deception.
p99): line up requests by duration; the 99th-percentile value tells you how bad it is for the worst-off users. p50 is the median.1 − SLO, the allowance for being broken, spendable on release velocity (covered in Ch3).This chapter follows Ch3 "Embracing Risk" inside Part II, "Principles." Ch3 argued that a reliability target must be below 100% and handed you the error budget, but left a gap: how do you pick the actual number, and what do you measure? This chapter fills it. Downstream, Ch6 "Monitoring Distributed Systems" makes alerting SLO-driven. Real-world match: any team that has to answer "is our service good enough," "can we ship this," or "what should we promise externally."
Ailment one: the three words get used interchangeably. An engineer's "SLA" usually means an internal target, a salesperson's means a contract clause, and the dashboard labelled "SLA" is really plotting an SLI. The vocabulary doesn't line up, so the discussion never converges. The book's test is one sentence: ask what happens if it isn't met — if there is no explicit consequence, you are almost certainly looking at an SLO, not an SLA. Ailment two: measuring the wrong thing. You measure "did the server return 200"; users care about "did I get a result within a tolerable time." Those agree in normal operation and diverge violently during an incident.
Ailment three: with no shared referee, priority is decided by volume. Developers are measured on what they ship, SRE on nothing breaking; the same event pulls the two in opposite directions. Hence the book's sharp self-test: if you can never win a conversation about priorities by quoting a particular SLO, it is probably not worth having that SLO.
What happens if you don't fix it? Take a checkout endpoint serving 20 million calls a day. A success-rate SLO of 99.9% allows roughly 20,000 failures a day; 99.99% leaves 2,000 — one three-minute burst of errors eats the whole thing. One extra decimal place flips the answer to "should we ship this?" — and if that number was never seriously chosen, the answer comes from the most senior person in the room instead.
The SLI is the measurement — note the phrase "carefully defined"; section 3 is entirely about it. The SLO is the line, structured either as SLI ≤ target or lower bound ≤ SLI ≤ upper bound. The SLA is the contract: an SLO plus what happens when it is missed, usually a refund or penalty. The book is explicit about ownership: SRE typically does not write SLAs (they are tied to business and product decisions), but SRE assesses how hard those SLOs are to meet and is responsible for not triggering the penalty clause. A concrete contrast: Google Search has no public SLA, only internal SLOs; the products sold to enterprises do have real SLAs.
Table 1 · Three words, one question: what happens if it is missed?
| SLI · indicator | SLO · objective | SLA · contract | |
|---|---|---|---|
| What it is | A measurable quantity | A line drawn on that quantity | The line + a consequence |
| Example | Successful-request ratio over 5 min | ≥ 99.9% over a 28-day window | Below 99.9% → 10% bill credit |
| Who decides | SRE together with users | SRE together with product | Business and legal; SRE only assesses risk |
| If missed | N/A — it is just a number | Freeze releases, reprioritise; stricter than the SLA | You pay; looser than the SLO, and advertise conservatively |
The most-skipped warning in the chapter: start from what is easy to measure and you will end up with a pile of useless SLOs. The right order is to find out what users care about and then approximate it — what they care about is often hard to measure, so you settle for a proxy, but "finding a proxy for the user's experience" and "picking one of the metrics you already have" are different activities. Quantity matters too: too many indicators and none is really watched; too few and large parts of the system go unexamined — the answer is a handful (Table 2). One boundary is worth pulling out: everyone cares about correctness, but it is a property of the data, not of the infrastructure, so it is usually not SRE's to guarantee.
Table 2 · Default indicators by system type
| System type | Core SLIs | The question being answered |
|---|---|---|
| User-facing serving | Availability, latency, throughput | Could we respond? How long did it take? How many could we handle? |
| Storage | Latency, availability, durability | How long to read/write? Can we access it on demand? Is the data still there? |
| Big data / pipelines | Throughput, end-to-end latency | How much data processed? How long from ingestion to completion? (per-stage targets where needed) |
| Everything | Correctness | Is the answer right — everyone cares, but it is a property of the data, not usually SRE's to meet |
First, where. Most indicators are naturally collected server-side, but the book warns that server-side-only measurement misses an entire class of problems that hurt users without showing up in server metrics — when page JavaScript stalls, every backend request is a clean 200 with a beautiful latency while the user stares at a blank page. There, "how long until the page becomes usable" is the better proxy.
Then aggregation, with two traps. The first is the window: one system serving 200 requests/s on even seconds and 0 on odd ones has exactly the same per-minute average as one serving a steady 100/s, yet its instantaneous load is twice as large — stretch the window and the spike disappears. The second is the average: it hides the fact that most requests are fast while a long tail is very much slower. Hence the principle: most metrics are better thought of as distributions than as averages. A high-order percentile (p99, p99.9) gives you a plausible worst case; the median describes the typical case; the higher the variance, the more the typical experience is shaped by the tail. The book also cites the finding that people generally prefer a slightly slower system to one with high variance in response time.
Finally, standardise. The book recommends turning SLI definitions into templates: aggregation interval ("averaged over 1 minute"), aggregation region ("all the tasks in a cluster"), measurement frequency ("every 10 seconds"), which requests count ("HTTP GETs from black-box probes"), how the data is acquired, and how latency is defined ("time to last byte"). Those fussy conventions are exactly what decide whether two teams can reconcile their numbers — "time to first byte" and "time to last byte" can differ by an order of magnitude on a streaming response.
A usable SLO nails down the percentile, the aggregation window and the measurement point (the book's model wording is the last row of Table 3). If the shape of the curve matters, write several targets (90% / 1 ms, 99% / 10 ms, 99.9% / 100 ms). If your load splits into classes, define one objective per class: bulk clients care about throughput (95% of Set calls under 1 s), interactive clients about latency (99% of small Set calls under 10 ms) — cram both audiences into one SLO and you usually satisfy neither. One hard constraint carries over from Ch3: insisting that an SLO be met 100% of the time is neither realistic nor desirable; allow an error budget and track its burn daily.
An SLO drives a loop: (1) measure the SLI → (2) compare it to the SLO and decide whether action is needed → (3) work out what would meet the target → (4) do it. The book's example: latency is climbing and will miss the SLO in a few hours, so you test the hypothesis that the servers are CPU-bound, then add servers to spread the load. Without the SLO you know neither whether to act nor when. This is the upstream of Ch6's rule — alerts should fire because an SLO is about to be missed, not because one machine hit 80% CPU.
Publishing an SLO sets expectations. Tactic one is a safety margin: run against a tighter internal SLO than the one you advertise, so chronic problems can be handled before they become externally visible. Tactic two is deeply counterintuitive: don't overachieve — users build on the reality of what you offer, not on what you say you'll supply. Which brings the chapter's most famous story: Google's distributed lock service Chubby ran far above its SLO for years, so reliably that dependent services simply assumed it was always up and nobody wrote a fallback path; when it did wobble, a crowd of unrelated services failed with it. SRE's answer was planned outages: when quarterly availability exceeds the target, deliberately take the system down to bring it back toward the target line, surfacing unreasonable dependencies early instead of letting them accumulate until a real outage.
The trade-offs here are not about which technology to use, but about where the line goes, how precisely it is written, and who defends it.
Table 3 · Four ways to write the same latency SLO, from useless to usable
| Wording | What's wrong | Consequence |
|---|---|---|
| "Latency should be low" | No quantity, no line | The argument never ends; nobody can rule on compliance |
| "Average latency < 100 ms" | Uses the mean | The tail is averaged away: the target is met while a slice of users waits seconds |
| "p99 < 100 ms" | No window, no measurement point | Two teams each compute "we met it" — over a minute or a day? at the client or the server? |
| "99% of Get calls (averaged over 1 minute) complete in under 100 ms, measured across all backend servers" | — | Actionable, arguable, alertable; add p90 / p99.9 if the shape matters |
Table 4 · The book's five rules for choosing targets, and the cost of breaking each
| Rule | The book's reasoning | Typical outcome if ignored |
|---|---|---|
| Don't pick a target based on current performance | Copying today's numbers assumes today's architecture is right | You lock yourself into a system that requires heroics to sustain and can't improve without a redesign |
| Keep it simple | Complicated aggregations obscure changes and are hard to reason about | Nobody can explain why the target broke; step (3) of the loop stalls |
| Avoid absolutes | "Infinitely scalable, always available" is unrealistic and expensive | The target is decoration from day one |
| Have as few SLOs as possible | Just enough for good coverage — and you must be able to defend them | Dozens of SLOs, none ever quoted in a prioritization meeting — so none should exist |
| Perfection can wait | Refine and tighten as you learn the system | Set it too strict, miss it, relax it — a target that has been relaxed has no authority left |
Tight or loose? Too tight: the error budget is empty every week, release freezes go from exception to norm, alert fatigue follows and the team learns to ignore alerts — worse than having no SLO at all. Too loose: users are already leaving one-star reviews while the dashboard stays green. The workable compromise is to set a defensible line at a user-perceptible threshold, then write down when and by how much it will be tightened.
Where to measure, and whether to offer an SLA? Server-side is cheap, complete and quick to debug but furthest from the user; client-side is closest to real experience at the cost of instrumentation and noise. Most teams end up mixed: the primary SLO sits server-side so it can be measured reliably, with a small set of client metrics used for calibration — a lasting divergence between the two means the primary is measuring the wrong thing. As for SLAs, the broader the audience, the harder it becomes to change or withdraw an unwise one, so advertise conservatively and pin down how things are counted: Google Compute Engine's SLA defines Downtime as an interruption of one or more consecutive minutes — partial minutes don't count at all.
This chapter turned reliability from an adjective into an engineering contract you can hand over. The SLO objects and error-budget burn-rate alerts you find today in Prometheus, Grafana and Cloud Monitoring are essentially its productisation — "can we ship this?" went from being a meeting to being a query. It also rewrote the right answer in interviews: asked "what's your SLO," replying 99.9% is a fail. The passing answer is a chain — which indicator, measured at which layer, aggregated over which window, why that number, and what concretely happens when it breaks. That last one is the tell: if you can't name a consequence, it's still decoration.
200 OK responses gave 99.7% availability; counting "users who actually got results within 5 seconds" left about 80% (18% slower, 1% timed out, 1% errored) — confirming that the wrong measurement point makes an SLO self-deception. AJ Ross & Matt Brown, Google Cloud, "CRE life lessons: Available or not?", 2017 ↗99.9% externally over one month against 99.95% internally — confirming both the safety margin and "don't overachieve". Google Cloud, "SRE fundamentals: SLIs, SLAs and SLOs", 2018 ↗≥ 99.99% monthly uptime for instances in multiple zones, ≥ 99.9% for a single instance, with 10% / 25% / 100% bill credits by tier when missed — and Downtime defined as one or more consecutive minutes. Google Compute Engine Service Level Agreement ↗p99.9 has a cost too: high-order percentiles are statistically noisy and swing wildly on low-traffic services, producing false alerts.1 · The work happens before the nines — find out what users care about, translate it into a measurable SLI, and pin down where it is measured and how it is aggregated.
2 · The test for the three words: ask what happens if it is missed. No explicit consequence means SLO; a payout means SLA. SRE owns SLI and SLO; the SLA belongs to business and legal.
3 · Choose indicators from the user inward, not from what you can already measure, and keep them few and sharp. Serving systems: availability, latency, throughput; storage adds durability; pipelines: throughput and end-to-end latency. Correctness is a property of the data and usually not SRE's to meet.
4 · The measurement point decides the truth: server-side-only measurement misses problems that hurt users invisibly (99.7% vs 80%).
5 · Two aggregation traps: the window flattens spikes, the average flattens the tail. Treat metrics as distributions and use percentiles.
6 · A usable SLO reads: "99% of Get calls (averaged over 1 minute) complete in under 100 ms, measured across all backend servers." Add percentiles to constrain shape, split by workload class, and never demand 100%.
7 · Five rules: don't copy current performance, keep it simple, avoid absolutes, have as few as possible, perfection can wait. Self-test: could quoting it win a prioritization argument?
8 · SLOs drive a four-step loop — measure, compare, decide, act — and alerts should fire on an SLO about to break. Two expectation tactics: keep the internal target stricter than the public promise, and don't overachieve — Chubby was so reliable it was treated as infallible, and planned outages were what pulled expectations back to the line.