BOOK DEEP READ · SRE · CHAPTER 3
Site Reliability Engineering · Ch 3 · Google (Marc Alvidrez) · 2016
No app promises you it will never break. But inside a company, someone has to answer a very concrete question: how much broken time are we willing to allow ourselves in a year? Chapter 3 of Google's SRE book is about how Google not only refuses to dodge that question — it turns the answer into a budget you get to spend.
You want your home to be safer. A good lock costs a little. A reinforced door costs more. A ring of cameras costs a lot. A guard on duty around the clock costs a fortune. Each step up multiplies the price while adding only a sliver of safety. At some step you notice something: the extra money you're spending on "safer" now exceeds the value of everything in the house. Past that point you're not being careful — you're losing money.
Even if you're willing to pay, your users may not notice. Between them and your service sit a phone, a Wi-Fi router, a home broadband line, a carrier network — and that whole path is not very reliable to begin with. The book puts it bluntly: someone using a phone that flakes out now and then cannot tell the difference between a service that almost never breaks and one that almost-almost never breaks.
If "never break" is both expensive and invisible, then state an allowance for breaking, openly: say how long this service may be broken in a year, and that gap is its budget.
Then run it like a family travel budget. Money left? Go out and enjoy it — ship freely, run bold experiments, and small mishaps simply draw down the budget. Money gone? Stay home — new features stop, everyone turns to fixing stability, and you wait for the next allowance. The old argument over "should we ship this one" — previously settled by whoever was loudest or most senior — becomes a glance at how much is left in the ledger.
One detail people miss: how do you count "how long it broke"? The instinct is to watch the clock — how many minutes were we down today. But a service spread across the world almost never falls over as a whole; it breaks a little here and a little there. So the more useful method is to count heads: of all the requests that arrived today, how many were not served properly. That way "half-broken" makes it into the ledger too.
The allowance isn't one number for the whole company. An internal reporting tool and a payments system were never meant to have the same target. And a single storage system serving both "click and get an answer now" workloads and "crunch a batch overnight" workloads should offer two tiers at two prices, so the people using it pick what they need — want it steadier, pay more.
One honest cost: this only works if you actually dare to stop. If every empty budget can be waived with "but this feature is really important," it's just a pretty decoration on a dashboard.
Reliability isn't better the higher it goes — it is a thing with a price. Rather than swearing "we will never break," admit up front how long you may break in a year and treat that as budget: run hard while there's balance, stop and fix the road when it's gone. The argument becomes an audit.
Want the mechanisms, the numbers and the diagrams? → Switch to the deep read
You'd assume reliability is "the higher the better, push it as far as you can." In fact, past a point, each extra increment is invisible to users and a net loss to the business. This chapter converts reliability from a moral question into an accounting one: pick an explicit target below 100%, and treat "1 minus the target" as money you may spend — the error budget. Balance left, ship freely; balance gone, stop and fix. SRE never manages "not breaking"; it manages the measurement, pricing and allocation of risk.
99.9%, 99.99%.1 − SLO. A 99.99% target leaves 0.01% of permitted unreliability to spend on release velocity and experiments.p99): line up every request by how long it took; the 99th-percentile value tells you how bad it is for the users having the worst time.This chapter opens Part II, "Principles." Ch1 introduced the phrase "error budget"; this chapter is where it gets measured, priced and given rules of use. Ch4 ("Service Level Objectives") then works out how to define SLIs, SLOs and SLAs properly, while Ch5 ("Eliminating Toil") and Ch6 ("Monitoring") are the day-to-day machinery of spending the budget. Real-world scope: any team that must answer "can we ship this," "how many nines should we target," and "is one more round of redundancy worth paying for."
The industry default is almost reflexive: more reliability is always better. This chapter dismantles exactly that, and does it without hedging — in the book's own words, past a certain point, increasing reliability is worse for a service (and its users) rather than better.
First cost: it isn't linear. Experience says an incremental improvement in reliability may cost 100× more than the previous increment. That money goes to two places: redundant compute resources — the extra machines, the extra replicas, the headroom that lets you take systems offline for maintenance; and opportunity cost — engineers assigned to hardening are engineers not shipping user-visible features. The second is the deadly one, because it never appears on any invoice.
Second cost: users can't perceive it. Between you and your user sit a phone, Wi-Fi, home broadband and a carrier backbone, all far less reliable than your service. The book is direct about it: a user on a 99% reliable smartphone cannot tell the difference between 99.99% and 99.999% service reliability. Money spent in the fifth decimal place disappears into the user's background noise.
Third cost: with no shared yardstick, reliability is settled by argument. Product development is measured on how much shipped; SRE is measured on nothing breaking. Both sides execute their own KPI and get pulled in opposite directions by the same fact. What's missing isn't goodwill — it's a referee quantity both sides accept. Leave it unsolved and you get one of two failures: over-hardening, where the money burns, the features don't ship, and not one user thanks you; or a table-thumping decision every release, won by whoever outranks whom.
Put reliability on the x-axis and its cost on the y-axis and you don't get a straight line — you get a curve that steepens as you move right. Going from 99% to 99.9% may mean one more replica and a health check. Going from 99.99% to 99.999% typically means multi-region active-active, sub-second automatic failover, dedicated capacity headroom and a team standing guard year-round — in time terms, squeezing the year's permitted unavailability from about 52 minutes down to about 5 minutes. And across the right-hand stretch runs a user-side floor: perceived reliability ≈ your service × their device × their network, so however fine you polish your own term, the difference drowns in the jitter of the other two. Cost explodes, perception goes to zero — where those two lines cross is this chapter's central judgment: reliability has a point past which you lose. SRE's job is redefined accordingly: explicitly align the risk a service takes with the risk the business is willing to bear.
A budget needs a measurement. The intuitive one is time-based availability: availability = uptime ÷ (uptime + downtime) — under which 99.99% is roughly 52 minutes of unavailability a year. That works in a single-datacenter, alive-or-dead world. But Google states plainly that it is usually not meaningful for them: services are globally distributed and fault-isolated, so at any moment they are almost always serving at least some subset of traffic somewhere in the world — never fully down, and therefore never yielding a clean "downtime duration." The real failure shape is "this region degraded, that class of request failed."
So the chapter changes the unit: define availability in terms of request success rate, i.e. aggregate availability = successful requests ÷ total requests. The book's worked example is concrete: a system serving 2.5M requests a day against a daily availability target of 99.99% can serve up to 250 errors and still hit its target. The budget becomes an integer you can check today and watch burn down. The same unit works offline — a batch pipeline computes it as successfully processed records ÷ total records — and targets are typically set quarterly but tracked weekly or even daily, so drift is caught while there's still time to react.
How many nines is not an SRE's guess; it comes from working with product owners to turn business goals into objectives you can engineer to. The chapter gives four concrete questions.
(a) What level of availability is required? Unpack it: what do users expect? Does this tie directly to revenue — yours or your customers'? Free or paid? What do competitors offer? Consumer or enterprise? The answers diverge sharply. The book uses Google Apps for Work (the enterprise productivity suite) as its example: it carries an externally committed, financially backed SLA, with internal targets set higher still than the external promise. Whereas when Google acquired YouTube, it deliberately set an availability target lower than its enterprise products — YouTube in 2006 was a fast-moving consumer product, and rapid feature development was worth more than the extra reliability.
(b) Do different types of failures hurt equally? Not remotely. The same "5,000 failed requests a year" spread evenly across days goes largely unnoticed; concentrated into one full-site outage it makes the news. The shape of failure matters as much as the total.
(c) Use cost to locate the service on the risk continuum. This is the chapter's most usable move: convert nines into money. The worked example is crisp — a service with $1M in revenue currently at 99.9%; raising the target to 99.99% adds 0.09% of uptime, worth $1M × 0.0009 = $900. The criterion becomes unarguable: if that extra nine costs less than $900 to build, do it; more, don't. The number isn't the point — the move is, because it compresses a debate about "being more responsible" into a comparison both sides can check. Internal services with no direct revenue translate the same way: convert downtime into blocked engineer-hours times their cost.
(d) Are there other metrics that matter more? Availability often isn't the important one. The book cites AdSense — ads served onto third-party pages. They load asynchronously, so a missed impression is bounded, but slowing down the host page is real damage. What it truly defends is latency, and its latency target is set relative to how fast pages themselves load in a given region. Pick the wrong metric and all the budget machinery downstream is managing the wrong thing.
Infrastructure services — storage, messaging, compute, the internal substrate — face a problem consumer services don't: by definition they have many clients, whose needs conflict. The book's example is Bigtable, Google's distributed table store. One class of client is serving user requests in real time and cares about latency: return fast or return an error. Another is running offline analysis and cares only about how much data moves per unit time. One SLO for both loses twice: build every cluster to the low-latency standard and the offline crowd pays for something it can't use; build to the throughput standard and the online crowd's user experience collapses.
The answer is not a compromise value but tiered supply, priced openly, chosen by the client: a set of low-latency, low-utilization, higher-unit-cost clusters and a set of high-throughput, high-utilization, lower-unit-cost clusters, each with its own SLO and its own internal price. The elegance is in binding choice to cost — when "more reliable" carries a price tag, internal customers state honestly how reliable they need to be. In a world where it's all free, everyone claims they need the top tier.
The first four sections set the risk; this one spends it. The chapter first lays the structural conflict between product development and SRE on the table — it concentrates in four things, none of which has an objective answer: how fault-tolerant to make the software (too little and it's brittle and unusable; too much and it's very stable but nobody wants it), how much to test (too little means embarrassing outages; too much and you lose the market), how frequently to push (every push is risk — how much effort should go into reducing it), and how long and how large to canary (a longer window is safer and slower).
Four questions, two entrenched positions, no persuasion possible. The error budget doesn't answer them; it gives them a shared unit of account. The construction is minimal: product management sets the SLO; actual availability is measured by a neutral third party — the monitoring system; the difference is the quarter's remaining budget of unreliability. And one rule: as long as budget remains, new releases can be pushed.
What it really changes is the incentive structure. First, it also puts a price on setting the target too high: a team that keeps failing to ship may choose to loosen the SLO to buy back room to innovate — the budget runs both ways, it isn't only a stick. Second, acts of nature count too: a datacenter fault or network blip eats the same budget. That looks unfair, and precisely because of it both sides are motivated to reduce them rather than blame force majeure. Third, the nature of outages changes: they stop being a "bad thing" and become an expected cost of innovation that both teams manage together — which is what the chapter title actually means.
Table 1 · Two availability metrics: when to use which
| Time-based availability | Request success rate (aggregate) | |
|---|---|---|
| Formula | uptime ÷ (uptime + downtime) | successful requests ÷ total requests |
| Fits | Single datacenter, single instance, alive-or-dead failures | Globally distributed, fault-isolated services; batch pipelines (by record) |
| Captures partial failure? | No — while any traffic is being served, there is no downtime to measure | Yes — partial failures land honestly in numerator and denominator |
| Actionability | Weak: "30 minutes left this year" barely guides today's decision | Strong: 2.5M × 99.99% → 250 errors allowed today |
| Cost | Matches SLA contract language, easy to explain externally | Requests aren't equal (heartbeat ≠ checkout); a known approximation |
Table 2 · How many nines, derived from service type — with the cost of overshooting
| Service type | Typical order | Why that order | Cost of aiming higher |
|---|---|---|---|
| Internal tools / offline analysis | 99% (≈3.65 days/yr) | Users are colleagues and can wait; throughput beats latency | Near-pure waste — nobody benefits from the extra redundancy |
| Fast-moving consumer product | Deliberately below enterprise | YouTube case: velocity is worth more right now than extra nines | The stability bought doesn't offset the product opportunity lost |
| Enterprise, SLA-backed | 99.9%+ (≈8.8 hrs/yr) | Contractual, with payouts; internal target must sit above the external promise | On-call, redundancy and real monitoring — a cost step change |
| Revenue-carrying transaction path | 99.99% (≈52 min/yr) | Downtime converts directly into money; set it with the $900 arithmetic | Multi-region failover, second-scale detection, automated rollback |
| Substrate many services depend on | Tiered, not a single value | Bigtable case: low-latency and high-throughput tiers built and billed separately | One size fits nobody: half overpay, half get a broken experience |
| Almost any service | 99.999% is usually not worth it | The user's own path is less reliable; the increment sinks into background noise | The next increment may cost 100× the last |
Table 3 · Four reactions when the budget runs out: which are policy, which is decoration
| Reaction | What it means | Consequence |
|---|---|---|
| Freeze releases | Stop features; ship only emergency and security fixes until back above the SLO | The policy works — this is the chapter's default path |
| Loosen the SLO | Renegotiate with product: this service genuinely needn't be this stable | Legal and encouraged — provided the target changes explicitly and on the record |
| Invest more | Shift engineering toward reliability and slow the burn rate | Legal, but own the bill: feature output drops for that period |
| Waive it, case by case | "But this feature is really important" — ship anyway | Decoration — once waivers become routine the budget is just a number |
This chapter teaches you to set a target for your own service, but whether you can hit it depends on the layer beneath you. Google filled that in later, in "The Calculus of Service Availability": you're only as available as the sum of your dependencies. The mechanics are plain multiplication — five critical dependencies at 99.99% each already cap you at about 99.95% (0.9999⁵ ≈ 0.9995), before you've written a single bug of your own. The resulting rule is hard-edged: a critical dependency's target must be one nine better than yours — counting dependencies before setting a target is far cheaper than adding redundancy afterwards.
The error budget has gone from a Google-internal mechanism to industry furniture: mainstream observability platforms ship a burn-rate panel by default, cloud vendors sell SLOs as managed products, and release systems read the remaining balance to decide whether to proceed. The deeper effect is on how people think — "our goal is 100% availability" is now a sentence that gets challenged in any halfway-serious design review with "on what basis, and have you costed it?" In interviews and architecture reviews the chapter maps to a handful of recurring questions: what is your SLO and how was it derived? Do you measure availability by time or by request success, and why? What actually happens when the budget is gone? How many nines does each critical dependency carry, and does the product still clear your target? People who can't answer the last one usually haven't absorbed this chapter.
99.99% daily target can serve up to 250 errors and still hit it; and the cost/benefit calculation that prices one extra nine at $900. Site Reliability Engineering, Ch3 Embracing Risk (Google, full text) ↗99.99% rather than 100%, and a critical dependency needs one more nine than your own target. The Calculus of Service Availability, ACM Queue 2017 (Google Research) ↗ · official PDF ↗99.9th percentile rather than the mean — the paper describes an SLA of the form "respond within 300ms for 99.9% of requests at a peak load of 500 requests per second," because mean-based targets systematically ignore the users having the worst time. Dynamo: Amazon's Highly Available Key-value Store, SOSP 2007 (Amazon CTO's site, PDF) ↗99.9% and 99% differ by a handful of errors and are statistically hard to tell apart), and many critical dependencies are external SaaS whose true availability you can neither control nor see. This is why the later SRE Workbook devotes a chapter to implementing SLOs outside Google.① The core claim: past a certain point, increasing reliability makes a service worse. SRE's job is to align the risk the service takes with the risk the business will bear — not to chase never-breaking.
② Cost is non-linear: the next increment may cost 100× the last, split between redundant resources and opportunity cost (the features never built), the latter appearing on no invoice.
③ Perception has a floor: phones, Wi-Fi and broadband are less reliable than you are — a user on a 99% reliable smartphone can't tell 99.99% from 99.999%.
④ Change the unit: time-based availability is near-meaningless for globally distributed services (they're never fully down), so use aggregate availability = successful requests ÷ total requests. The watchable number looks like this: 2.5M requests/day × 99.99% → 250 errors allowed today.
⑤ Four questions for consumer services: what availability level / do failure shapes hurt equally / pin it down with cost / is another metric more critical (AdSense really defends latency).
⑥ The cost arithmetic is the most usable tool: $1M revenue, 99.9% → 99.99% adds 0.09% uptime worth $900 — build it below that price, skip it above.
⑦ Infrastructure must be tiered: Bigtable's low-latency and high-throughput clients conflict, so build separate tiers, attach different SLOs, bill differently — when it's free, everyone demands the top tier.
⑧ The error budget: product sets the SLO, monitoring is the neutral referee, the gap is the allowance; balance left means ship, balance gone means freeze. It also prices over-ambitious targets (you may loosen the SLO), and acts of nature count, so both sides work to reduce them.
⑨ Don't forget what's underneath: you can't be more available than the sum of your dependencies — five at 99.99% already cap you near 99.95%; the rule is that a critical dependency needs one nine more than your target.