BOOKS DEEP-READ · SRE · CH 10
Site Reliability Engineering · Ch 10 · Google · 2016
The reason you can order something at 2am and it just works is that someone is on call — phone on the nightstand, ready to get up and fix things. This chapter isn't about the fixing. It's about when that phone should ring at all: how Google turns the numbers scattered across hundreds of thousands of machines into a single judgement, "somebody has to act, now."
You probably assume the hard part of monitoring is noticing problems. In practice the hard part is not crying wolf. An alarm that has gone off a hundred times and meant nothing ninety-nine of them has already taught the on-call engineer to ignore it — that's not laziness, that's how people respond to false alarms. So the real question of this chapter is: how do you make the phone ring less, without missing the real thing?
Early monitoring worked like bolting a smoke detector onto every machine: ask each one in turn "is your CPU high? is your disk full?" and let whoever crosses a line shout. That's fine for a few dozen machines and awful past a few thousand. A routine rolling restart sets off hundreds of detectors at once while no user notices anything; meanwhile the failures that actually hurt often look fine machine by machine — one percent of requests failing on each box adds up to a great many people staring at an error page. Worse, a detector like that can only say "over the line right now". You can't ask it what the number was last Tuesday at this hour, or how long this trend has before it hits the wall.
The idea is to flip it: machines don't judge themselves, they just report numbers.
Every program keeps a little chalkboard by its front door: how many requests I've handled in total, how many of those failed, how many people are queued up right now. The monitoring system walks the neighbourhood every ten seconds and copies the boards down. What it collects is not "good" or "bad" — it's numbers over time, like a temperature chart rather than the sentence "he's sick".
Each copied line carries a few tags: which service, which datacenter, which machine. So "should we call someone" becomes arithmetic on those lines: add up the thousands of machines belonging to one service (stop looking at them one by one), turn it into a proportion (what fraction of requests failed, so the rule works whether you run three machines or three thousand), and finally require it to stay over the line for five minutes — anything that twitches once and heals itself was never worth a phone call.
The lovely part is that one set of copied numbers serves three jobs at once: it fires the page, it draws the dashboards, and during an investigation it lets you go back and ask what yesterday looked like. And the judgement has moved from "which machine feels unwell" to "are users actually being hurt".
Copying numbers is so cheap that everyone copies everything, and a few years later there are more charts than anyone can account for — storing them, and finding the one that matters, becomes its own problem.
Good alerting isn't about picking the right threshold. It's about recording the world as curves you can do arithmetic on, and then making "should we wake someone" an expression over those curves — an expression can be reviewed, changed and reused. Ten thousand thresholds scattered across ten thousand machines cannot.
Want /varz instrumentation, labelled time series, the rate-and-ratio rule algebra and the diagrams? → switch to the deep read
The claim of this chapter: alerting should not be "a threshold per machine" but "an expression evaluated over labelled time-series data" — take a rate, aggregate across instances, turn it into a ratio, then require it to cross a line and stay there. Google's Borgmon is one implementation; its ideas reached the rest of us as Prometheus. The chapter's epigraph is a traditional SRE blessing: May the queries flow, and the pager stay silent.
This chapter opens Part III, "Practices". It follows Ch. 6 (Monitoring Distributed Systems, which settles what to monitor: the four golden signals, alert on symptoms) and Ch. 4 (SLOs, which settles where the threshold comes from), and leads into Ch. 11 (Being On-Call) and Ch. 12 (Effective Troubleshooting) — what happens after the phone rings. It is one of the few chapters that shows a specific tool, Borgmon, and its real-world counterpart is the standard stack you use today: Prometheus plus Alertmanager plus Grafana.
Everyone says "add monitoring". The hard part is that at scale the old model collapses in three places at once. The old model is host-based checking, à la Nagios: run a script against each machine, read its exit code as good-or-bad, shout when it crosses a line.
① Threshold explosion. 1,000 machines × 20 checks each = 20,000 independent thresholds. One rolling restart turns hundreds of them red while no user notices a thing — the textbook source of noise.
② It doesn't aggregate. One percent of requests failing on each machine is inside every per-machine threshold, yet across 1,000 machines it is a thousand failed requests a second. Conversely a machine can be pegged at 100% CPU with its traffic already drained by the load balancer, harming nobody. "Which machine feels unwell" and "are users hurt" are different questions, and host-based checks can only express the first.
③ No history, only now. A check script yields good-or-bad, not a number. So the three questions you actually ask during an incident are unanswerable: what was this last Tuesday at this hour? how long until this slope hits the wall? when exactly did it start bending?
The consequence isn't "we noticed late". It's that the whole alerting system loses credibility: noise buries the real signal and the on-call learns to look away. A Google Cloud team later put it bluntly: "If everything is important, nothing is important."
The first foundation is white-box instrumentation. Every Google service binary carries an HTTP endpoint, /varz, that dumps a plain-text list: one variable name and its current value per line. Adding a metric is roughly a one-line change for the developer — the cost of exporting a variable is pushed to nearly zero, which is the precondition for everything that follows and the most underrated design decision in the chapter; a mature binary exporting hundreds or thousands of variables is normal.
What matters is that the program has no idea the monitoring system exists. It only lays its internal state out by the door; who copies it, how it's computed and what alerts fall out is decided elsewhere. That is the real benefit of pulling — not saving a few packets, but fully decoupling "what we collect" from "how we judge": you can change the alerting criteria without touching production code.
Every scraped value is stored as a time series identified by a variable name plus a set of labels: job (which service), instance (which task), datacenter (where). Borgmon gets its target list from service discovery and scrapes each one, so scaling up or down needs no config change; on each scrape it also synthesises a few time series of its own (did this scrape succeed, how long did it take), so "collection itself is broken" can be alerted on by the same rules — an easily missed loop.
The data lives in a fixed-size in-memory arena: a ring buffer per series where new points evict the oldest, checkpointed to disk periodically. Why dare keep it all in RAM? Do the arithmetic: at a 10-second interval, 12 hours is 4,320 points; at a couple of dozen bytes per point that's about 100 KB per series, so a hundred thousand series is on the order of 10 GB. What you buy is re-evaluating every rule every few tens of seconds — alerting latency is purchased with memory.
Labels look mundane and are the watershed of the chapter: they make one rule cover an entire service possible. You no longer write "the error rate of machine A" but "the summed error rate of every instance where job equals checkout" — go from 3 instances to 3,000 and the rule doesn't change a character.
Borgmon rules are arithmetic expressions over vectors (all series matching a selector), evaluated periodically, with results written back into the arena as new series. They come in two kinds: computation rules turn raw data into meaningful quantities (reused by dashboards and later rules), and alerting rules add a threshold on top.
Step one: take the rate of a counter. The cumulative value means nothing on its own ("8.4 billion requests served" says nothing about now); what means something is its slope, rate(), i.e. per second. Here the chapter states a counter-intuitive preference: use a counter rather than a gauge whenever you can. A gauge (current queue length) captures one instant per sample, and any spike between two samples is lost forever; a counter accumulates everything that happened in the interval, so even if you miss a scrape, the rate computed at the next one is still correct. Counters are immune to missed samples, gauges are not — worth memorising. (The price: a counter resets to zero when a task restarts, and the rate implementation must recognise that, or it will produce an absurd negative number.)
Steps two and three: aggregate across instances, then make it a ratio. Sum the error rate and the request rate across every instance of a job and divide, giving a dimensionless error fraction. Why insist on a ratio? Because it is independent of traffic: errors / requests > 0.01 holds equally at 300 QPS overnight and 30,000 QPS at peak, whereas an absolute "more than 50 failures per second" is guaranteed to either cry wolf by day or stay silent at night. Latency is the same story — export bucketed durations and compute p99 rather than a mean, or a few very slow requests get averaged away by many fast ones.
Step four is duration. A condition holding for one instant is not a reason to call: a network blip or a GC pause can push a ratio over the line. So alerting rules carry a minimum duration (in Prometheus, for: 5m): the condition must hold continuously before it fires. This is the chapter's most practical little mechanism, and an explicit bill: a 5-minute window buys quiet at the cost of 5 minutes of detection delay, and how that bill should be paid depends on your error budget — a 99.9% monthly SLO allows only about 43 minutes, so 5 of them is a ninth of the month's allowance.
Once fired, an alert does not become a phone call directly; it goes to the Alertmanager: deduplication (one problem firing on dozens of instances sends one notification), grouping (related alerts in one envelope), inhibition (an upstream alert suppresses the downstream cascade — when a whole datacenter is off the network, you should not get one page per service inside it), silences (shut up during planned maintenance), and finally routing: page only when somebody must act this minute, otherwise file a ticket. This layer is the most overlooked, yet it is where "number of rule firings" gets compressed into "number of human interruptions".
One Borgmon cannot scrape the planet. The solution is elegant: Borgmon exports its own time series in the same format, so an upper-tier Borgmon can treat a lower one as just another scrape target. Each tier keeps only aggregated results, so volume narrows as you go up — cluster-level Borgmon holds the detail, the global one only watches cross-cluster totals. This is exactly Prometheus federation today.
One more necessary complement: white-box monitoring cannot see "users can't reach us at all". Broken DNS, a misconfigured load balancer, an expired certificate — internal metrics look perfect. So Google also runs a black-box prober that issues requests like a user and exports its results as time series, judged by the same rules. White-box tells you why it's broken; black-box tells you whether users are actually broken. Neither substitutes for the other.
The soul of this chapter isn't "which tool" but which criterion you use to declare trouble. The four are not mutually exclusive — real systems run two or three at once — but they cost different things.
| Criterion | How it decides | Strength | Cost / what it misses |
|---|---|---|---|
| Per-machine threshold | one check per host, shout when over | direct, points at a specific machine; fine for hardware faults | king of noise at scale (1,000 hosts × 20 checks); cannot express user impact |
| Aggregated ratio (this chapter) | sum across instances, take a ratio, over the line for N minutes | scale-independent; one rule per service; aimed straight at symptoms | needs instrumentation and a TSDB first; hides a single sick instance (keep a per-instance rule as backstop) |
| Error-budget burn rate | how fast the current badness would burn the monthly budget | alert density tracks business tolerance; lowest noise | requires mature SLOs; slow burns are caught late, so you stack windows (fast 1h / slow 6h) |
| Black-box probe | request from outside, like a user | catches DNS, load balancer and certificate failures that metrics can't see | sparse sampling, covers only the probed path; can't tell you why |
The second table is a choice you face at almost every instrumentation point — and get wrong almost as often.
| Counter | Gauge | |
|---|---|---|
| Meaning | a monotonically increasing total | a current value that rises and falls |
| How to use it | take the rate, then aggregate | read the value, or its max / percentiles |
| Missed sample | no distortion; the next rate is still right | that interval's spike is gone forever |
| Classic trap | resets to zero on task restart; the implementation must detect it | taking a rate of a gauge, producing a meaningless number |
| Good for | requests, errors, bytes, summed latency | queue depth, memory in use, open connections, temperature |
One decision rule the chapter never formalises but you can use as is: an alert is worth existing only if, when it fires, some person must do something at that moment. If the answer is "it can wait until morning", make it a ticket; if it's "a script could handle this", go write the script.
The weight of this chapter is that the monitoring stack you use today is its direct descendant. Prometheus maps onto Borgmon almost one to one: /metrics for /varz, scraping for scraping, labelled series for labelled series, PromQL's rate() and label aggregation for Borgmon's rule algebra, Alertmanager for Alertmanager. Learning this chapter gives you the shared mental model behind Prometheus, Grafana, Datadog and every cloud provider's monitoring.
What came next is public too: Borgmon runs one deployment per cluster, so configuration and operational burden grow linearly with cluster count. In 2020 Google published the Monarch paper at VLDB, describing the planet-scale, multi-tenant in-memory time-series database that replaced it — the method was right, the shape keeps changing.
① In one line: alerting is not a threshold per machine, it is an expression evaluated over labelled time series. Borgmon is the implementation; Prometheus is its public heir.
② Host-based checking collapses in three places: threshold explosion, no aggregation (can't answer "are users hurt"), and no history — only now.
③ The foundation is white-box instrumentation: programs lay their counters out on /varz at near-zero cost, unaware monitoring exists, which decouples "what we collect" from "how we judge".
④ The data model is the watershed: name plus labels identifies a series, so one rule covers a whole service and going from 3 to 3,000 instances changes nothing.
⑤ An all-in-memory arena buys re-evaluating every rule every few tens of seconds — alerting latency is bought with RAM.
⑥ Judgement is four steps of algebra: rate → aggregate across instances → ratio → over the line and staying; each step's output is a new series reusable elsewhere.
⑦ Prefer counters to gauges: counters are immune to missed samples, gauges permanently lose the spike between two scrapes.
⑧ Ratios beat absolutes, percentiles beat means: the first makes rules scale-independent, the second keeps slow requests from being averaged away.
⑨ The Alertmanager layer (dedup, grouping, inhibition, silences, page-versus-ticket routing) is where "rule firings" becomes "human interruptions"; the duration window trades detection delay for quiet.
⑩ Scale comes from Borgmon scraping Borgmon (hierarchical federation) and the user's view comes from black-box probes: white-box says why it broke, black-box says whether users are broken.