Monitoring Distributed Systems: if the response is mechanical, it should never wake a human

In one line

You'd expect a chapter on monitoring to be about collecting more. It is almost entirely about subtraction: Google explicitly gave up on monitoring that tries to learn thresholds and infer root causes, demanding instead that it be simple, fast and predictable; it elevates four golden signals — latency, traffic, errors, saturation — as the minimum sufficient set; and it puts a near-brutal bar on paging: every page must be urgent, actionable and require human judgement — and the moment its response is mechanical, it belongs to a machine, not to a sleeping engineer.

Glossary

Where it sits

Written by Rob Ewaschuk, this chapter grew out of an internal Google document of his, My Philosophy on Alerting, that circulated widely before the book existed. It closes Part II, "Principles": Chapter 3 taught you to price reliability with an error budget, Chapter 4 to write that target down as an SLO, Chapter 5 to cut toil away — and this one answers what you should actually watch, and when a human deserves to be woken. It sets up Part III's practices: the on-call chapter and the troubleshooting chapter both assume your alerts are already clean. In the real world it maps onto every team's Prometheus/Grafana dashboards and PagerDuty rotation.

The problem it solves

"More monitoring can't hurt" is one of the most expensive wrong sentences in the industry. A service can easily export ten thousand metrics, every one of them can carry a threshold alert, and each addition makes the team feel slightly safer. The actual result: the on-call phone fires a dozen times a night, and twelve of those need nobody to do anything. Three months later the on-call engineer has built the muscle memory of glance-dismiss-sleep — and the one real alert, sitting in that pile, gets dismissed too. That is what this chapter is for: not "how do I monitor more," but "how do I make every wake-up worth it."

It also guards the other flank — finding out you're down only when users complain. But the book's judgement is blunt: over-monitoring is a harder problem to solve than under-monitoring, because what you have to repair is a person's trust in the system, and that takes far longer to rebuild than to destroy. The staffing figure the book gives says something too: on a typical SRE team of ten or so, one or two people's time goes into building and maintaining monitoring.

The core, properly explained

1. Start with "why monitor" — five uses, and only one costs sleep

The chapter opens by laying the uses out: analysing long-term trends, comparing over time or between experiment groups, alerting, building dashboards, and conducting ad hoc retrospective analysis. Of those five, only alerting consumes human sleep — and that distinction is the chapter's foundation. Dashboards can be as elaborate as you like, metrics as numerous, logs as long-lived: none of it disturbs anyone. But the moment a rule is wired to a pager it starts spending money, and the currency is the on-call engineer's attention. The companion principle is one people routinely forget: monitoring should never require a human to interpret it. Software does the interpreting; people are notified only when they need to take action. A system that needs someone watching a graph to decide whether a wobble counts is already a design failure.

2. Symptoms or causes: alert on what broke, dashboard on why

The chapter splits the question monitoring answers in two: the symptom — what is broken; and the cause — why. The book's examples are refreshingly concrete: the symptom is "I'm serving HTTP 500s and 404s," the cause might be "database servers are refusing connections"; the symptom can even be "users in Antarctica aren't receiving their animated cat GIFs," with the cause being a CDN that has blacklisted a range of client IPs.

The conclusion is that alerts should only fire on symptoms: a symptom is a user actually suffering, whereas the cause is one of a hundred possibilities and you will frequently guess wrong. Causes live in dashboards and white-box metrics — the things you dig through after a symptom has summoned you. And there's an elegant observation here: in a layered system, one person's symptom is another's cause. "The database is slow" is a symptom to the DBA and a cause to the frontend that calls it. So symptom and cause aren't absolute labels — they are relative to the layer you own.

3. Black-box and white-box: one wakes people, one explains

White-box monitoring reads what the system exports from inside: counters from HTTP handlers, logs, runtime statistics. Its unique power is seeing masked problems and imminent ones — requests that succeeded only on retry, where the user felt nothing but the retry rate has gone up tenfold; a disk that isn't full yet but will be in four hours at the current rate.

Black-box monitoring probes from outside as a user would, and is symptom-oriented: what it reports is always actively happening rather than "might be about to." That makes it naturally suited to paging — when it fires, someone really is hurting right now.

Hence Google's stance: paging leans heavily on black-box monitoring plus a small number of symptom-oriented white-box rules; the bulk of white-box data goes to dashboards and debugging. Put the other way round — if most of your pages are wired to internal white-box metrics (CPU is high, a queue is long, a process restarted), you are mostly being woken to handle things no user has noticed.

4. The four golden signals (the chapter's headline)

The most-quoted passage in the book: if you can only measure four metrics of your user-facing system, measure these — latency, traffic, errors, saturation. The promise is direct: measure all four and page a human when one is problematic (or, for saturation, nearly so), and your service will be at least decently covered by monitoring.

Latency: how long it takes to serve a request. This holds the chapter's most counterintuitive rule: track the latency of successful and failed requests separately. A service that has lost its database connection returns HTTP 500 extremely fast — a few milliseconds to say "I can't." Blend them together and during the worst minutes of an outage your dashboard's average latency will improve. The book adds a line that gets quoted constantly: a slow error is even worse than a fast error — so failed-request latency deserves watching in its own right.

Traffic: a measure of how much demand is being placed on the system, in units that fit the business. For a web service it's usually HTTP requests per second, broken down by the nature of the request (static assets versus dynamic endpoints differ in cost by an order of magnitude); for an audio streaming system it's network I/O rate or concurrent sessions; for a key-value store, transactions and retrievals per second.

Errors: the rate of failing requests, in three flavours — explicit (an HTTP 500), implicit (an HTTP 200 carrying the wrong content), and by policy (if you committed to one-second responses, a request that succeeds in 1.2 seconds is by definition an error). Implicit errors are the hardest to catch — usually only end-to-end content checks find them — and they are exactly the kind that damage user trust most.

Saturation: how "full" the service is, emphasising the most constrained resource. It is the only one of the four with a predictive character, and it comes with three points: ① many systems degrade before utilisation reaches 100%, so you want a utilisation target rather than waiting for full; ② rising latency is often a leading indicator of saturation — measuring p99 latency over a small window (say one minute) can flag saturation well before the resource genuinely runs out; ③ it should also answer predictions like "at this rate your database will fill its hard drive in four hours."

5. Tails and resolution: why you bucket, and how often to sample

The chapter gives the tail its own section, because confusing "an average of 100 ms" with "99% of requests at 10 ms and 1% stuck at 5 seconds" is the classic statistical mistake — the averages can match while the user experience could not be further apart. An average is a lie told by a bimodal distribution.

The fix isn't computing a few more percentiles; it's changing how you collect. Don't compute an average server-side and report it — bucket requests by how long they took: 0–10ms, 10–30ms, 30–100ms, 100–300ms… with roughly exponentially growing boundaries. Now what you ship is a distribution: any percentile can be derived afterwards, and buckets from many machines can simply be added together. This is exactly where the histogram in today's Prometheus comes from.

Resolution is the other trade-off: sampling more densely gives a fuller signal, but the cost of collecting, transferring, storing and querying rises with it. The chapter is refreshingly concrete — for a web service targeting 99.9% annual uptime (no more than about nine hours of aggregate downtime a year), probing for a success status more than once or twice a minute is probably unnecessarily frequent; and checking hard-drive fullness more than once every one to two minutes is probably unnecessary too. Beyond that you are burning money.

But the opposite failure is just as real: a per-minute CPU number hides the truth — the multi-second spikes that drive high tail latency simply vanish in a one-minute average. The book's answer is neat: sample at high frequency inside the server, aggregate in place, report infrequently — record CPU utilisation once a second, increment a bucket at 5% granularity, and ship the whole bucketed distribution once a minute. You get per-second resolution while paying per-minute transfer and storage costs.

6. As simple as possible, no simpler

The chapter lands on a value judgement: a monitoring system should be as simple as possible, but no simpler. The pressure comes from every direction — catch ever rarer anomalies, add more signal sources, let it infer causes for you — and pile enough of that on and the monitoring system itself becomes a complex system that needs a dedicated owner and that nobody dares touch during an incident. The book's three subtraction tests are genuinely usable:

• The rules that catch real incidents most often should be as simple, predictable and reliable as possible — they're your workhorses; no cleverness.
• Data collection, aggregation and alerting configuration that is exercised less than once a quarter should be a candidate for removal — it isn't helping you, and it is quietly rotting (nobody has verified it still works).
• Signals that are collected but not exposed in any prebaked dashboard nor used by any alert are candidates for removal too.

This also explains Google's apparently unambitious choice: they deliberately avoided "magic" monitoring that tries to learn thresholds and infer causality automatically. The reason is practical — when something actually breaks, you'd first have to debug the magic system's own reasoning, and an incident gives you no time for that. Their experience is that simple, fast monitoring plus good tools for post hoc analysis beats a clever system nobody can follow.

Key trade-offs

Table 1 · Black-box vs white-box: which one, when

Table 2 · The four golden signals: what to measure, and where it goes wrong

Table 3 · Which tier should a rule output to

The chapter also supplies a checklist for "should this alert exist at all," worth pinning to a wall:

• Does this rule detect an otherwise undetected condition that is urgent, actionable and actively or imminently user-visible?
• Will I ever be able to ignore this alert knowing it's benign? Can I exclude that situation from the rule itself?
• Does it definitely indicate that users are being negatively affected — and are there cases (drained traffic, test deployments) that should be filtered out?
• What can I do in response? Must it happen now or could it wait until morning? Can the action be safely automated?
• Is someone else already being paged for this? If so, at least one of the pages is redundant.

The accompanying sentiment is equally blunt: pages should be urgent, important, actionable and intelligent, and each one should ideally concern a novel problem rather than the fortieth recurrence of an old one. Err on the side of deleting a noisy alert — over-monitoring is a harder problem to solve than under-monitoring.

Why it matters / in real systems

This is probably the most externally influential chapter in the whole book: the four golden signals have become the industry's shared vocabulary for monitoring. Open a Grafana service dashboard template, Kubernetes' default monitoring, the metrics a service mesh like Istio generates for free, or a Datadog / New Relic service overview, and the skeleton is some variant of these four; the histogram being a first-class citizen in Prometheus descends directly from this chapter's "ship distributions, not averages." It has also grown two widely used dialects: RED (Rate, Errors, Duration) is the four signals trimmed for request-driven services, dropping saturation; USE (Utilisation, Saturation, Errors) is the resource-side dual. The common pairing in practice is RED for services, USE for machines — which together cover the four golden signals completely.

In interviews this chapter is the source of the standard answers on monitoring and observability: should alerts fire on symptoms or causes? (symptoms — causes are one of a hundred and you'll guess wrong) why isn't average latency enough? (an average is a lie told by a bimodal distribution; look at the distribution and the tail) why separate success and failure latency? (failures return fast, so blending them makes an outage look like a performance win). Answer those three cleanly and you've read the chapter properly.

Pitfalls and disputes

• Reading "four golden signals" as "four metrics are all you need": it is a minimum sufficient set, not a complete one. The four signals exist to wake people; troubleshooting still needs plenty of white-box data. Cut down to four and you'll be blind during an incident.
• Black-box only: it reports only what is broken right now — it cannot see failures masked by retries and cannot predict what is about to break. An entire class of "this is about to go wrong" signals disappears.
• Muddled latency accounting: blending failed requests into latency manufactures a fake "latency improved" during a major outage. For the same reason, never average the p99 across machines — percentiles are not additive; add the per-machine bucket counts first, then compute the percentile. (Which is precisely the benefit of shipping distributions rather than single values.)
• "Automate the mechanical response" has its own cost: automating a band-aid genuinely frees the on-call engineer, but it can also ensure the underlying cause never reaches the top of anyone's list — exactly the Gmail Workqueue argument. The chapter records it honestly as a tension between short-run and long-run availability and does not pretend there is one answer.
• Its era shows (the most important caveat): written in 2016, in the Borgmon age, the chapter is mostly about pre-aggregated metrics plus static thresholds. Two things have changed since. Google itself replaced most static thresholds with SLO burn-rate alerting (above). And the observability camp — built on distributed tracing and high-cardinality events, with OpenTelemetry as the de facto standard — offers the strongest counterargument to this chapter: pre-aggregated metrics can only answer questions you thought to ask in advance, whereas modern microservice failures are mostly "unknown unknowns" that demand raw events you can slice arbitrarily after the fact. Cross-service attribution, a whole dimension this chapter barely touches, is exactly the gap tracing fills.
• In fairness, though: the chapter's values have not aged. "Alert on symptoms, not causes," "every page must earn itself," and "the monitoring system must be simple enough to trust mid-incident" hold just as well across metrics, logs and traces. The tooling changed; the tests didn't.

Takeaways