BOOK DEEP READ · System Design Interview Vol 2 · Ch 5
System Design Interview Vol 2 · Ch 5 · Alex Xu & Sahn Lam · 2022
That wall-mounted dashboard at the office, packed with squiggly lines: how hot the machines are running, whether checkout got slower. And the phone call at 2am telling you a server is unwell. This chapter is about how those lines get collected, and about the thing that decides whether to wake you up — built from scratch.
Your company's core storage system breaks. You open the monitoring dashboard to see what went wrong — and discover the monitoring system won't open either, because its data happens to live in the very storage system that just died.
That is the first iron law of monitoring: the fire alarm cannot be wired to the burning building's electricity. It has to be sturdier and more independent than everything it watches. Google's own monitoring system keeps all its data in memory purely to escape this loop — absurdly expensive, and worth it.
When one machine misbehaves, you log in and look. What about ten thousand? And these numbers never stop arriving: a hundred measurements per machine, reported every ten seconds, which adds up to a million fresh numbers every second, around the clock.
Worse, the job has a strange shape: the incoming volume is enormous and utterly relentless, while almost nobody is reading — it keeps writing when nobody is watching the dashboard. Ordinary databases are designed for roughly balanced reads and writes, and get flattened here.
1. Collecting. Either every machine keeps a little blackboard by its door with its current numbers, and a meter reader walks the route copying them down (if a door doesn't open, you've learned something about that house for free); or each machine sends a runner who drops a note at the mail room (the only option for machines hiding behind a firewall, or for jobs that don't live long enough to be visited).
2. Storing. Consecutive readings look remarkably alike: time ticks along evenly, and the values often don't budge at all. So don't transcribe each one in full — just record "how much this differs from the last", the way a ledger says "unchanged, unchanged, plus one" instead of "10032, 10032, 10033". The same data gets more than ten times slimmer.
3. Old data. Nobody, a year later, cares what the CPU was doing at 3:07pm last Wednesday. So the last few days are kept verbatim, anything past a month is squeezed into one summary per minute, and older still into one per hour — like a diary gradually abbreviated into a year-end review. The cost is real: a spike that flashed by once is gone forever after the summary smooths it away.
4. When to wake you. One data centre drops off the network and a thousand machines shout "I'm unwell" at once. Send each one and your phone rings a thousand times in thirty seconds — after which you silence it and miss the next real emergency. So before anyone gets woken, the thousand shouts about one event must be gathered into a single sentence.
A monitoring system is not "a website with charts on it". It is an assembly line running from a million devices to your phone: the first half is about swallowing a flood of numbers quickly and cheaply, the second about waking exactly one person when something is truly wrong — and not pestering them when it isn't.
Monitoring is a peculiar write-heavy, read-light system: it shrinks data by recording only changes, holds costs down by abbreviating old data into summaries, and protects humans by folding a thousand duplicate shouts into one. Plus one iron law — it must never depend on the systems it is watching.
Want the mechanisms, the numbers and the diagrams? → Switch to the deep read
You think of monitoring as a read system — it's just charts, isn't it? In fact it is one of the most relentlessly write-heavy systems a company runs: a million writes per second, forever, with reads arriving only in occasional bursts. This chapter decomposes "design a monitoring and alerting system" into a pipeline — collection, transport, storage, query, alerting, visualization — and every trade-off in it is driven by the same constraint: writes are enormous and constant, reads are sparse and bursty, and the whole thing must be more independent than everything it watches.
host=i-001, used to slice and aggregate after the fact.Chapter 5 of Volume 2, on the "internals of things you build yourself" thread — a relative of Chapter 4: that one builds Kafka, this one builds the whole Prometheus + Grafana + Alertmanager stack; Chapter 6 (ad click aggregation) later pushes the streaming pre-aggregation here much further. In the real world it maps onto any company's observability platform: Prometheus, Datadog, Google's Monarch, Uber's M3.
Quantify the brief first, because nearly every design decision here is forced by magnitude. The book's scenario: the infrastructure of a company with 100 million daily active users — roughly 1,000 server pools, 100 machines each, 100 metrics per machine, or about 10 million time series, sampled every 10 seconds. So:
10,000,000 ÷ 10s = 1,000,000 writes / second
A million data points per second, and this is the baseline, not the peak — it is the same at 3am, and must be retained for a year. The read side? A few dozen dashboard panels and an alert rule evaluation every minute: orders of magnitude smaller than the write load, but bursty (during an incident everyone digs through history at once).
What happens if you don't solve it? Brute-force it with an ordinary relational database, one row per data point, and a year is on the order of 3×10¹³ rows, with indexes larger than the data; disk and invoice collapse first. Worse is the second failure mode: the system holds up, but the alerting is garbage — one flaky data centre emits a thousand notifications, and three months later the on-call engineer dismisses every alert on reflex. This chapter has to solve both: swallowing it, and speaking like a human.
The model is almost embarrassingly plain: a metric name, a set of labels, and a run of (timestamp, value). For instance
cpu.load {host=i-001, env=prod, region=us-west}
The crucial part: every distinct combination of name and labels is its own time series. Labels are where all the power lives — you don't know in advance how you'll want to slice things during an incident, so labels let you decide at query time whether to group by region or by instance type, with no pre-built rollup tables.
The cost lives in the same place: cardinality is the product of each label's value count. 1,000 machines × 5 environments × 20 metrics = 100k series, fine; swap in a label that grows with your user base and there is no ceiling at all (see §7). Labels are for dimensions with bounded values that you will genuinely group by.
Pull: each monitored process exposes a /metrics HTTP endpoint emitting its current values as text, and collectors fetch them on a fixed interval. This is Prometheus's approach, and it throws in two things free: debugging is trivial (open the endpoint in a browser and read the raw numbers), and health checking comes along for the ride — a failed scrape means the instance is gone, no separate heartbeat needed.
It must answer one question: who should I scrape? Machines scale up and down constantly, so the roster can't be hard-coded. The answer is a service discovery component (etcd / ZooKeeper / Consul) asked periodically for the live instances; when there are more targets than one collector can handle, make the collectors a cluster and shard targets across them with consistent hashing, each owning an arc of the ring, so scaling moves only a small fraction.
Push: an agent on each machine sends data in, and can pre-aggregate locally first. It wins in the three places pull cannot reach: targets behind a firewall or NAT, short-lived jobs (a batch job that exits after 30 seconds is gone before the scraper arrives — hence Prometheus's Pushgateway), and cases sensitive to reporting delay.
Storing each data point as a row in a relational database can work in theory; in practice, at a million writes per second, it demands very deep tuning expertise. A purpose-built TSDB wins on two things, and both are exploiting the shape of monitoring data itself.
First: compression. Monitoring data hands you two regularities for free — collection is scheduled, so the gap between consecutive timestamps is essentially constant; values are slow-moving, so disk usage this second usually equals last second's. So store differences rather than values: for timestamps record "how much later than the previous one", then "how much that gap differs from the previous gap" (delta-of-delta, almost always 0 under scheduled collection — one bit will do); for values, XOR each float against its predecessor, which is all zeros when unchanged and has only a few non-zero bits when barely changed, so only those bits travel. This squeezes a 16-byte data point down to 1.37 bytes on average (source in §6).
Second: tiered retention. Data value collapses fast with age, so split it by age:
Table 1 · Retention and downsampling tiers (the book's policy; relative volumes for 10M series at a 10-second interval)
| Age | Grain | What it's for | Relative points |
|---|---|---|---|
| Last 7 days | raw, 10s | Debugging: you need the spikes and the jitter | baseline 1× |
| 7–30 days | rolled up to 1 min | Week-over-week, capacity trends | 1/6 |
| 30 days–1 year | rolled up to 1 hour | Quarterly capacity planning, cost accounting | 1/360 |
| Beyond | cold storage / archive | Compliance retention; slow queries accepted | unit price an order of magnitude lower |
The two multiply. One concrete anchor — the InfluxDB benchmark the book cites: a single node with 8 cores and 32GB of RAM sustains 250,000 writes per second. Extrapolating, absorbing that steady 1M/s takes 4 such nodes as a floor, and with replicas and headroom on top, that is your real cluster size.
Wiring collectors straight into the TSDB runs, but has one fatal property: the moment the TSDB wobbles, collectors start dropping data — and the moment monitoring data most needs to survive is precisely when things are breaking. The standard answer is a queue (Kafka) in between: collectors only write to the queue; consumers read, aggregate, and land it in the TSDB. The queue is partitioned by metric name so that one metric always reaches the same consumer, which is what makes aggregation correct; when the TSDB dies the data piles up in the queue instead of evaporating, and catches up on recovery.
The skeleton is simple: rule config → alert manager → notification queue → sender workers. A rule is declarative config expressing three things: what to query, what counts as abnormal, and how long it must stay abnormal to count. In Prometheus style, roughly
expr: up == 0 for: 5m severity: page
That for: 5m is the most underrated line in the chapter: the anomaly must persist five minutes to fire, trading a little latency for the elimination of most jitter-induced false alarms. The real engineering, though, sits in the alert manager, whose entire job is protecting people:
up == 0 together, and that should become one "100 instances unreachable on rack A".On visualization, the chapter is blunt: don't write your own — Grafana already speaks to Prometheus and InfluxDB directly. The "query service" is likewise optional: mainstream TSDBs ship query languages (PromQL, InfluxQL, Flux) that dashboards and the alert manager can use directly; that layer exists only to cache hot queries and centralise auth, and is not mandatory.
Table 2 · Direction of collection: pull vs push (the book's comparison, with each cost named)
| Dimension | Pull | Push |
|---|---|---|
| Debugging | Wins: open /metrics in a browser, read the raw numbers | You dig through agent logs to learn what was sent |
| Health check | Wins: a failed scrape means the instance is dead, free | Silence can't distinguish "dead" from "agent never installed" |
| Short-lived jobs | Loses: a 30s batch job misses the scrape window entirely | Wins: sends on exit (needs a Pushgateway to land it) |
| Firewall / NAT | Loses: the collector must reach every target | Wins: the target only needs one outbound address |
| Target roster | Requires service discovery — one more dependency | Wins: whoever shows up counts; no roster |
| Reporting delay | Bounded by the scrape interval (say 10s) | Wins: sends as it happens |
| Garbage defence | Wins: the server decides who gets scraped | Loses: any process can flood it; a test job can swamp production |
The conclusion is not "pull is better" — the Prometheus maintainers' own phrasing is the honest one (see §6). Large platforms usually keep both: long-running services get pulled, short-lived jobs and edge devices push.
Table 3 · Where to aggregate (same "average CPU across us-west", computed in three places — same answer, very different costs)
| Where | What it saves | What it costs | When to pick it |
|---|---|---|---|
| At collection (on the agent) | Least bandwidth; point count drops at the source | Only simple within-host aggregation; nothing cross-machine | Huge per-host metric counts, tight egress |
| In the pipeline (stream pre-agg) | Order-of-magnitude drop in TSDB writes, often below 1/10 | Needs stream processing (Flink et al.); raw points discarded never return; late data needs handling | Writes are already the bottleneck and the aggregation dimensions are fixed |
| At query time | Most flexible — slice it any way later; all raw data kept | Highest storage and query cost; wide queries can take tens of seconds | Volume is still manageable, or debugging needs the raw jitter |
Mature platforms mostly mix: popular, fixed aggregations are computed in the pipeline, everything else stays raw and is computed at query time — the same idea as "keep recent data verbatim, keep old data as summaries", applied a second time.
The weight of this chapter is that it describes the one system every other system depends on — and the moment it fails is exactly the moment you need it. That constraint reshapes its architecture; the second item below is the sharpest example.
FROM THE FIELD
16 bytes to 1.37 bytes on average; the paper reports this shrank the storage footprint about 10×, letting the data live entirely in memory, and versus their previous HBase-backed time-series store it cut query latency by 73× and raised query throughput 14×. This chapter's compression and "hot data in memory" descend directly from it.Pelkonen et al., Gorilla: A Fast, Scalable, In-Memory Time Series Database, VLDB 2015500 million metrics per second and persists 20 million per second to storage, each written to three replicas by quorum within a region; engineers configure per-metric retention (two days to five years) and granularity (one second to ten minutes). This is tiered retention plus pipeline pre-aggregation at real scale.Uber engineering blog, M3: Uber's Open Source, Large-scale Metrics Platform for Prometheus (2018)Putting a high-cardinality field in a label. The classic one-shot incident: add a user_id or request_id label and time series jump from millions to billions, blowing out the TSDB's in-memory index — precisely when you have lost the tool you'd use to diagnose it. The test is simple: a label's value count must not grow with your user or request count. The chapter only glances at this; it is the system's true fragility.
Assuming downsampling is lossless. It is irreversible, and the trap is subtle: percentiles don't add up — you cannot recompute a day's p99 from a pile of per-minute averages. Review an old incident six months later with only hourly averages and the three-minute spike that blew the SLO has been flattened into a straight line. Either pre-aggregate quantiles and maxima alongside when you downsample, or accept that detail vanishes past a certain date. That's a cost decision, not a technical detail.
Alerting on causes instead of symptoms. Put a threshold alert on every component and one failure floods you with hundreds, burying the thing that matters: "users can't check out". The classic Google SRE position is to alert on symptoms and leave causes to diagnosis — consistent in spirit with this chapter's focus on dedupe and grouping, though it never takes up alerting philosophy itself.
Period limitations. Written in 2022, this is a metrics-centric view; logs and traces are declared out of scope on page one. The "high-cardinality wide events" camp argues the opposite: rather than defining metrics up front, keep raw events with rich dimensions and slice at query time — because "reduce to metrics first, then downsample" discards exactly what you need to debug a novel problem. Both have costs: this chapter's route is cheap and predictable, the other is expensive but answers questions you didn't anticipate.
① It's a write system: 1000 pools × 100 machines × 100 metrics ÷ 10s = a constant million writes per second, with reads orders of magnitude smaller but bursty — that asymmetry drives every decision.
② Data model: one time series = metric name + labels. Labels buy you arbitrary slicing after the fact; the price is cardinality growing as a product — put a high-cardinality field in and you have an incident.
③ Two directions of collection: pull gives free health checks and easy debugging but needs service discovery (and consistent hashing to shard targets across collectors); push handles firewalls and short-lived jobs at the price of making silence unexplainable. Mature platforms keep both.
④ Storage wins on two tricks: delta + XOR compression (16 bytes → 1.37 bytes), then tiered downsampling at "7 days raw / 30 days per minute / 1 year per hour" — and they multiply.
⑤ Put a queue in the middle (Kafka, partitioned by metric name): absorbs bursts, decouples, and piles data up rather than dropping it when the TSDB dies — the moment monitoring data most needs to survive is the moment things break.
⑥ Aggregation has three homes (agent / pipeline / query time): earlier is cheaper, later is more flexible, and raw points dropped in the pipeline never come back.
⑦ The alert manager exists to protect humans: dedupe, group by root cause, inhibit downstream, deliver at-least-once; a duration condition like for: 5m kills most jitter false positives.
⑧ The iron law: a monitoring system must not depend on what it monitors. Google's Monarch keeps everything in memory and regionalizes precisely to break that circular dependency. Visualization and query layers go the other way — Grafana on top of a TSDB is plenty; don't build your own.