CHAPTER DEEPREAD · SRE · CH 25
Site Reliability Engineering · Ch 25 · Google · 2016
The "recommended for you" row on your phone, your end-of-month statement, the ordering of your search results — none of that is computed the instant you open the app. Somewhere, overnight, a production line is grinding mountains of raw records into those finished goods. Chapter 25 of Google's SRE book is about why that line keeps breaking, and what to do about it.
Picture a central kitchen. At three in the morning an alarm goes off, everyone clocks in at once, and the day's ingredients get washed, chopped, cooked and boxed before dawn. Sounds orderly. Then it grows, and every kind of trouble arrives.
One pot holds up the whole floor. The work is split between a hundred cooks. Ninety-nine finish in ten minutes; the last one was handed an entire cow and needs three hours. Because the next step can't start until everybody is done, the whole line just waits — and hiring another hundred cooks changes nothing. The problem isn't too few hands, it's work that was divided unevenly.
The storeroom gets mobbed at the bell. The alarm rings and a thousand cooks charge the storeroom at the same instant, jamming the door. They can't get in — and neither can the restaurant next door, which had nothing to do with any of this.
Occasional collisions. There's a second line upstairs. One starts every three hours, the other every four. Mostly they miss each other, but every twelve hours they land on the same moment, and that's the time something breaks. Afterwards you investigate and each line, examined on its own, looks perfectly fine.
You can't tell alive from stuck. From across the room the kitchen lights are on and figures are moving. You genuinely cannot tell whether they are cooking or standing around. By the time you find out, it's usually the customers who noticed first that no food arrived.
Change the arrangement: the line never opens and closes — it simply stays on. Work arrives, someone picks it up. In the middle sits a dispatch desk that records who has which job and how far along it is. The peaks flatten out, the storeroom stops getting mobbed, and for the first time you can see exactly which step a job is stuck on.
One detail is especially neat. When a cook takes a job, they also take a numbered ticket. Suppose they drop out of contact for half a minute; the dispatch desk assumes they're gone and hands that dish to someone else. When the first cook comes back and tries to hand in the plate, their ticket is void and the desk refuses it. So the same dish never reaches a table twice. The desk itself is duplicated across several sites, too — one site loses power, another keeps dispatching.
The honest cost: "always on plus numbered tickets" is far more machinery than "an alarm clock and a script." If your job is modest and running it once a day is genuinely enough, the old way is less trouble.
The enemy of a data pipeline is usually not the volume of data but the act of clocking in on a schedule — which inevitably produces idle troughs, crushing peaks, and a middle where you can't tell alive from stuck. Past a certain size, leave the line running, and give every job a numbered ticket.
Want the mechanisms, the diagrams and the trade-off tables? → Switch to the deep read
You think the enemy of a pipeline is data volume; it is really the periodic schedule itself. This chapter argues that the periodic pipeline — a batch job fired by a timer — is simple and effective at small scale, but once the data grows, the chain deepens, and it all has to run on a shared cluster, it rots in four predictable directions: a hanging chunk, a thundering herd at start-up, moiré spikes where independent schedules collide, and a running job you cannot tell apart from a stuck one. Google's answer was not to tune the batch job harder but to change the paradigm: turn the pipeline into an always-on task system with real correctness guarantees.
Written by Dan Dennison, this chapter belongs to the data-reliability cluster in Part III (Practices). It follows Ch 24, "Distributed Periodic Scheduling with Cron" — that chapter is about getting cron itself right in a distributed world; this one makes the more uncomfortable point that even a correct cron does not save the periodic paradigm at scale. It leads into Ch 26, "Data Integrity." Real-world counterparts: MapReduce / Hadoop / Spark batch jobs, DAG schedulers like Airflow, Luigi and Argo, and continuous engines like Flink, Beam and Dataflow.
The classic pipeline pattern is almost a single sentence: read it in, transform it, write it out — ETL. MapReduce gave that pattern horizontal scale, and so it became the default: write a batch job, attach a cron entry, run it once a night.
An anchor to make this concrete: a pipeline computing recommender features, run once a day over roughly 10 TB of logs, chained across 8 stages, on a cluster shared with online serving. While the data is small it is flawless. But once scale, depth and shared tenancy arrive together, the bill comes due: delivery slides from "definitely done by 4 a.m." to "no promises," and every downstream report and model retrain drifts with it; and the pipeline's start-up peak punches through shared storage, taking out online services that have nothing to do with it.
The cost of not fixing this isn't "a bit slower." It is that you can no longer make any promise about the freshness of the data — and promises (SLOs) are what the first four chapters of this book are entirely about.
A single-stage pipeline (read → transform → write) is stable. But real work needs steps: clean → join → aggregate → sort → score → export. So it becomes a chained multi-stage pipeline, and between every pair of stages sits a barrier: every shard of the previous stage must finish before the next stage may begin.
That is the root of the fragility. Depth 8 means eight rounds of lockstep; one slow shard anywhere on the chain stalls the whole thing — total runtime is the sum of "the slowest shard in each stage," and has nothing to do with the average. "Watch the tail, not the mean" is promoted here from a measurement habit to an architectural constraint: the deeper the chain, the more times the tail gets to bite you.
A stage's work is cut into chunks and handed to workers, ideally in equal sizes. Reality is data skew: shard by user ID and one whale occupies an entire chunk; shard by country and the US chunk is a thousand times the Iceland chunk.
The result is what the chapter calls a hanging chunk: seven shards finish in a dozen minutes and the eighth runs for three hours, while nearly every worker idles and the barrier waits on that one. The most common misdiagnosis is "add machines" — skew is a problem of data distribution, not capacity; double the fleet and that chunk still takes three hours. Only three things work: re-sharding (make the split match the distribution — salt hot keys, sub-split fat chunks), redundant execution (run a second copy of a lagging chunk and take whichever finishes first — precisely the backup tasks of the MapReduce paper), and changing the paradigm so lockstep is no longer required.
A periodic pipeline's resource curve is a square wave: the timer fires, thousands of workers start at once, read the same input at once, and hit the same storage service at once. The instantaneous peak can be tens of times the mean, while a shared cluster is usually provisioned for the mean.
What follows is a self-inflicted denial of service — the chapter's thundering herd. The pipeline punches through shared storage or the network, hurting itself first (timeouts, retries, cascading failure) and then hurting unrelated services on the same cluster. Worse, it recurs on a schedule: the capacity you tuned yesterday blows up again today when the data grows 20%. The basic countermeasures are start-up jitter and rate-limited ramp-up — don't let every worker through the door in the same second — but be clear that this only shaves the peak. The square wave is still there.
This one is nastier. Yours is not the only pipeline on the shared cluster. Pipeline A runs every 3 hours, pipeline B every 4; each is comfortably within capacity and each is healthy. But every 12 hours they land on the same instant, and nobody ever provisioned for that sum.
The chapter calls this the moiré load pattern, after the third pattern that surfaces when two grids overlap: invisible most of the time, explosive on contact, and long-period, hard to reproduce, hard to attribute. You go through the logs and find "that inexplicable timeout at 03:00 last Tuesday," then another two weeks later, with everything normal in between. The genuinely frightening part is that it is nobody's fault in particular: each pipeline, examined alone, is perfectly healthy. This is the archetypal failure visible only at cluster level — and the hidden price of putting pipelines on shared infrastructure.
Halfway through a periodic run, what signals do you actually hold? The process is alive; the CPU is busy. None of that distinguishes "working" from "stuck on a hanging chunk." By the time you notice, it is usually a downstream consumer that noticed the data never arrived.
The chapter's verdict is that the periodic paradigm is inherently hard to monitor, because every run is a one-off with no steady state and therefore no "normal rate" to compare against. Doing it well means manufacturing your own signals: the slope of per-stage completion over time, this run's deviation from the same point in previous runs, and above all data freshness — how far the output lags the present moment. That last one is what users actually care about, and the right thing to write an SLO against.
Google's answer was an internal system called Workflow (never open-sourced), whose core is one sentence: stop clocking in on a schedule. The pipeline runs continuously; work arrives and a worker picks it up. The chapter explains its shape by analogy to MVC:
Staged execution survives (each stage's output is the next stage's input), but it becomes rows of task state inside the Task Master rather than the start and end boundaries of a batch job. The peaks and troughs flatten, and latency moves from "wait for the next scheduled run" to very nearly continuous.
Running continuously forces one question: what if a worker only looks dead? It may just have been network-stalled for 30 seconds. If the Task Master has already reassigned its work and the worker then revives and submits, the same output gets written twice — classic split-brain.
Workflow's answer is collaborative locking with unique tokens: every unit of work is handed out with a unique sequence token; a worker must present that token when it submits, and the Task Master checks whether it is still the currently valid one. A zombie worker whose work was reassigned holds a stale token, and its submission is refused outright. All output must be committed through the Task Master — there is no side channel.
The spirit is exactly a database's optimistic lock: you don't have to prevent the failure, only guarantee that stale writes cannot land. Combined with each stage's output having a unique owner inside the Task Master, the pipeline produces exactly one correct result even as workers crash, restart, or hang.
Having become the single truth, the Task Master has also become the single point of failure. That is where the chapter ends: use distributed consensus to replicate the Task Master across clusters, elect one leader with the rest on standby, and fail over automatically when a whole cluster goes. The practical advice given is plain — keep a copy in at least two clusters, plus a third party as tiebreaker so two replicas can't deadlock on a tied vote. It is a direct application of Ch 23, "Managing Critical State": hand critical state to a real consensus implementation instead of improvising one out of heartbeats and timeouts.
Table 1 · Periodic batch vs continuous processing: what the paradigm swap actually buys
| Periodic pipeline (timer + batch job) | Continuous pipeline (Workflow / streaming) | |
|---|---|---|
| Output latency | One period at best (hours to a day) | Near-continuous (seconds to minutes) |
| Resource curve | Square wave: idle troughs, herd peaks | Flat, high utilization |
| Capacity planning | Must buy for the peak, idle the rest of the time | Buy for the mean |
| Monitorability | Poor: one-off runs, no steady-state baseline | Good: continuous rate and backlog metrics |
| Cost of correctness | Low: on error, just rerun the batch | High: needs tokens, idempotency, barriers |
| Operational burden | Low: one timer, one job | High: a standing service, state and leader election to keep alive |
| Fits when | Volume is steady, chain is shallow, daily is enough | Volume is large, chain is deep, freshness carries an SLO |
Table 2 · The four diseases of a periodic pipeline: symptom → root cause → do this / not that
| Disease | Symptom | Root cause | Do this | Not this |
|---|---|---|---|---|
| Hanging chunk | Nearly all workers idle; the barrier waits on one chunk | Data skew — the split doesn't match the distribution | Re-shard (salt hot keys, sub-split); run lagging chunks redundantly | Add machines — skew isn't a capacity problem |
| Thundering herd | Shared storage and network time out at start-up, hitting neighbours | Square-wave load; peak far above the mean | Start-up jitter plus rate-limited ramp; switch paradigm once big enough | Just raising timeouts and retries — that amplifies the cascade |
| Moiré load | Rare, periodic, unreproducible giant spikes | Independent schedules occasionally align | Watch total load at cluster level; stagger periods or add global admission control | Investigating pipeline by pipeline — each looks healthy alone |
| Blind monitoring | "Still running," in fact long since stuck | One-off runs with no steady-state baseline | Make data freshness and per-stage completion rate the SLIs | Treating "process alive / CPU busy" as health |
Table 3 · How to deploy the Task Master (the authoritative state)
| Shape | What it survives / doesn't | Fits |
|---|---|---|
| Single instance, one cluster | Simplest; the pipeline stops dead if the process or cluster goes | Experimental or cheaply rerunnable pipelines |
| One cluster plus hot standby | Survives process crashes, not a whole-cluster failure | Ordinary business pipelines |
| ≥2 clusters plus tiebreaker (consensus) | Survives losing a cluster outright; costs cross-region latency, heavier operations, longer failover | Core pipelines with a hard continuity requirement |
This chapter earns its place by turning "batch pipelines are hard to keep alive" from folklore into four predictable failure modes. Orchestrating DAGs today with Airflow, Luigi, Argo or Dagster is still fundamentally the periodic paradigm, and all four diseases apply — the canonical Airflow incident is a few hundred DAGs scheduled on the same minute, taking out the metadata database and the executor together: a textbook thundering herd. Flink, Beam, Google Dataflow and Kafka Streams are the open-source embodiment of the continuous paradigm; Beam's model descends directly from Google's MillWheel and Dataflow papers.
Worth remembering most is the isomorphism of mechanism: Workflow's "Task Master plus unique token," Flink's "JobManager plus checkpoint barrier," and the Kafka transactional producer's "producer epoch fencing zombie producers" are the same idea — concentrate authoritative state in one place, then fence out the stragglers with a credential that expires.
1. A pipeline's enemy is not data volume but the periodic paradigm itself; at scale its failures are predictable.
2. The structural root: real pipelines are chained multi-stage, with a lockstep barrier between stages. Total runtime is the sum of each stage's slowest shard — the average is irrelevant.
3. Hanging chunk: skew leaves one chunk holding up everyone while workers idle. Machines don't fix it — re-shard, or run lagging chunks redundantly (backup tasks).
4. Thundering herd: the timer fires and thousands of workers arrive at once, peaking far above the mean, punching through shared storage and hurting neighbours. Jitter and rate limits shave the peak; the square wave remains.
5. Moiré load: two individually healthy pipelines whose periods occasionally align (3h and 4h collide every 12h) sum into an unplanned spike — hard to reproduce, hard to attribute, and nobody's fault alone.
6. Monitoring blind spot: one-off runs have no steady-state baseline, so "process alive" can't separate working from stuck. Measure data freshness and per-stage completion rate.
7. Google Workflow changes the paradigm to always-on: Task Master holds the one authoritative state (Model), workers are stateless (Controller), configuration is the View.
8. Correctness rests on collaborative locking with unique tokens: output must be committed through the Task Master with a valid token, so a zombie worker's stale submission is refused — the same spirit as an optimistic lock, and as Kafka's producer epoch.
9. The authoritative state is itself a single point, so replicate it across clusters with consensus leader election; the book advises at least two clusters plus a tiebreaker. Mapping to today: Airflow / Argo / Dagster are the periodic paradigm (all four diseases apply), Flink / Beam / Dataflow the continuous one.