BOOKS DEEP-READ · SRE · CH 24
Site Reliability Engineering · Ch 24 · Google (Štěpán Davidovič) · 2016
Think of the standing order on your bank account that pays the rent on the 8th of every month. Paying twice is a mess; not paying at all is also a mess. On servers, the little butler that does "run this thing every day at such-and-such time" is called cron. SRE Chapter 24 is about what happens to that butler when the job no longer belongs to one machine, but to a whole datacenter.
On a single machine, cron is the most boring tool in the box — nobody has worried about it for decades. But once you have a hundred machines, the utterly ordinary sentence "run this once a day at 2am" stops being guaranteed. Either it never ran that day and nobody noticed, or it ran twice and paid everyone twice.
There are only two naive options, and both fail. Give the alarm clock to one designated machine: the day that machine dies, the alarm and the whole list of alarms die with it — and they die silently, because an alarm that never rang doesn't page anyone. You may not find out until Monday. Put the same alarm on every machine: at the appointed second, a hundred machines all start working, and the same job gets done a hundred times.
First, the machines elect one of themselves to be on duty — nobody may self-appoint; it takes a majority to agree. Only the one on duty starts jobs. The others don't start anything, but they must write the same ledger, so any of them can take over at a moment's notice.
How the ledger is kept is the whole trick. For every job, the one on duty writes two lines: before starting, "I am about to do this"; after starting, "I started it, its name is X." Why bother with the first line? Because the one on duty can collapse at exactly the moment of starting. The stand-in opens the ledger, sees the first line but not the second, and knows this job may or may not have happened — it is stuck in a window nobody can read. So the stand-in goes and asks the workshop, "is there a job called X running in there?" If the answer comes back, fine. If it doesn't, all that's left is the rule you agreed on in advance.
Google's rule is: rather miss a run than run it twice. The reasoning is practical — a missed run is visible (a gap in the data, an alert) and you can just run it again; a duplicate run (two emails, two charges) often cannot be taken back.
The price is stated plainly too: it means your job really will occasionally not run, so you must add an alert for "this didn't happen" — otherwise you have only traded noise for silence.
Almost everyone writes "midnight" in their schedule. So at the stroke of the hour thousands of jobs ask the datacenter for machines at once, a flash flood — and the other 59 minutes sit idle. Which is why this cron lets you say "pick any moment inside this window for me," flattening the peak.
The moment a scheduled job leaves a single machine it stops being a small tool and becomes a service that has to elect a leader and keep a shared ledger. And it cannot have it both ways: you must choose in advance between missing and repeating, then put an alert on whichever one you didn't choose.
Want the crash-window timeline, where the state lives, and what Kubernetes admits in public? → Switch to the deep read
On one machine cron is a tool nobody has thought about in decades; move it into a datacenter and it instantly becomes a stateful service that only distributed consensus can make correct. And the hardest line in the chapter is this: it cannot have exactly-once — you pick a side between "may skip" and "may repeat," then cover the other side with idempotency and alerting.
crontab is the schedule itself, and 0 2 * * * means "every day at 02:00."Written by Štěpán Davidovič, this chapter belongs to the run of Part III ("Practices") chapters about reliability of stateful systems. It follows Ch23, Managing Critical State — that chapter is about consensus itself, and this one is its smallest, most concrete application: the state the whole system must hold under consensus amounts to little more than "when did each job last launch." It leads into Ch25, Data Processing Pipelines — the periodic jobs cron fires. Real-world counterparts: Kubernetes CronJob, scheduled triggers in Airflow and Argo, managed cloud schedulers, and every "runs nightly" reconciliation, cleanup and reporting job you own.
Single-machine cron has a one-line reliability model: it equals the reliability of that machine. It quietly assumes two things — the process stays alive and local state doesn't vanish. Neither holds in a datacenter: machines get preempted, reimaged, or lose a whole rack of power. And the crontab lives on that machine too, so when it goes, even the list of what should have run goes with it.
The lethal part isn't the probability, though — it's the failure mode. A request that turns into a 500 shows up on a dashboard. A scheduled job that did not happen emits nothing at all: no error, no log line, no metric. The classic discovery path is somebody asking on Monday morning, "where's Friday's report?"
So at scale two things become mandatory: (1) decouple "start on time" from any one machine, and (2) record "did it run" somewhere that cannot die. And the moment several machines could each execute while the state must be visible across all of them, you have walked straight into the chapter's real problem: double launches and missed launches — one of the two will happen to you.
The design makes cron a replicated service: replicas elect a leader via Paxos, and only the leader actually launches. Followers launch nothing, but they must track the same state, because any of them may have to take over.
The tempting shortcut — "run it everywhere and dedupe with a distributed lock" — is not enough here: holding the lock doesn't mean downstream honours it. A holder can stall for tens of seconds between acquiring the lock and launching (a long GC, a congested network); the lock expires, somebody else takes it, and the stalled process wakes up and launches anyway. Two "unique launchers" now coexist. To make a lock actually safe you need fencing: each grant carries an increasing number and the downstream — here, the datacenter scheduler — rejects stale ones. Which is exactly Ch23's punchline: you thought you were using a lock; you were implementing consensus.
This is the technical heart of the chapter. At each scheduled point the leader performs not one action but three: (1) write "about to launch X for schedule time T" into the consensus log; (2) call the datacenter scheduler; (3) once it returns, write "launched" back into the log.
Why the first write? Because the leader can vanish at any instant. Three cases: a crash before (1) — the new leader sees nothing, so it knows nothing launched and can launch; a crash after (3) — it knows it launched and skips; a crash between (1) and (3) — unknown: the request may never have left, or the scheduler may have accepted it and only the acknowledgement was lost. That window is usually tens to hundreds of milliseconds wide, and it never goes away.
The way out is to make "just go and ask" possible: derive each launch's instance name deterministically from (job name, schedule time) — say job-X@2026-09-25T02:00. The incoming leader takes that name to the scheduler and asks whether such an instance exists. Found: record it as launched. Definitively absent: launch it. Only when the scheduler itself can't answer (restarting, request timing out) do you fall through to the default policy.
What to take away: cron and the scheduler are two systems, and there is no atomic commit across two systems. Every exactly-once fantasy dies in that seam. The chapter's virtue is that it refuses to pretend the seam can be closed and instead picks a side out loud: Google's cron prefers to skip rather than to double-run, because a skip is observable and repairable while a duplicated external side effect usually isn't.
The state that must survive is astonishingly small: per job, the schedule time of the last launch, plus any pending launch intent. That makes "carry your own consensus log" a very good deal — the whole state fits in memory with periodic snapshots, and one launch costs one round trip plus one fsync (sub-millisecond within a datacenter, a few milliseconds for the fsync — single-digit milliseconds in total), which is nothing against a schedule measured in minutes.
Hanging the state on etcd, ZooKeeper or a database works too and saves you writing replication, but it pins cron's availability to another system — and worse, when that system is down you cannot even answer "did this run?"
The chapter draws a firm line: distributed cron's job ends when the task has been handed to the datacenter scheduler. How long it runs and whether it produced the right answer belong to the scheduler and the task. That isn't laziness, it's scope control — the moment cron starts tracking long-running lifecycles and expressing "run B after A succeeds," it has become a workflow engine, which is a harder problem wearing the same clothes.
The practical rule that falls out: keep whatever cron fires as short as possible, ideally just the launch; give genuine multi-step flows to a workflow engine.
Humans have a powerful bias toward round numbers. Suppose a cluster carries 2,000 periodic jobs and 40% of them say 0 0 * * *: that is 800 jobs asking for resources in the same second, the scheduler queues, startup latency spikes — and the next 59 minutes are idle. So this cron extends the crontab syntax to let you say "pick any value in this range for me," spreading launches across the window. The overwhelming majority of daily jobs genuinely do not care whether they start at 00:00 or 00:37.
Time zones are the second trap: on a DST switch day, local time loses or repeats an hour, so "every day at 02:30" either doesn't exist or fires twice. The standard engineering answer is to keep all internal bookkeeping in UTC and convert only for display.
Table 1 · Skip or repeat: pick a side by the nature of the job, and say which side you picked
| Job type | Lean toward | Why | How to cover the other side |
|---|---|---|---|
| Push / email / charge / issue credit | Rather skip | External, irreversible side effect; one duplicate is a real loss | Idempotency keys all the way to the payment gateway and mail provider; alert on "this run did not start" |
| Daily / monthly close, regulatory reports | Rather repeat | A miss is a compliance incident; a recompute only costs machines | Write results keyed by (job, period) as an overwrite — re-runs are naturally idempotent |
| Cache warming, index rebuilds, cleanup | Either | Naturally idempotent; a repeat just burns CPU | Set concurrency policy to "skip this run if the last one is still going" |
| A stage of a data pipeline | Rather repeat | Downstream overwrites by time partition, so re-runs converge | Put the schedule timestamp in the output path; never append |
Table 2 · Where "did it run" is stored
| Option | Cost per launch | Failure surface | Fits |
|---|---|---|---|
| Own consensus log | 1 RTT + 1 fsync, single-digit ms | Self-contained, no external dependency | The chapter's choice: the state is tiny, so owning it is cheap |
| External strongly consistent store | One remote write plus that store's latency | One more hard dependency; when it's down you can't answer "did it run" | Teams already running etcd or ZooKeeper |
| A lock and no recorded state | Cheapest | A successor cannot answer "did the last one run?" | Don't |
Table 3 · Which tier you actually need
| Situation | Choice | Cost / precondition |
|---|---|---|
| One machine, misses tolerable | System cron | Machine dies, everything stops; you must add a "didn't run" alert |
| Already on Kubernetes | CronJob | It only promises "approximately once"; set concurrencyPolicy and startingDeadlineSeconds explicitly |
| Multi-step, dependencies, retries | Workflow engine (Airflow / Argo / Temporal) | Another system to operate, in exchange for replayable, observable intermediate state |
| Cross-datacenter, tens of thousands of jobs | Build distributed cron | Consensus, resolution logic and herd-flattening are all yours to carry |
The value of this chapter isn't that it teaches you to build a cron. It's that it reopens a problem everyone assumed was solved and hands you an answer you can copy honestly: admit exactly-once is unavailable, and state which side you chose. Every mainstream scheduler today is saying the same thing in its own words.
EVIDENCE FROM THE FIELD
too many missed start times ... or check clock skew. Missed runs aren't a theoretical risk; they're a constant you have to hard-code.kubernetes/kubernetes · pkg/controller/cronjob/utils.goIn an interview: asked to "design a distributed scheduled-job system," the scoring rubric is essentially this chapter's table of contents — who launches (election), where "did it run" lives (state under consensus), how the crash window is resolved (two synchronization points plus a derivable instance name), which semantics you chose and declared (skip vs repeat), how the herd is flattened, and where the boundary is (no babysitting long jobs). Saying "exactly-once isn't available here" unprompted already wins half the room.
1. Once cron leaves a single machine it is a stateful service that needs consensus. Single-machine cron is as reliable as that machine, and its failures are silent.
2. Core structure: replicas plus Paxos leader election; only the leader launches. Followers launch nothing but must track state to be able to take over.
3. Core mechanism: two synchronization points — write "about to launch" before, "launched" after. A crash between them is the unknown window.
4. Resolve it with a deterministically derivable instance name (job + schedule time): the successor asks the datacenter scheduler by name, and only falls back to policy when it gets no answer.
5. No atomic commit across systems, therefore no exactly-once. Google's cron chooses skip over double-run — a trade-off, not a truth; reconciliation jobs should choose the opposite.
6. The durable state is tiny (one "last launch time" per job), so owning a consensus log is the cheap option: roughly 1 RTT + 1 fsync, single-digit milliseconds per launch.
7. Hold the boundary: cron launches and does not track long jobs. Dependencies and retries mean you want a workflow engine.
8. Treat the herd: let the system place launches inside a window instead of stacking everything on 0 0 * * *; keep internal time in UTC.
9. Alert on absence. If you chose "rather skip," you owe the system a way to turn that silence back into a signal — which is all Kubernetes' "approximately once" really says.