Book Deep-Read · DDIA · Chapter 8
Designing Data-Intensive Applications · Ch 8 · Martin Kleppmann · 2017
The earlier chapters sold you on spreading data across many machines. This one is the bad news: the moment your system is many machines talking over a network, a whole zoo of problems appears that simply didn't exist on a single computer. DDIA (Designing Data-Intensive Applications) Chapter 8 lays those traps out one by one to scare you—not to stop you, but to cure you of the fantasy that a pile of machines behaves like one big, obedient machine.
Picture you and some colleagues in different cities who can only reach each other by mailing letters. You send a letter asking "did you do it?"—and then… silence. Now what? Did the letter not arrive? Did they fall ill? Did they reply but the reply got lost? Or are they doing it slowly and just haven't answered yet? You genuinely cannot tell. One computer shouting across the network at another feels exactly like this: silence has many explanations, and you must keep getting things right without being able to tell them apart.
A single computer has one redeeming habit: it either works, or it crashes completely (freeze, blue screen)—and a total crash is obvious, so you just reboot. It rarely goes "half-dead." But a crowd of computers produces partial failure: some machines are fine, some aren't, some links work and some don't—and you often can't tell which is which. That "can't tell who's broken" uncertainty is the real torment of distributed systems.
This chapter teaches you to be permanently suspicious of three things. ① The network—letters get lost, arrive late, arrive out of order; your only tool is "wait a while, and if no reply, assume it's gone," but how long to wait is pure guesswork. ② Clocks—every machine's watch runs slightly differently, like a room full of people with unsynchronized watches, so you cannot use "whose watch shows a later time" to decide who acted last, or you'll mis-judge and lose data. ③ Sudden pauses—a machine can freeze for seconds with no warning (like someone dozing off mid-sentence), then wake up thinking nothing happened and carry on doing what it should have stopped doing.
The mindset is a single line: assume anything that can go wrong will, then design it so the result is still correct even when it does. One classic trick is "take a number": hand each operation a ticket number that only ever increases; if you wake from your nap still clutching an old number, the counter sees it's stale and refuses to serve you—so the damage never happens.
The reason distributed systems pile on all this seemingly fussy machinery—majority votes, timeouts, retries, reject-the-old-ticket—is that the network, the clocks, and the other machines can none of them be fully trusted. The honest cost: all this defensive plumbing makes a distributed system far slower and more complex than a single machine—that's the tuition you pay for not being able to trust anything.
With many machines, the worst trouble isn't "one of them broke"—it's "you can't even tell whether one broke." The network will lie to you, the clocks will lie to you, machines will nod off—treat them all as untrustworthy, and design so that even when they fail, nothing goes haywire. That's the distributed way.
Want the actual mechanisms, notation, and diagrams? → Switch to the deep read
This chapter is the book's cold shower: it systematically catalogs everything that can go wrong in a distributed system—partial failure, unreliable networks, unreliable clocks, unannounced process pauses—and teaches a systematic pessimism: assume anything that can break will break, then design the system to stay correct anyway. It offers no solutions, only a hard look at the traps, setting up the next chapter on consensus: only once you see why distributed systems are this hard can you ask what we can still guarantee.
This chapter is in Part II, "Distributed Data," right after replication (Ch 5), partitioning (Ch 6), and transactions (Ch 7). Those chapters quietly assumed networks, clocks, and machines were roughly trustworthy; this one punctures those assumptions one at a time. It is Part II's "problem" chapter—all trouble, no answers; the very next chapter, "Consistency and Consensus" (Ch 9), is the "solution" chapter: given all this uncertainty, what can we still reliably guarantee? Only after absorbing Ch 8's pessimism do you earn Ch 9's optimism.
A single computer has a property we take for granted: determinism—same input, same output; and when hardware fails, it usually crashes wholesale (all-correct or all-stopped), which you notice immediately and fix by rebooting—rarely "half working." Distributed systems shatter that comfort: partial failure is the norm—a node is down, a link is severed, a machine has slowed to a crawl, yet the system as a whole keeps running. Worse is the nondeterminism: you send a request and get no reply, and you simply can't distinguish whether the request never arrived, the peer crashed, the reply was lost, or the peer is merely slow. The task of this chapter isn't to "fix one fault"—it's to force a new worldview: stop treating a crowd of machines as one reliable big computer; assume every part can fail in the worst possible way, and build a reliable system on top of that.
Put a single machine and a distributed system side by side and it's clear: the single machine is like a reliable processor—same op, same result forever, and it fails cleanly. A distributed system is a shared-nothing crowd talking over a network—which introduces partial failure: some parts break, some stay fine, and you often can't tell where the boundary is. That "can't tell who broke" nondeterminism is the core engineering difficulty. Accepting it is the starting point for everything in this chapter: rather than dreaming of components that never fail, build a correct whole out of components that do fail.
Shared-nothing systems can communicate only through an asynchronous packet network—"asynchronous" meaning the network makes no promise about delivery time: packets may be lost, delayed, reordered, or duplicated. The most painful case: after you send a request you get no reply—maybe ① the request was lost, ② the peer crashed, ③ the reply was lost, or ④ the peer is just slow. All four look identical from the sender's side; you can't tell them apart (see Figure 1).
The only usable tool is the timeout: wait a while, and if no reply, assume the peer is dead. But timeouts cut both ways—too short, and you misjudge a merely-slow healthy node as dead, triggering needless failover (or executing an action twice); too long, and even a real crash leaves you waiting ages before you react. Since async network delay has no upper bound in theory, no perfect failure detector exists. The bulk of delay often comes from queueing: at network switches, the OS, the VM, and the receiving process, packets pile up when things are busy. DDIA's stance is blunt: a network's reliability must be measured, not assumed—even inside a single data center.
Every machine has its own quartz clock, which drifts. Machines resync periodically via NTP, but that sync is subject to network delay and is only so precise. DDIA's numbers make it vivid: Google budgets 200 ppm (parts per million) of drift for its servers—about ~6 ms even if you resync every 30 seconds, and up to ~17 seconds if NTP fails for a whole day. A tiny error—yet enough to get "who came first" wrong.
The key is to tell apart two kinds of clock. ① The time-of-day clock returns "what time is it now," gets nudged forward and back by NTP, and can even jump backward (leap seconds, manual adjustment)—so it must never be used to measure intervals or order events. ② The monotonic clock only guarantees "always moving forward," good for "how long has elapsed," but its absolute value is meaningless and not comparable across machines. The most dangerous misuse is stamping writes with a time-of-day clock and resolving conflicts by last-write-wins (LWW): if two machines' clocks are out of sync, a write that physically happened later may carry a smaller timestamp, be judged "older," and be silently discarded (see Figure 2)—and data vanishes without a sound.
The honest approach is to admit a clock carries a confidence interval—"it's about X, ±e." Google's Spanner takes this to its logical end: its TrueTime API returns a time interval rather than a point, and when committing a transaction it deliberately waits out that uncertainty (commit wait), yielding a trustworthy global ordering.
Even if the network and clocks behave, a node can pause at any moment with no warning: most commonly a stop-the-world garbage collection—the whole process freezes for tens of milliseconds to seconds, and in extreme cases (large heaps) minutes; or a VM suspended for migration, a laptop lid closed, OS paging thrash. The killer detail: the paused process has no idea it stopped—it was fine before freezing, and after thawing it just carries on, oblivious that the world has moved.
The classic wreck: a node takes a lease from the lock service and becomes leader; just as it's about to write shared storage, it pauses for 15 seconds in GC; meanwhile its lease has long expired and the system elected a new leader. When it thaws, it still believes it's the leader and writes with stale authority—split-brain: two "leaders" corrupt the data at once. The fix is a fencing token: every grant carries a monotonically increasing number, and storage accepts only writes whose number is larger than the last one, rejecting smaller ones. The just-woken stale leader clutches an old number, and its write is blocked (see Figure 3)—the system no longer trusts "I think I'm the leader," only the number on the token.
The soul of this chapter: there's no free determinism in a distributed system—it's trade-offs all the way down. Start with the two most consequential dials—timeout length and whether to defend against Byzantine faults.
Table 1 · Timeout too short vs too long: both wrong, pick the tolerable wrong
| Timeout too short | Timeout too long | |
|---|---|---|
| Consequence | Misjudges a merely-slow healthy node as dead | Even a real crash leaves you waiting a long time to detect and fail over |
| Knock-on risk | Needless failover; an action may be executed twice; the failover itself adds load and can cascade | Users keep suffering during the outage; recovery time (RTO) grows |
| Root cause | Async network delay is unbounded → no perfect failure detector, only a choice between two errors | |
Table 2 · Two fault assumptions: defending against "lying nodes" costs worlds apart
| Non-Byzantine (crash / omission) | Byzantine (may lie / act arbitrarily) | |
|---|---|---|
| Assumption | Nodes may crash, slow down, drop messages, but if they speak they tell the truth | Nodes may send wrong / contradictory / malicious messages (faulty or compromised) |
| Cost | Protocols are simpler, cheaper (Paxos / Raft live here) | Needs BFT protocols, typically > 2/3 honest nodes; expensive and slow |
| Fits | Controlled data centers—the default for most systems | Aerospace, mutually-distrusting parties, blockchains—adversarial settings |
One level up sits the choice of which system model to reason with—it fixes how bad your algorithm assumes the world can be.
Table 3 · System models: pinning down "how unreliable is the world"
| Timing assumption | Meaning | Realism |
|---|---|---|
| Synchronous | Network delay and process pauses have a known upper bound | Too optimistic; almost never holds in reality |
| Partially synchronous | Bounded most of the time, occasionally exceeds it | Closest to real systems; most algorithms target this |
| Asynchronous | No timing assumptions at all—can't even use a clock | Most conservative; algorithms proven correct here are the sturdiest |
Also distinguish two kinds of correctness property: safety—"nothing bad ever happens" (e.g., a token is never mistaken, no wrong result is returned), which must hold at all times; and liveness—"something good eventually happens" (e.g., a request eventually gets a reply), which is allowed the qualifier "eventually." The common engineering trade-off: sacrifice liveness (wait longer, be temporarily unavailable) before violating safety (never return a wrong result)—and GitHub's outage below is exactly that choice in the flesh.
This chapter is distributed engineers' mandatory disillusionment course because nearly every large-scale production incident maps onto its checklist: clocks gone wrong, network partitions, process pauses, failure-detection misfires. It gives a shared language for debugging and design: "is the leader down?" becomes "is this a real crash or a network partition / GC pause?"; "whose write wins?" becomes "don't trust the wall clock—use a fencing token / version number"; "do we fail over?" becomes "what's the timeout, and could it mis-fire and cascade?" In real systems, ZooKeeper / etcd guard critical state with consensus plus fencing tokens, Spanner quantifies and waits out clock uncertainty with TrueTime, and Jepsen deliberately hammers databases under network partitions to expose consistency promises they can't keep.
0.2% of DNS queries—a live confirmation of "time can jump backward; never assume the clock is monotonic." The fix was a single character (turning "== 0" into "<= 0"). Cloudflare Blog, "How and why the leap second affected Cloudflare DNS," 2017 ↗43 seconds of connectivity loss between East and West coasts triggered Orchestrator's automatic failover, the primary diverged across sites, and it snowballed into 24+ hours of degraded service—a confirmation of the "network partition + automatic failover" cascade. GitHub stated plainly it would prioritize data integrity over availability—safety over liveness, in the real world. The GitHub Blog, "October 21 post-incident analysis," 2018 ↗<10 ms; TrueTime returns a time interval and "waits out" that uncertainty at commit, enabling globally externally-consistent transactions—turning "put a confidence interval on the clock" into a shipping product. Corbett et al., "Spanner," OSDI 2012 ↗① In one line: the essential difficulty of distributed systems is partial failure + nondeterminism—not "one of them broke," but "you can't even tell whether one broke."
② Don't treat a crowd of machines as one reliable big computer; build a correct whole out of failing parts through systematic pessimism.
③ Network untrustworthy: async delay is unbounded, "no reply" has four indistinguishable explanations; the timeout is the only tool, but it's a guess and there's no perfect failure detector.
④ Clocks untrustworthy: separate the time-of-day clock (can jump—never order with it) from the monotonic clock (intervals only); LWW on timestamps causes silent data loss; the fix is a confidence interval (Spanner TrueTime).
⑤ Processes pause with no warning (GC / migration / paging): the lock holder may be dozing and wake up still thinking it's the leader → split-brain.
⑥ Fencing tokens (monotonically increasing numbers; storage accepts only larger) block the just-woken stale leader—trust the token, not "I think I'm the leader."
⑦ Key trade-offs: timeouts are wrong either way—pick the tolerable wrong; data centers default to no Byzantine defense; reason with the partially-synchronous model; safety before liveness.
⑧ Why it matters: a shared language for debugging and design, and the target-setup for Ch 9 "Consensus"—see the trouble clearly before asking what can still be guaranteed.