BOOK DEEP-READ · DDIA · CHAPTER 12
Designing Data-Intensive Applications · Ch 12 · Martin Kleppmann · 2017
When you change your shipping address on a shopping site, that one edit has to reach several different systems: the one behind the order page, the one behind the search box, the one support staff look at, the one that runs reports overnight. The first eleven chapters were all about doing one system well. In this last one, the author finally says the whole thing out loud: how should all these systems be joined together so they stop contradicting each other? This is not a summary — it is his own argument.
The usual approach is "change it everywhere": the program updates the main database, then goes and updates the search copy too. Sounds obvious, but it has a hidden flaw. When two people edit the same record at almost the same moment, the main database may see Ann first and Bob second, while search happens to see Bob first and Ann second. The two sides end up with different final answers — and neither will notice, let alone fix it.
The hard part is not writing the data. It is that nobody is in charge of the order: each system only sees the edits that reach it, and acts on whatever sequence it happened to observe. It is like a dozen departments each keeping their own ledger — every book looks fine alone, but they never reconcile. And the more systems you have, the more wires between them: ten systems wired pairwise is up to ninety connections.
The author's answer is almost disappointingly plain: keep one single ledger, make every change line up and get written into it first, and treat every other system as nothing more than a copy of that ledger. The order is fixed once, in the ledger; nobody gets to invent their own. Downstream systems copy from that one sequence — some faster, some slower, but what they end up with is guaranteed to match.
The same move solves two chronic headaches. Adding a new system stops being scary: want to launch a new recommender? Let it copy the ledger from page one. And mistakes become undoable: replay the ledger with the fixed logic, build a fresh copy alongside, and switch over once it checks out — the old copy sits untouched, so you can always go back. The author puts it more boldly still: what we call a "database" is really just storing, indexing and looking things up sold together in one box. Take the box apart, hand each job to the tool best at it, and thread the ledger between them — now you have one enormous database spread across the whole company.
You still have to guarantee a thing happens exactly once. A user double-taps the pay button, and de-duplication at any middle layer can quietly fail. The author's answer is to put the two ends in charge: issue a unique ticket number with the operation at the moment it starts, carry it all the way through, and have the far end honour the ticket, not the request — the same number only ever counts once.
The other half of the answer sounds more like a merchant than an engineer: not every rule has to be enforced on the spot. Airlines overbook flights anyway; they reconcile afterwards and compensate whoever got bumped. "Never be wrong" is traded for "notice when we're wrong and make it right." And that is exactly where this architecture's honest cost lies: every copy lags a little by design — you get rebuildability and room to grow, and you pay with "just because you wrote it doesn't mean everyone can see it yet." The book closes on a turn few engineering books take, asking not what we can build but what we should: models trained on history quietly copy old prejudice into the future, and the user data you hoard is less an asset than a liability waiting to go off — delete it once you no longer need it.
Stop letting each system do its own writing in its own order. Keep one ledger, make every change queue up in it, and treat the other systems as copies; for correctness, trust no middle layer — have the two ends honour a ticket number; and for rules you cannot enforce on the spot, reconcile afterwards and make amends.
Want the actual mechanisms, notation and diagrams? → Switch to the deep read
You expect a final chapter to revise; this one declares. Kleppmann gives his own answer: take the packaged product called "the database" apart (unbundling) — storage, index maintenance and querying were always three jobs that different tools could do — and re-thread them into an organization-wide dataflow using one ordered, replayable event log. Correctness, meanwhile, should not be delegated to low-level transactions; it has to be enforced end-to-end, and "consistency" should be split into timeliness and integrity, two properties that deserve very different treatment. He closes with a question engineering books rarely ask: given these capabilities, should we use them all.
This chapter closes Part III, "Derived Data," and the book itself. It carries forward the "immutable input + derived results" worldview of Ch 10 and Ch 11, pulls back in replication, partitioning, transactions and consensus from Ch 5–Ch 9, and lands on a way of assembling systems at organizational scale. In the real world it maps onto any company running a primary database, a search cluster, a cache, a warehouse and a feature store at the same time — which is to say almost every company.
The subtext of the first eleven chapters was always "pick the right tool." Real companies never have just one: a mid-sized e-commerce shop runs PostgreSQL as the primary, Elasticsearch for search, Redis for hot data, a warehouse for analytics, plus separate feature stores for risk and recommendations — the single fact "the user changed their address" has to show up in five or six places. No database wins at every access pattern, so having many systems is not bad design; it is inevitable.
The real question becomes: how do you keep those five or six copies consistent, and add a seventh without everything falling over? Left unsolved, you get the pain every engineer knows — search results that disagree with the detail page, reports a few hundred dollars off the live books, and a fragile new dual write bolted into the application for every system you onboard. Worse, this kind of inconsistency does not heal on its own; it is a different animal from replication lag, which does.
Get the target straight first. A dual write is those two innocent-looking sequential calls in application code: db.update(...) then search.index(...). It has two diseases. A race condition: two clients modify the same record at nearly the same instant, the primary sees Ann then Bob while the search index sees Bob then Ann, and the two are left with different final values — not "consistent in a moment," but permanently divergent. And partial failure: the first write succeeds, the second fails, and the two systems fork. You could wrap both writes in a distributed transaction, but that demands heterogeneous systems supporting the same atomic commit protocol and holding locks on each other while committing.
The fix is to reframe the problem: instead of the application writing to two places at once, designate one system of record. All writes land there first and become an ordered event stream; every other system is demoted to derived data that simply subscribes and follows. The decisive gain is that ordering happens exactly once — every consumer sees the same sequence, so their final states must converge.
The author's conclusion is refreshingly practical: absent a widely supported distributed transaction protocol, log-based derivation is the most promising approach to integration — it swaps "mutual exclusion + atomic commit" for "deterministic recomputation + idempotent retries," which is one notch weaker but asynchronous and forgiving of failure. The limits deserve saying plainly too: the foundation is total order, and deciding that order takes a single point or a round of consensus, so throughput is capped by one machine. Multi-region active-active deployments, microservices holding independent state, and clients that edit offline all resist being squeezed into one global queue — and DDIA can only call this an open research problem.
Derived data lives by recomputation, and there are two ways to compute: batch processing chews through history in one pass, stream processing handles events one at a time. The Lambda architecture was the popular compromise: a slow-but-accurate batch layer plus a fast-but-approximate speed layer, merged at query time. DDIA's criticism is blunt: the same business logic has to be written twice, in two frameworks, and kept equivalent forever — a permanent maintenance tax — and the merging logic is itself a fresh source of bugs. The author argues for unification: if the log is replayable, "batch" is just "running the same stream operators over a bounded slice of history." One codebase, one set of semantics.
Hidden here is the chapter's most useful trick: reprocessing is a legitimate mechanism for evolving an application. Swapping in new recommendation features or reshaping a data model no longer requires a risky in-place migration — replay the log with new code, build a parallel new view, keep both alive for a while, cut reads over once it validates, and cut back if anything breaks. This pushes Ch 4's "evolve schemas without downtime" all the way up into the derived layer: migration stops being a big bang and becomes a reversible, gradual process. The cost is that the log must be retained long enough (weeks to months is typical), and recomputing history burns a chunk of extra compute.
This section is the intellectual core. The author's observation: what a single database does internally and what a company's cross-system dataflow does are the same thing at two different scales.
Inside a database you find a write interface, a storage engine with a write-ahead log, secondary index maintenance (updating indexes as records are written), materialized view maintenance, triggers, a replication log, and query execution. Pull them apart: an event log like Kafka ≈ the replication log; a stream processor ≈ triggers and materialized view maintenance; the downstream search indexes, caches and OLAP tables ≈ assorted indexes. So the dataflow across an organization is one enormous database turned inside out — what the author calls the "meta-database of everything."
Follow that view and there are only two routes to integrating heterogeneous systems, one per side:
foreign data wrapper being the classic). It leaves the write side alone and is quick to adopt; but every query fetches across systems live, so performance and availability are at the mercy of the slowest source.The author's own restraint belongs here, or the chapter reads as evangelism: unbundling is not meant to replace databases. He says plainly that a single integrated product is faster and simpler within the scenarios it was built for; unbundling wins not on performance but on breadth of coverage — it only pays off when no single piece of software can satisfy all your requirements at once. He also leaves a note of regret: we still lack a high-level language for composing storage systems. We can pipe commands together the way Unix does, but not storage systems.
Push all of that up to the application layer and you get a genuinely useful mental model: every journey from "written" to "seen" has two segments — the write path is work done up front at write time, the read path is work done live at query time. And caches, indexes and materialized views are all the same move: sliding that dividing line toward the write path.
The model explains a lot of decisions usually made on instinct. Why build a full-text index? Because it does the "find by keyword" work ahead of time. Why not materialize every query? Because each notch you slide costs more write-side maintenance and more lag. The choice is never "should we cache," it is "where should this boundary sit." The author pushes the same flow all the way to end-user devices: the local state in a single-page app is a replica of server state, and WebSocket push extends the write path's terminus onto the user's screen.
Once it is assembled, how do you keep it correct? The author opens with a sharp question: are transactions actually enough?
Take "exactly once." A user taps pay, the page hangs, and they tap again. De-duplication in the middle layers is useless here: TCP's retransmission de-duplication only holds within one connection; a database transaction guarantees this one is all-or-nothing, but the user submitted two independent, individually valid transactions; and even 2PC's cross-system atomic commit cannot stop a human hitting submit twice.
The only place that works is the ends: have the client generate a unique operation ID (a UUID, say) when it initiates, carry it all the way to the final endpoint, and use it there as the de-duplication key — the same ID only counts once. This is the end-to-end argument of Saltzer, Reed and Clark (1984) resurfacing in data systems: certain functions can only be implemented correctly at the endpoints that hold the complete semantics; however well a middle layer does it, it is an optimization. The corollary stings: low-level reliability mechanisms — TCP checksums, transactions, replication — are all useful, and none of them is sufficient on its own to make the application correct.
Now constraints. A uniqueness constraint like "usernames must be unique" fundamentally needs consensus (Ch 9). But a log architecture has an elegant answer: partition by the constrained value (hash the username, say), so every request for a given username necessarily lands in the same queue; a single-threaded consumer processes them in order and the first one wins, the rest are rejected — a deterministic verdict with no cross-machine locking. Cross-partition operations (transferring from account A to B) follow the same shape: atomically append the request itself to the log as one event with an ID, then derive both account balance changes downstream. The effect is equivalent to 2PC, in a far simpler and more fault-tolerant shape.
Finally, the chapter's most valuable distinction: consistency is really two things.
The author's line, in substance: a violation of timeliness is "eventual consistency"; a violation of integrity is perpetual inconsistency. Its value is this — ACID transactions deliver both bundled together, at a steep price, while dataflow systems can unbundle them: hold integrity absolutely, via deterministic derivation plus end-to-end operation IDs, while relaxing timeliness. That is precisely why they can dodge cross-partition coordination and still run fast and stable; the author calls such designs coordination-avoiding — not uncoordinated, but with the expensive coordination compressed down to the small part that truly needs it. There is a counterintuitive corollary: plenty of constraints need not be enforced on the spot. Airlines overbook and warehouses run short, and in both cases the answer is let it through, reconcile later, apologize and compensate when it breaks.
Last comes "trust, but verify." Software has bugs, hardware corrupts silently, people fat-finger things — never treat "the data cannot be wrong" as an axiom. Event-sourced systems have a natural advantage: the raw events survive and the derivation is deterministic, so you can replay and compare against what you currently hold. That is auditing. The author is cautious about blockchains, but says clearly that he would like their cryptographic auditing and integrity-checking ideas (Merkle trees and the like) to reach mainstream data systems.
The book's final section turns to ethics, and not briefly. Three points. Predictive analytics copies historical prejudice into the future: a model trained on past data hardens old discrimination into an "objective algorithm," and the veneer of data makes it harder to appeal. Feedback loops are self-reinforcing: someone judged high-risk gets worse terms, becomes likelier to fail, and thereby "confirms" the algorithm. And data is a liability, not an asset: hoarded user data can leak, be abused, or break the law, and should be deleted once it is no longer needed. He also punctures "the privacy policy counts as consent" — users have no real choice, so it is not freely given.
Table 1 · Three ways to integrate: distributed transactions vs dual writes vs log-based derivation
| Distributed txn (2PC) | Application dual write | Log-based derivation (this chapter) | |
|---|---|---|---|
| Who decides order | Locks + atomic commit, forced mutual exclusion | Nobody — each system acts on arrival order | The log, once; downstream copies it |
| Consistency | Immediately consistent | Possibly permanently divergent | Eventually consistent (ms ~ seconds of lag) |
| One downstream dies | May stall every participant | Forks on the spot, needs manual repair | Only its own offset lags; catches up on recovery |
| Adding a 7th system | Enroll it in the transaction; complexity spikes | Another dual write in the application | Replay the log from the start; upstream untouched |
| Heterogeneous stores | Requires one shared protocol — rarely available | Attach anything, break anything | Only requires consuming events |
| Use when | Homogeneous, strict constraints, contained scale (core financial ledgers) | Basically don't | Heterogeneous integration, derived data (search/cache/DWH/features) |
Table 2 · Lambda architecture vs unified batch-and-stream
| Lambda architecture | Unified (one log + one set of operators) | |
|---|---|---|
| Approach | Batch layer computes exact values, speed layer approximates, merged at query time | Streams only; "batch" = same operators over a bounded slice of history |
| Code | Same logic written twice, kept equivalent forever | One codebase |
| Merge cost | The merging logic is itself a new bug source | None |
| Fixing errors | Wait for the next batch run to overwrite | Replay the log, rebuild the view, validate, cut traffic |
| Cost | Two clusters, two operational burdens | Log retained long enough (weeks~months); recompute burns capacity |
Table 3 · Two integration routes: unify reads vs unify writes
| Federated query (unify reads) | Unbundling (unify writes) | |
|---|---|---|
| What changes | Add a query proxy; write side untouched | Rework the write path; all writes hit the log first |
| Where data lives | Still in each system, pulled at query time | Each system holds its own derived copy |
| Coupling | Tight at query time: one slow/dead source, one slow/dead query | Loose: consumers follow asynchronously, independently |
| Cost to adopt | Low — good for stopping the bleeding | High — an architectural investment |
| Use when | Ad-hoc cross-store analysis, legacy integration | A core data platform meant to evolve for years |
Table 4 · Timeliness vs integrity: does this constraint really need strong coordination?
| Constraint | Cost of violating it | Which to hold | What people actually do |
|---|---|---|---|
| Unique username | Two identical names, manual rename | Integrity | Partition by username; order within the partition decides, first wins |
| Transfers don't mint money | Books permanently fail to reconcile | Integrity, never yield | Append the request with an ID atomically, derive balances deterministically — 2PC-equivalent |
| Seeing the latest balance | Briefly reading a stale value | Timeliness, relaxable | Accept seconds of lag; route critical pages to the primary |
| No overbooked seats | Compensation and rebooking, absorbable | Relaxable | Deliberately overbook + compensate later; cheaper than coordinating |
| No overselling stock | Refund and apologize when short | Depends on order value | Cheap and frequent: allow then reconcile. Costly and scarce: reserve stock at checkout |
| Recommendation freshness | Slightly worse suggestions | Heavily relaxable | Minutes of lag are entirely acceptable |
One test runs through all of it: ask first whether a violation heals itself. What heals (timeliness) should be relaxed as far as it will go, buying throughput, availability and a simpler architecture; what does not heal (integrity) must be held with deterministic derivation and end-to-end IDs. Pricing those two separately is the sharpest tool this chapter hands to architectural decisions.
This chapter's argument is now the default shape of a modern data platform, just under other names: Kafka as the integration bus, CDC tools like Debezium turning database changes into streams, Flink / Kafka Streams as the materialized-view maintainer turned inside out, and derived stores landing in Elasticsearch / ClickHouse / lakehouse table formats (Iceberg, Delta, Hudi). That architecture diagram at any mid-to-large company — Kafka in the middle, a fan of derived stores below — is this chapter's unbundling blueprint. In interviews and design reviews it cashes out into several weighty sentences: why dual writes are wrong, why "replay, rebuild, then cut traffic" is safer than an in-place migration, why unique usernames need no distributed lock, and the best tool of all — splitting consistency into timeliness and integrity and pricing them separately.
N systems means up to N(N−1) directed pipelines; N=10 is already 90), whereas one durable, ordered, replayable log collapses that into "everyone writes to and reads from one bus" — confirming that unbundling works by unifying the write path. J. Kreps, "The Log," LinkedIn Engineering 2013 ↗① In one line: the final chapter is a manifesto, not a summary — take the database apart and reassemble it as an organization-wide dataflow around one ordered, replayable event log.
② The target is the dual write: two systems each deciding their own order, so one race or partial failure leaves them permanently divergent, with no self-healing.
③ The fix is a designated system of record plus derived data: order is decided once on the log and everyone else just subscribes and copies. Against distributed transactions, it trades "mutual exclusion + atomic commit" for "deterministic recomputation + idempotent retries," buying asynchrony and fault tolerance.
④ Batch and stream are one thing: Lambda's disease is writing the same logic twice; unify onto one replayable log and reprocessing becomes the proper way to evolve an application — build a new view, validate in parallel, cut traffic, roll back if needed.
⑤ Unbundling: a database's replication log / triggers / index maintenance map onto an organization's Kafka / stream processors / derived stores. Two routes exist — federation unifies reads, unbundling unifies writes. And the boundary between write path and read path is movable: cache, index and materialized view are three positions of one boundary — the further toward writes, the faster the reads, the heavier the writes, the more visible the lag.
⑥ Correctness is end-to-end: the only thing that delivers "exactly once" is a client-generated operation ID carried the whole way; middle-layer de-duplication (TCP, transactions, 2PC) is only an optimization.
⑦ The most valuable distinction: timeliness (self-healing) vs integrity (permanent inconsistency). Dataflow systems unbundle what ACID ties together — hold integrity, relax timeliness, and thereby avoid expensive coordination.
⑧ Not every constraint must be enforced on the spot: allow it, reconcile later, compensate and apologize often beats strong consistency. And "trust, but verify" — audit by replaying the raw events.
⑨ The closing section is ethics: predictive analytics hardens bias, feedback loops reinforce themselves, and user data is a liability rather than an asset — delete it once it is no longer needed.