Book Deep Reads · System Design Interview Vol 2 · Chapter 12
System Design Interview Vol 2 · Ch 12 · Alex Xu & Sahn Lam · 2022
You send a friend $1 inside a payments app, or hand a teammate some in-game gold. This chapter is about how that transfer gets written into the books. Its central problem is counter-intuitive: the hard part isn't moving the dollar quickly — it's still being able to explain, a year later, exactly why that dollar left.
Think of a wallet as a paper ledger. There are two ways to keep one.
The first is to write down a single number: you have 100, you send 1, so you erase the 100 and write 99. Compact, instantly readable — and whatever you erased is gone forever.
The second is to only ever add lines: "Mar 1, sent 1" / "Mar 2, received 5" — and simply never write a balance at all. To find out what you have now, you add the lines up. Wordier, but the balance on any past day can be recomputed, and everyone who adds it up gets the same answer.
Where this chapter ends up is: replace the first ledger with the second.
Two things are hard at once.
First, a transfer has to happen "at the same time": A loses the dollar and B gains it, and those two strokes must count as one action. What if the power cuts between them? And the two accounts usually don't even live on the same machine.
Second, money gets questioned. When the books don't balance, when a customer disputes a charge, when a regulator asks — you must be able to say what the balance was at that instant and which entries made it so. A ledger holding one number can't answer that. It remembers the present, not the reason.
The chapter solves this in three moves.
One: make "at the same time" real. It works like reserving a restaurant table — first put a hold on A's dollar (still A's money, but unspendable), then reserve a slot on B's side, and only once both sides agree does it settle. If anyone backs out, release the hold and drop the reservation. Holding a table isn't dinner, but at least two parties never sit down in it at once.
Two: swap "erase and rewrite" for "only add lines." Every entry is a line that is never modified. The balance stops being something you store and becomes something you compute. So you don't re-add from day one every time, you take a "snapshot" now and then and only add the lines after it.
Three: don't lose the ledger. The same ledger is copied to three machines at once, and every line counts as written only after at least two of the three nod. Lose any one machine and the other two hold an identical book — just keep writing.
If what flows through your system is money, or inventory, or anything that will be questioned after the fact, this way of keeping books is worth the trouble. The cost is real too: the ledger only grows, storage grows with it, and a mistake can't be edited — only cancelled out by another line. For a team used to "just fix that row in the database," that's a real adjustment.
Where a digital wallet ends up is this: the balance stops being a number that gets rewritten and becomes the sum of a list of lines that are never rewritten. Precisely because it is computed, it can be recomputed to prove itself — and copied to several machines without drifting.
Want the mechanisms, notation and diagrams? → Switch to the deep read
You think designing a digital wallet means designing a balance table. What this chapter actually does, start to finish, is delete the mutable "balance" field. It starts from a number in Redis and is pushed by three requirements — cross-partition atomicity, 1 million TPS, and reproducible auditing — all the way to its endgame: an immutable event log, a deterministic state machine, Raft replication, and TC/C across shards. What you walk away with isn't a wallet blueprint; it's a stance: correctness comes from replayability, not from trust.
This is the second-to-last chapter of Vol 2, directly after Ch11 "Payment System." Ch11 handles money crossing the company boundary (third-party payment service providers, where a timeout is neither success nor failure); Ch12 turns inward, to a transfer between two accounts inside one system. It also sets up Ch13 "Stock Exchange," which pushes the same "single-threaded deterministic processing plus replay" idea down to microseconds. Real-world counterparts: a PayPal balance, an Alipay balance, a brokerage cash account, an exchange's user balances, in-game currency.
The prompt is one sentence: let wallet A transfer $1 to wallet B. All the difficulty lives in three non-functional requirements — the transfer must be atomic, the system must sustain 1 million TPS, and the books must be reproducible (auditable).
Each cuts something away. Atomicity is the most direct: A and B almost certainly sit in different partitions, and all-or-nothing across machines is one of the most expensive things in distributed systems. The 1M TPS target cuts off "write every transfer to a relational database" — parsing, index maintenance, row locks and the transaction log don't reach that scale.
Reproducibility is the easiest to overlook and the real watershed: if you only store "what the balance is now," then "what was it at 21:00 last Tuesday, and which entries made it so" is unanswerable, forever. Which means no second opinion when reconciliation disagrees, no evidence when a customer disputes, no way to replay history against a fixed rule. With money, "can't explain it" is as fatal as "got it wrong."
The book starts with the obvious design: a stateless wallet service, balances hashed by userId into an in-memory key-value store (a Redis cluster), a transfer being two arithmetic updates. It fails on every requirement: Redis has no cross-node transaction, so the window where A is debited and B isn't is real; in-memory balances vanish on power loss; and it remembers the "now" but not the "why." Start with atomicity — the hardest of the three.
Two-phase commit is the textbook answer: in the prepare phase the coordinator asks every participant "can you commit?", each locks the relevant rows, writes its redo record and answers yes; in the commit phase the coordinator orders commit or abort.
Its two flaws land squarely on the wallet case. First, locks are held for the length of the whole protocol — two network round trips, single-digit milliseconds in one datacenter and tens of milliseconds across them, which on a hot account is enough to cap throughput. Second, if the coordinator dies between the phases, participants hang: locks can't be released, rows can't be touched, nobody moves until it recovers. Clean semantics, far too heavy for a system aiming at 1M TPS.
Try-Confirm/Cancel looks like 2PC, and differs in one sentence: each phase is a local transaction that commits immediately, so locks are released within milliseconds.
The price is that three things the database used to cover are now yours:
And one conceptual cost, a favourite interview follow-up: the state before Confirm is externally visible. A's available balance is already down $1 and B's isn't up yet — for a moment that dollar sits in nobody's account. Serious ledgers model this explicitly as a funds-in-transit account rather than pretending the window doesn't exist.
Saga takes the other road: split the transfer into a sequence of local transactions T1, T2, …, each paired with a compensating transaction C1, C2, …; on failure, run the compensations backwards. Two ways to coordinate — orchestration, where one coordinator issues the steps, so whole-flow state is visible and debuggable; and choreography, where services subscribe to events and react on their own, minimal coupling but no single place that knows the global progress.
The dividing line against TC/C is one sentence: Saga goes all the way and then backs out; TC/C reserves everything and then settles. Once Saga's T1 commits, the money really moved. TC/C pays an extra RPC and the cost of building a status table, and buys "the whole thing can still be cancelled before it settles" plus parallel branches. The book's steer is plain: latency-sensitive with few steps → TC/C; long flows spanning several teams' services → Saga.
Atomicity is handled; reproducibility isn't. Four terms have to land cleanly:
now(), no randomness, no calls to external services inside them, or a replay computes a different answer and the whole scheme collapses.What about reads? If balance queries also hit that one state machine, the write path dies instantly. CQRS moves reads out: a separate set of processes consumes the event stream and materializes read models, and every query hits the model. The cost must be stated: the read side is eventually consistent, so a user may see a stale balance for tens to hundreds of milliseconds after a transfer — which is why the success screen usually echoes what the state machine computed rather than querying the model.
Reproducibility then comes for free: start from a snapshot, feed the later events forward, and you have the balance at that instant and the entries that made it. Reconciliation, dispute evidence, replaying yesterday's real traffic against a new version of the rules — all of it rests on this one property.
One state machine is nowhere near 1M TPS. The book closes the gap in three moves.
Fast — drop the database. An event log needs exactly one operation: append to the end, which is what disks are best at, while a relational database pays for parsing, index maintenance, row locks and its own transaction log to stay general. Appending straight to a local file with state resident in memory buys an order of magnitude per node. A quick sanity check shows the disk isn't the bottleneck: at 100 bytes an event, a million events per second is about 100 MB/s, while an NVMe SSD's sequential write bandwidth is in the GB/s range. Snapshots handle restarts, so recovery doesn't replay from the genesis event.
Survive — one node's memory plus one local file dies with the machine. Use Raft to replicate the state machine across 3 or 5 nodes: the leader ships log entries to followers, a majority durably storing an entry makes it committed; if the leader dies, a follower is elected with an identical log and carries on. That is a replicated state machine — deterministic logic plus an identical input sequence yields an identical state. Event sourcing and Raft are a natural pair, because both are built on the same object: one ordered, immutable log.
Scale — one Raft group is ultimately capped by its leader. Hash wallet IDs into several Raft groups: serial inside a group, fully parallel across them, add groups for more throughput. Which makes a cross-group transfer a distributed transaction again, handled by the same TC/C from earlier — a full circle, where the two threads of the chapter finally meet.
Table 1 · Three ways to move $1 across partitions: who carries the guarantee
| Dimension | 2PC | TC/C | Saga |
|---|---|---|---|
| Implemented by | Database / transaction manager | Your own business code | You, plus a coordinator or event bus |
| Lock hold time | The whole protocol (two round trips) | One local transaction (milliseconds) | One local transaction |
| On failure | Database rolls back, no intermediate state | Cancel compensates | Compensations run in reverse order |
| Intermediate state visible | No | Yes (frozen, not yet credited) | Yes (money really moved) |
| Extra code you write | Almost none | Three interfaces + a phase status table + the idempotency / empty-rollback / hanging traps | A compensation per step + whole-flow state tracking |
| Coordinator dies | Participants block, locks stuck | Participants don't block, they await a retry | No block; the flow just stalls |
| Fits | One database or few partitions, modest throughput | Latency-sensitive, few steps, needs "cancellable before settling" | Long flows across teams' services, seconds of eventual consistency acceptable |
Table 2 · How to store a balance: one mutable field, or a list of immutable events
| Dimension | Balance field | Event sourcing |
|---|---|---|
| Current balance | One primary-key read, fastest | In-memory state or a CQRS model — fast, one hop more |
| Balance at a past instant | Impossible without a separate change log | Snapshot plus replay, native |
| Fixing a mistake | Edit the row — which also means anyone can edit it quietly | Only by appending a reversing event; the trace can't be erased |
| Storage cost | Proportional to accounts | Proportional to transactions, monotonically growing |
| Throughput ceiling | Bounded by random writes and row locks | Sequential append plus memory — an order of magnitude higher |
| Team barrier | Low, everyone knows it | High: event schema evolution, snapshots and replay tooling are all yours to build |
Two sizing questions come up constantly. Replica count: 3 replicas tolerate 1 failure and need 2 acks; 5 tolerate 2 but need 3 and carry worse tail latency — single-datacenter deployments default to 3, and 5 is for spanning datacenters. Shard count: more shards mean more parallelism, but also a higher fraction of cross-shard transfers, and every one of those pays the full TC/C cost. So the shard key should be chosen to keep as many transfers as possible inside one group — real systems often cluster by business domain (same merchant, same region) rather than hashing purely at random.
The endgame here isn't a teaching toy — it is very nearly the standard shape of a modern distributed database: data sharded by key, each shard a consensus-replicated group, cross-shard writes going through a two-phase protocol. TiKV, CockroachDB and YugabyteDB all share that skeleton; the difference is that they built a general database while a wallet builds a single-purpose ledger.
"Immutable log plus deterministic replay" is not wallet-specific either: Kafka turned it into infrastructure, and MySQL's binlog and PostgreSQL's WAL are the same idea — write down what happened, in order, and let state be the derivative. It also explains why crash recovery, replication and change data capture, three apparently unrelated features, all ride on one log.
now(), a random number or an external call inside an event and replay computes something else — and replay is the foundation the whole architecture rests on. Non-deterministic inputs must be frozen into literal values when the event is created.① What the chapter really does is delete the mutable "balance" field, replacing it with immutable events, a deterministic state machine and consensus replication.
② Three requirements set everything: cross-partition atomicity, 1M TPS, reproducible auditing — the third is the one that forces event sourcing.
③ 2PC's two flaws: locks held for the whole protocol, and participants blocking when the coordinator dies.
④ TC/C lifts the two phases into the application with each phase committing at once; the price is handling idempotency, empty rollback and hanging yourself, backed by a phase status table committed with the business change.
⑤ Saga does it all then backs out; TC/C reserves then settles. Latency-sensitive → TC/C; long cross-team flows → Saga.
⑥ The four terms: command (intent, rejectable), event (fact, immutable and necessarily deterministic), state (a derivative), state machine (serial consumption of a FIFO queue). CQRS moves reads out at the cost of an eventually consistent read side.
⑦ Fast comes from sequential appends, in-memory state and snapshots; survival from a Raft replicated state machine; scale from many Raft groups — and cross-group transfers loop back to TC/C.
⑧ "Shard, one consensus group per shard, a two-phase protocol above" is the skeleton of TiKV and CockroachDB; state-derived-from-log also runs through WAL, binlog, Kafka and CDC.
⑨ The honest costs: storage only grows, event schemas must stay compatible forever, immutability collides with deletion rights, and the team barrier is high — without the 1M TPS constraint, a relational database with an audit table is enough.