CHAPTER DEEP READ · System Design Interview Vol 2 · Ch 11
System Design Interview Vol 2 · Ch 11 · Alex Xu & Sahn Lam · 2022
You tap "Pay now" in a shopping app, and a second later the page says "payment successful." This chapter builds what runs inside that second: it takes money from the buyer, and one day hands money to the seller — not a penny more, not a penny less.
This system is remarkably slow. A million transactions a day works out to about a dozen per second — less traffic than a mid-sized forum. And yet it is the hardest chapter in the book to get right.
Because the difficulty was never speed. It is that "did the money move?" must have a definite answer.
You press pay, the phone spins for thirty seconds, and a line appears: network error. Were you charged? You don't know. Neither does the server. The request may never have arrived — or the bank may have taken the money and only the receipt got lost on the way back.
The system now has two options. Treat it as failed, and the customer may be charged for nothing. Treat it as succeeded, and the merchant may ship for nothing. Worse, the customer will press the button again — and one purchase becomes two charges.
① Give every payment a ticket number. However many times the same ticket is handed over the counter, the clerk does the work once and simply copies out the first receipt again. "I'm not sure it went through, so let me try again" turns from a dangerous act into a safe one.
② Write every amount down twice. Every movement of money lands on two accounts at once, one side down and the other up, and the whole book always sums to zero. Miss an entry and the sum gives it away — money is conserved by the shape of the bookkeeping, not by the care of the bookkeeper.
③ Compare books every night. Your record of the day, line by line, against the record the payment company sends you. Matches pass. Mismatches get pulled out: the ones that are just a fee can be fixed by rule, and whatever is left goes to a human. This step is not insurance — it is a necessity, because the first two tricks never cover everything.
④ Card numbers never enter your house. The card form belongs to the payment company; the number goes straight into their machines and you get back a stand-in code. What you cannot store, you cannot leak.
All of this turns "paying" from a single act into a process with states: it can sit in "processing," it can land in a retry queue, and every day it leaves behind a small tail that a human must settle by hand — that inconvenience is the price of being correct.
A payment system is absurdly slow and absurdly hard, because its only job is to make "did the money move?" have a definite answer. A timeout tells you neither success nor failure, so every payment carries a ticket number that is honoured once, which makes retrying safe again; every amount is written on two sides, so a missing entry cannot hide; and every night the books are compared against the payment company's, so whatever the first two missed gets pulled out and fixed by hand. The card number, meanwhile, is never let into your machines at all.
Want the mechanisms, the notation and the diagrams? → Switch to the deep read
On the surface this chapter wires up a payment. What it actually teaches is how to make "did the money move?" have exactly one answer, inside an outside world you do not control and cannot rerun. First counter-intuition: this is the lowest-throughput system in the book — a million transactions a day is roughly 11.6 TPS, which an ordinary relational database handles in its sleep. Not one design decision here is about performance. Second: correctness is not bought by being careful, it is bought in three stacked layers — idempotency keys make retries safe, a double-entry ledger makes money conserved and auditable, and daily reconciliation concedes outright that the first two still leak, so the books get counted again every night.
Chapter 11 of Volume 2, and the first half of the book's "money must not be wrong" thread: the next chapter, Digital Wallet, pushes this chapter's ledger all the way to event sourcing, CQRS and a Raft-replicated state machine, while Ch7 (Hotel Reservation) is the other face of the same family — concurrent contention over finite inventory. In the real world this is e-commerce checkout, the charge at the end of a ride, a SaaS subscription renewal, and anything that calls an outside institution where money genuinely moves.
Put the scale on the table first: a million transactions a day is 1,000,000 ÷ 86,400 ≈ 11.6 TPS. In this book that number is nearly comic — Ch4's rate limiter fields a million QPS, Ch10's leaderboard peaks at 2,500 QPS, and the payment system runs in double digits. Every reflex about caching, sharding and going async fails here at once: there is no performance problem to solve.
The real problem is singular, and it comes from outside your boundary. A payment crosses five doors — your service → PSP → acquiring bank → card scheme → issuing bank — and you own the first. When the call times out, a timeout is neither success nor failure. It is a third state: unknown. And money hates nothing more than unknown.
What happens if you don't solve it? All three of these: ① double charges — one retry, two debits; ② lost bookkeeping — the PSP charged the card, your ledger never recorded it, the seller never gets paid; ③ month-end doesn't balance — the gap is real, but nobody can say which transactions it is made of. The first two hurt users; the third hurts the company: a payment system that cannot explain its own discrepancies is, to finance and to regulators, not a system at all.
The first decision people skip is hiding in the data model: "a checkout" and "a payment" are not the same object. Three items in a cart may come from three sellers, so one checkout (a payment event, keyed by checkout_id) splits into three payment orders (keyed by payment_order_id), each making its own PSP call and each succeeding or failing alone.
Why not bundle them into one charge? Because they are owed to different payees, and you have no transaction that spans the PSP — when the second fails, the first has already landed at a bank. There is no rollback, only compensation.
The division of labour is worth memorising too: the payment service receives the payment event and runs risk and anti-money-laundering checks; the payment executor executes a single payment order against the PSP; only after that do the ledger record entries and the wallet update the seller's balance. The two unremarkable columns on the payment order — ledger_updated and wallet_updated — are the most engineering-minded detail in the chapter: they turn "did this step happen?" into queryable state, so an order stuck halfway can be swept up by a program instead of guessed at by a person.
This is the heart of the chapter. No network gives you exactly-once: don't retry and you may lose the payment, retry and you may duplicate it. So in engineering, exactly-once = at-least-once (retry) + idempotency (a retry has no new effect).
The mechanism: the caller generates a UUID per payment and puts it in an HTTP header (Idempotency-Key: 8f3c…); the server, before doing anything real, inserts that key atomically under a database unique constraint. If the insert succeeds you are first and may proceed; if it fails, someone is already doing it or has done it — return the first result, or reply 409 and let the caller ask again shortly.
Three details decide whether this works, and each is an interview watershed:
It also blocks an attack for free: with a nonce inside the key, a replay attack fails — resending the identical request is simply treated as a retry.
This section is the art of not doing something. The moment a card number passes through your process, your logs or your database, the whole system falls inside PCI DSS scope. So the standard answer is to outsource the collection step entirely to the PSP: its hosted checkout (a full-page redirect) or its embedded component (iframe / JS SDK), with the card going from the browser straight to the PSP and only a token coming back to you — valid solely under your merchant account, useless to a thief elsewhere. What you cannot store, you cannot leak.
The cost is that the flow becomes asynchronous: the user completes payment on the PSP's page (possibly via a 3DS challenge) and the result arrives by webhook. So you must handle the webhook landing before the user gets redirected back, and you must verify the callback signature — otherwise anyone can post "payment succeeded" to your endpoint.
The ledger uses double-entry bookkeeping: every transaction writes two rows, one debit and one credit of equal amount, and the whole book always sums to zero. The buyer's account down 100, the platform's payable-to-seller account up 100 — both written together.
It looks like an accountant's old habit, but it is a hard engineering property: it turns "is anything missing?" into a question a program can answer — a non-zero sum means something broke, and the entries lead you to which transaction it was. Keep only a balance table instead and a missed update stays silent forever.
Which yields a second rule: the wallet is not the source of truth; the ledger is. A balance is a derived view that can be recomputed — the same principle as Ch10's in-memory leaderboard being a rebuildable derived view.
The payment service, the executor, the ledger and the wallet are four services, and the PSP is outside your company entirely. No transaction on this path can wrap all of them. The two textbook options fare badly: two-phase commit (2PC) has participants vote and then commit together, and a dead coordinator can leave them holding locks — while the PSP will never take part in your vote; Saga runs local transactions in order and compensates in reverse on failure (refund, reversal), at the price of hand-written compensation and intermediate states that are visible to users.
So the book's default is the third path: make every step idempotent, retry on failure, and leave the rest to reconciliation. Failure follows a standard pipeline: the payment order goes into a retry queue (exponential backoff with jitter, so clients don't retry in lockstep and flatten the PSP), and whatever still fails lands in a dead letter queue for a human. Meanwhile the PSP's daily settlement file is compared to the ledger line by line: matches pass; classifiable differences (fees, cut-off timing, FX) generate adjusting entries by rule; unclassifiable ones go to a manual queue — that last handful always exists, and admitting it and giving it a proper channel is the most mature thing in this chapter.
Table 1 · How to handle card data: compliance scope differs by an order of magnitude
| Approach | What it buys | What it costs (honestly) | When to pick it |
|---|---|---|---|
| PSP hosted checkout (full-page redirect) | Smallest compliance scope: the card never touches you; 3DS and local payment methods are the PSP's problem | The user leaves your domain — conversion and brand experience suffer | The default whenever payments are not your core competency |
| PSP embedded component (iframe / JS SDK) | Visually stays on your page, better conversion; the card still goes straight to the PSP | Harder integration; the script must load from the PSP's domain and cannot be bundled into your build | The realistic choice for most commerce |
| Collect and store cards yourself | Full control of the flow; routing across PSPs and negotiating leverage | The whole system lands in the strictest PCI DSS scope: audits, isolation, key management and breach liability are all yours | Large enough that acquiring in-house pays, with a dedicated compliance team |
Table 2 · Consistency across services and across companies: three roads
| Option | What it guarantees | What it costs (honestly) | When it applies |
|---|---|---|---|
| Two-phase commit (2PC) | All commit or all roll back — the strongest semantics | A failed coordinator can strand participants holding locks; unusable across companies — the PSP will not join your vote | A few strongly consistent cases inside one trust domain |
| Saga (compensation) | Local transactions in order, compensated in reverse on failure | Compensation logic is yours to write; intermediate states are visible — a user may see "charged but order not active" | Long flows across services where a legitimate compensating action exists |
| Idempotency + retry + reconciliation (the book's default) | Eventual consistency where differences are detectable and explainable | A window of inconsistency; you must run a reconciliation pipeline and staff the manual queue | The default for payments — get here first, then argue about distributed transactions |
Table 3 · How a payment that is unknown or failed gets settled
| Mechanism | Fits | What it costs (honestly) |
|---|---|---|
| Synchronous retry (backoff + jitter) | Refused connections, 5xx, timeouts — the "might not have arrived" errors | Without an idempotency key this is the direct cause of double charges; backoff without jitter makes every client hit the PSP together |
| Asynchronous retry queue | The PSP is down, or the answer is minutes to hours away | The user sees "processing" — product and support need language for that state |
| Dead letter queue + human | Retries exhausted, or a business error that needs judgement | Pure labour; it must leave an audit trail, or the corrections become tomorrow's discrepancies |
| Leave it to reconciliation | Orphan transactions the PSP has and you have no record of at all | Slowest to fix (next day) — but it is the only net that catches unknown unknowns |
The transfer value goes far beyond payments: anything that calls an external system with an irreversible side effect is the same question — sending an SMS, issuing an invoice, dispatching a shipment, wiring money to a third party, deducting credits from a model API. And the test is always the same: ask "what if this happens twice?" — if the answer is unacceptable, you need an idempotency key; then ask "how would I notice one went missing?" — if you cannot answer, you need reconciliation.
maxNetworkRetries config option to 0. You can also set a higher number to reattempt multiple times, with exponential backoff. Idempotency keys are added where appropriate to prevent duplication." Note the causality between those two sentences: retries can be on by default precisely because the keys are there. Stripe official stripe-node README · github.com/stripe/stripe-nodestripe-js README · github.com/stripe/stripe-js11.6 TPS needs no performance design at all. This chapter tests how you handle uncertainty.① Payments are the lowest-throughput system in the book: 1M/day ≈ 11.6 TPS. There is no performance problem, only a correctness problem.
② The cause is the control boundary: your service → PSP → acquirer → card scheme → issuer, and you own only the first door.
③ A timeout is not a failure — it is a third state, "unknown." Retry blindly and you double charge; give up and you lose the order.
④ One checkout splits into N payment orders (one per seller), and no transaction can wrap them: compensation exists, rollback does not.
⑤ Exactly-once = at-least-once + idempotency. Claim the key atomically before executing, and store the first response with it.
⑥ Wherever the retry happens, that is where idempotency goes; a key with a nonce blocks replay attacks for free.
⑦ Double-entry makes conservation of money machine-checkable: two rows per transaction, sum zero; the wallet is only a derived view of the ledger.
⑧ Card numbers stay out of your servers: hosted checkout or embedded component plus tokenization, shrinking PCI DSS scope to a minimum.
⑨ No transaction crosses company lines: the default answer is idempotency + retry + daily reconciliation, not 2PC or Saga.
⑩ The failure pipeline: retry queue → dead letter queue → human with an audit trail. A handful of differences will never be explainable; giving them a proper channel is what maturity looks like.