BOOK DEEP READ · System Design Interview Vol 2 · Ch 4
System Design Interview Vol 2 · Ch 4 · Alex Xu & Sahn Lam · 2022
You tap "Place order" in a food delivery app. In that one second, the restaurant has to get the ticket, dispatch has to start looking for a courier, inventory has to come down, the coupon has to be marked used, fraud detection wants a look, and the analytics dashboard wants a tally. This chapter is about the pipe in the middle — and it teaches you to build one yourself.
This thing is famous for being fast, and yet it writes every single message onto a disk rather than keeping it in memory. Stranger still, it outperforms plenty of designs that never touch a disk at all.
Could you skip the pipe? The order service could just call everyone in turn: call the restaurant, call dispatch, call inventory, call fraud. Three problems, stacked.
First, the order service has to know every department, so each new department means editing the order code again. Second, fraud detection can handle five thousand things a second, and at the start of a big sale fifty thousand orders arrive each second — calling it does no good and drags the order service down too. Third, if the fraud machine restarts, those few minutes of orders are never checked by anyone, because a shout, once shouted, is gone.
What the pipe really is, is not a megaphone but a ledger. The order service calls nobody; it just adds a line to the last page of the book. Anyone who wants to know what happened comes and reads the book, and keeps track themselves of which line they got to.
All three problems dissolve. The order service knows the book, not the departments. Fraud detection can read slowly; the book will not hurry it. And when fraud detection restarts, it picks up from the line it noted down — the record is still there, so nothing is ever "missed and gone". A new department is easier still: it starts at line one and disturbs no one.
Because it only ever writes to the last page. A disk is slow at flipping back and forth hunting for a page; it is not slow at adding to the end. A ledger that only grows and never gets edited happens to avoid the one thing disks are worst at — and that is the entire secret behind daring to use a disk at all.
One ledger has one person writing on its last page, so however fast they are, there is a ceiling. The answer is several books, split by order number: numbers ending 0 to 3 go in the first book, 4 to 6 in the second, and several people write at once. The cost lands right there: within one book the order of events is perfectly reliable, but across books, who came first is anyone's guess.
A message queue is not a megaphone. It is a ledger that only grows at the end, where each reader tracks their own place. It dares to write to disk because it only writes at the end, and it scales because one book becomes several — and the moment you split it, all you still hold is the half-promise that each book is internally ordered.
Want the disk layout, the replication protocol and the delivery semantics? → Switch to the deep read
You think the hard part of a message queue is delivering messages reliably. In fact this chapter is, start to finish, solving a storage problem: once you decide to land messages as a log on disk that can only be appended to and never modified, everything else — partitions, replicas, consumer offsets, replay, even "exactly once" — grows out of that single decision.
Chapter 4 of Volume 2 is the book's first "build the wheel" chapter: the first three assembled existing components, and from here on you construct infrastructure yourself. It is also the floor the rest stands on — Chapter 5 on monitoring and Chapter 6 on ad click aggregation both assume this pipe already exists. The real-world counterpart is Apache Kafka, along with Pulsar and RocketMQ.
Make the scenario concrete. One "order placed" event in an e-commerce system has seven downstream departments that need to know: the merchant, dispatch, inventory, coupons, fraud, the recommendation model, and the warehouse. If the order service calls each of them synchronously, three things bite immediately:
Coupling: seven downstream addresses are hardcoded into the order service, so an eighth department means editing, testing and shipping it again. Mismatched speeds: peak order volume in a big sale is 50k QPS while the fraud engine can absorb at most 5k QPS. Contagious failure: any downstream whose response time goes from 20 ms to 2 s will fill the order service's 200 threads within seconds, so "fraud got slow" becomes "nobody can order".
So would a traditional broker (RabbitMQ, ActiveMQ) do? It fixes those three but leaves two more. Their semantics are deliver and delete: once a message is consumed it vanishes from the queue. That blocks two increasingly common needs — the same message being read independently by several unrelated consumers (the warehouse and fraud both want the same orders, and neither should "eat" them), and re-running last week's data after a bug. What this chapter builds is a queue that keeps its messages.
The path of least effort is to use a database table as the queue: insert to write, read a sorted batch, delete when done. At a few hundred a second this works fine; at tens of thousands it collapses, and for good reasons. Writes maintain a B-tree index (random I/O), and deletes leave holes in pages — and a queue table is precisely the worst case of frantic inserting plus frantic deleting.
The chapter's answer is an append-only log: an ordinary disk file that does one thing, add to the end. The file is cut into fixed-size segments; only the newest is writable, older ones are read-only and, when they expire, are deleted whole (never row by row, so there are no holes to reclaim).
The whole bargain rides on that word "sequential". The Kafka documentation cites a classic measurement: on an array of 7200 rpm SATA disks, sequential writes run at roughly 600 MB/s while random writes manage only about 100 KB/s — more than three orders of magnitude apart. In other words disks are not slow; random access is, and a log that only grows at its end sidesteps random access completely. Layer the operating system's page cache on top and a freshly written message is often collected by a consumer before it ever reaches a platter.
Table 1 · A database table as a queue vs an append-only log
| Database table | Append-only log (this chapter) | |
|---|---|---|
| Write | Insert plus index maintenance — random I/O | Append at the end — sequential I/O, whole batches at once |
| Delete | Row deletes leave holes; needs vacuum | Whole segment dropped, about as cheap as rm |
| Re-reading | You must build your own "who read what" table | One offset per group; replay comes free |
| Fits | Low volume, query by arbitrary fields | No editing single rows; high throughput, many subscribers |
A log has exactly one write point at its end, so throughput has a natural ceiling. Partitioning cuts a topic into N independent logs spread across brokers, each written separately — from then on you scale by adding partitions and brokers, not by buying a bigger machine.
The cost is due immediately: offset is monotonic only within a partition, and across partitions there is no global order at all. The promise of "ordered" is downgraded to "ordered within a partition". The standard practice is to use a business key as the partition key — every event for one order number hashes to the same partition, so you keep "this order's events are in order" and give up "the absolute sequence of every order on the site", which you never needed.
The consumer side mirrors this: within one group, a partition goes to exactly one consumer. That gives you both freedom from duplicate consumption inside the group and a hard ceiling — consumers beyond the partition count simply sit idle. So partition count is a parallelism budget you commit to in advance. A common rule of thumb is to plan around 10 MB/s per partition, which puts 1 GB/s at roughly 100 partitions.
More than half of the throughput in systems like this comes not from the storage structure but from batching across all three legs of the path. The producer accumulates messages in a local buffer and sends once a batch is full or a short wait has elapsed (linger.ms). The broker writes that batch to disk verbatim, neither unpacking nor re-encoding it. The consumer fetches in batches too. One network round trip, one system call, one disk append — amortised over hundreds or thousands of messages.
Hidden in there is a dial you have to turn yourself: moving linger.ms from 0 to 10 costs every message up to 10 extra milliseconds and buys bigger batches and possibly several times the throughput. Low latency and high throughput are directly opposed here; there is no having both.
Writing batches verbatim also pays a bonus. Since the bytes on disk are identical to the bytes owed to the consumer, the broker can have the kernel send page-cache data straight to the network card without passing through user space — that is zero-copy. It is not a standalone trick but a by-product of the decision not to unpack.
Does the broker push to consumers, or do consumers pull? This chapter picks pull, and the one-line reason is that push forces the broker to care about something it should not have to: how fast each consumer can digest. Hand pacing back to the consumer and batching and replay both come free (see Table 2).
Each partition has one leader and several followers, and all reads and writes go through the leader; a follower does one thing only, which is to keep pulling from the leader and appending the very same log. The elegance is there: replication reuses the log itself and needs no separate sync protocol. The followers keeping up form the ISR; one that falls too far behind is temporarily evicted. The producer's acks setting defines what counts as a successful write: acks=0 means fire and forget, acks=1 means the leader wrote it (and if it dies right then, whatever had not replicated is gone), acks=all means every ISR member confirmed.
One more mechanism gets skipped often and asked about in interviews constantly: the high watermark — the largest offset every ISR member already holds, and the furthest a consumer is allowed to read. The point is that consumers never see the messages that only the leader has and that could vanish along with it, so after a failover there are no ghost messages that you demonstrably read and that no longer exist.
These two kinds of state have opposite characters. Consumer offsets are written constantly — every consumer may commit once a second. Kafka originally kept them in ZooKeeper, quickly found that a coordination service cannot absorb that write rate, and moved them into an internal topic on the brokers themselves: using a log to store a log's progress, which conveniently reuses the whole replication and durability machinery. Metadata — which topics exist, how partitions are distributed, who leads what — is written rarely but must be strongly consistent and needs the ability to elect a controller. That is exactly the job of ZooKeeper or etcd.
Last comes rebalancing: when a consumer joins, leaves or crashes, or the partition count changes, the coordinator must reassign who reads which partition. The classic implementation is stop-the-world — the whole group halts, reassigns, then resumes. With a large group and many partitions that can stall for seconds to tens of seconds, which is the most tangible bill the consumer-group model presents.
Table 2 · Push vs pull
| Push | Pull (this chapter's choice) | |
|---|---|---|
| Who sets pace | The broker — it must know each consumer's capacity | The consumer — the broker need not care |
| Slow consumer | Gets overwhelmed; you must add back pressure | Simply reads slower; nothing falls over |
| Batching / replay | Broker has to guess the batch size; replay is hard, it does not track readers | Take what you ask for; rewind an offset to replay |
| Cost / examples | Low latency, messages leave on arrival. RabbitMQ push mode | Wasteful empty polls, fixed by long polling. Kafka, Pulsar, RocketMQ |
Table 3 · Three delivery semantics: how, at what cost, and when
| At most once | At least once | Exactly once | |
|---|---|---|---|
| How | acks=0, no retry; commit the offset before processing | acks=all plus retries; commit the offset after processing | Idempotent producer ids plus transactions binding "write result" and "commit offset" into one |
| What happens | One network blip loses data, and you never find out | Duplicates: crash before committing and the batch arrives again | Neither lost nor duplicated — but only inside the system boundary |
| Cost | Near zero | Downstream must be idempotent itself | Lower throughput, more complexity |
| Choose it for | Logs, metrics, telemetry | Most business traffic: orders, notifications, syncs | Ad billing, financial ledgers |
Ordering versus scale is the other place you must declare yourself. Want a topic ordered globally? Then it gets one partition, and throughput is capped at whatever a single write point sustains — tens of MB/s. What almost every situation should do instead is shrink the requirement: partition by business key and guarantee only that one entity's events are ordered. Asked in an interview how you guarantee message ordering, the right answer is not a setting but a question back: ordered between whom and whom?
Retention trades disk for the right to change your mind: keep one day and a bug can only be re-run for today; keep seven and a week of mistakes is recoverable, at seven times the disk. If all you care about is the latest value per key, there is also log compaction: keep only the last record for each key, and the log becomes a snapshot table you can retain indefinitely.
Apache Kafka maps onto this chapter's design almost line for line. Two other directions are worth knowing: Apache Pulsar separates brokers from storage (Apache BookKeeper), so scaling out moves no data, and RocketMQ adds transactional and scheduled messages. AWS SQS is another branch entirely — a genuine consume-and-delete queue with no replay and no partition ordering, in exchange for no operational work at all. The first question in any selection is always: do you need to keep the messages?
The interview value of this chapter is that it rarely fails to connect storage structure to scalability to consistency semantics in one line of reasoning. Asked "why is Kafka fast", answering "sequential writes and zero-copy" is reciting. Being able to say "zero-copy is possible because the disk format equals the wire format" and "ordering is possible because a partition has a single write point" is understanding.
0.01% — evidence pointing the other way: at scale, reliability is bought, so ask the price first. Netflix Tech Blog, "Kafka Inside Keystone Pipeline", 2016 ↗① Every choice in the chapter is reverse-engineered from one storage decision — land messages as an append-only log; partitions, replicas, offsets and replay are all corollaries.
② Skipping the queue costs you three things: coupling, mismatched speeds, contagious failure (slow downstream → thread pool full → nobody can order).
③ Don't use a database as a queue: writes are random I/O, deletes leave holes, and a queue is the worst case of both.
④ Disk is viable because it only ever appends: ~600 MB/s sequential vs ~100 KB/s random — disks aren't slow, random access is.
⑤ Partitions are the unit of scale and the boundary of the ordering guarantee; one partition per consumer in a group means partition count caps parallelism.
⑥ Throughput is accumulated by batching; linger.ms is the latency-for-throughput dial. Zero-copy is a by-product of "disk format = wire format".
⑦ Pull over push: the consumer sets its own pace, and batching and replay come free.
⑧ Replication reuses the same log; acks picks the durability tier; the high watermark ensures nothing a consumer read vanishes in a failover.
⑨ Offsets are written constantly → an internal topic; metadata must be strongly consistent → a consensus component. Rebalancing is stop-the-world.
⑩ Pick delivery semantics by business: at-most-once for telemetry, at-least-once plus your own idempotency for most things, and only billing and financial ledgers justify paying for exactly-once.