CS PAPERS DEEP-READ · PAPER 58
Kreps, Narkhede & Rao · LinkedIn · NetDB 2011
In 2011 three engineers at LinkedIn published a seven-page paper describing something they had built themselves: Kafka. It is a data highway running through the company: everything the servers notice (who clicked what, who searched for what, which machine is running hot) gets tossed onto that highway, and whoever needs it picks it up. Today, from the status of your online order to a bank's real-time fraud checks, there is a good chance Kafka or an imitator is behind it.
A sizeable website produces far more activity records — every click, refresh, and swipe — than it does "real" data like orders and accounts: orders of magnitude more. Back then there were two ways to handle it.
One was the traditional message queue, a fussy post office: every letter registered, signed for, logged as collected, then destroyed. Reliable, but the bookkeeping alone cost too much to survive that volume. The other was the log hauler: each machine piles its log files up, and on a schedule they get packed off to a big warehouse to be crunched together — the volume is fine, but a round trip takes hours, and by the time you work out "this person might want to see that," the person is long gone.
What was fast couldn't scale; what scaled wasn't fast.
Kafka goes the other way: the server stops keeping anyone's books.
It writes messages into a ledger that only ever grows — each new message becomes one more line at the end, never edited, never inserted in the middle. Whoever wants to read it remembers which line they got to and picks up from there next time. Nothing disappears once read; whole sections are thrown out only when they age out (say, after seven days).
So the self-evident rule that "a message can only be taken once" is gone: search reads it, recommendations read it, reporting reads it at midnight, each with its own bookmark, none disturbing the others. A bug in your program? Wind your bookmark back a few pages and read it again.
One: it only appends at the end. What a disk hates is hunting all over for a spot; what it does best is writing straight on.
Two: the server keeps no books. "Who has read up to where" was the most annoying ledger of all — a status per message, plus an index to look them up. Handed to the readers themselves, the server has just two jobs left: append, and fetch by line number.
Three: fetching takes no detour. A reader says "give me a stretch starting at line N," and the server pushes those bytes straight from the file onto the wire, without first hauling them through its own memory.
Four: one ledger becomes many. The ledger for a topic is split into several books kept on different machines, so they can be written and read at the same time.
One honest cost: in the paper's Kafka each message is stored only once, so if that machine's disk dies outright, anything not yet read is gone forever (keeping extra copies came later).
The "log" went from an unloved by-product to the company's main data artery: one stream sits there, anyone who wants it just taps in, and adding a new system no longer means disturbing the ones upstream. Today almost anything that talks about "real-time data" or "event-driven" architecture is built on this shape.
Turn the message hub from "a post office keeping everyone's books" into "a ledger that only grows and expires on a schedule": the server doesn't track who read what, readers hold their own bookmarks — so it is fast, many parties can read the same stream, and anyone can rewind and replay.
Want the architecture diagram, the offset and zero-copy mechanics, and the measured numbers? → Switch to the deep read
Kafka redesigns the messaging system as a distributed append-only log: messages in a topic are split across partitions, and each partition is simply a file that is appended to at the end; the server (broker) records nothing about who has consumed what and does not delete a message because it was read (whole segments are dropped on a time schedule instead), with progress held by the consumer. This "stateless broker + sequential I/O + pull" combination lets ordinary machines take hundreds of thousands of writes per second, while the same data is consumed by many parties at their own pace — and can be rewound and replayed.
The authors are Jay Kreps, Neha Narkhede, and Jun Rao of LinkedIn, and the paper appeared at NetDB'11 (a small workshop in Athens co-located with SIGMOD in June 2011), running only seven pages. It says outright that it combines the strengths of two existing lines: enterprise messaging systems (IBM WebSphere MQ, the JMS spec and implementations such as ActiveMQ and RabbitMQ) and log aggregators (Facebook's Scribe, Cloudera's Flume, Yahoo's Data Highway). Kafka was open-sourced in 2011 and became an Apache top-level project the following year; the three authors later founded Confluent.
Internet companies generate enormous volumes of log data: logins, pageviews, clicks, searches, plus operational metrics like call latency and CPU. The shift the paper points to is that this data used to be raw material for after-the-fact analysis, and is now fed directly into live product features: search relevance, recommendations, ad targeting, anti-abuse, the newsfeed. And it is orders of magnitude larger than "real" data such as orders and accounts: computing click-through rates alone means recording a row for every item on the page that was not clicked.
So the requirement became "handle the volume and be usable within seconds," and each existing class of system had only half of it. Enterprise messaging systems are feature-heavy (per-message acknowledgment and similar strong guarantees are a luxury for logs where losing a few pageview events is fine), do not treat throughput as the primary constraint (JMS has no API for a producer to batch, so every message costs a full TCP round trip), are weak on distribution, and assume near-immediate consumption — performance degrades badly once messages accumulate. Log aggregators, built for offline use, periodically dump into HDFS or a warehouse and are therefore inherently hourly; most also use a push model, which floods a slow consumer.
There are only four concepts: a stream of messages of one kind is a topic; producers publish to a topic; messages live on a set of servers called brokers; consumers subscribe to a topic and pull messages from it. To spread load, a topic is divided into multiple partitions and each broker stores some of them.
Physically a partition is a set of roughly equal-sized segment files (1GB each, say). When a producer sends a message the broker does exactly one thing: append it to the last segment file. Files are flushed only after a configurable number of messages or amount of time, and a message becomes visible to consumers only once flushed.
Here is the most unconventional cut: a Kafka message has no explicit ID; it is addressed by its logical offset in the log. That removes the auxiliary, seek-intensive random-access index mapping IDs to locations that traditional messaging systems maintain. The price is that offsets are increasing but not consecutive (the next one = current offset + the length of the current message — it is essentially a byte position).
So how is a position found? The broker keeps just one very small sorted list in memory: the offset of the first message in each segment file. Given a request for "up to N bytes starting at offset X," it finds which segment X falls in and reads on from there.
The most counterintuitive cut in the paper: the broker records nothing about how far any consumer has read. That strips a great deal of complexity and overhead from it — no delivery state per message, and so no index or disk writes to maintain that state.
But then the broker does not know whether all subscribers have read a message, so when may it delete? The answer is blunt: retain by time — anything older than a set period (typically 7 days, the paper says) is deleted whether or not anyone read it. The justification is that consumers are either real-time or hourly/daily, and since Kafka's performance does not degrade with data size, long retention is affordable.
The cut also yields a bonus: a consumer can deliberately wind its offset back and re-consume. This violates the usual contract of a queue, yet proves essential — when consumer-side logic is wrong, you fix it and replay. The paper notes explicitly that rewinding is easy in a pull model and hard in a push model; and pull is a deliberate stance: a consumer retrieves at the maximum rate it can sustain and never gets flooded.
① No message cache in the process — live off the OS page cache. Kafka is written for the JVM and yet caches no messages at the application layer: no double buffering, the cache stays warm across a broker restart, and with almost no message objects in the process, garbage-collection overhead is tiny. Better still, producer and consumer both access the segment files sequentially, with the consumer usually only slightly behind, which plays directly into the OS's write-through caching and read-ahead. The authors report production and consumption performance linear in data size, up to many terabytes.
② Use sendfile to save two copies and one system call. The usual way to send a local file to a remote socket takes four steps: disk → page cache, page cache → application buffer, application buffer → socket buffer, then out to the NIC — 4 data copies and 2 system calls. sendfile transfers bytes from a file channel straight into a socket channel, avoiding 2 of those copies and 1 system call. It works only because Kafka never has to alter a message in transit: the format it stores is the format it sends.
③ Batch. A producer sends a set of messages per request, and each consumer pull retrieves a batch (typically hundreds of KB), so the fixed RPC cost is amortized.
Consumer groups: within a group each message goes to exactly one member (point-to-point); different groups each get the full set independently (publish/subscribe), needing no coordination between them. Two decisions are worth remembering:
The paper is candid: Kafka in general guarantees only at-least-once delivery — exactly-once would typically need two-phase commit, which the authors considered unnecessary for their applications. When a consumer crashes without a clean shutdown, whoever takes over its partitions may re-read a short stretch (messages consumed but whose offset was not yet committed to ZooKeeper), producing duplicates; applications that care must de-duplicate by offset or by a unique key. On ordering: in-order within a partition, no guarantee across partitions. Each message also carries a CRC, and a broker hitting an I/O error runs a recovery pass that removes messages whose CRC no longer matches.
In production. Each user-facing datacenter has its own Kafka cluster; a separate cluster in the analysis datacenter runs embedded consumers that pull the live data across, and load jobs feed it into Hadoop and the data warehouse. End-to-end the whole pipeline averages about 10 seconds (the paper notes this is "without too much tuning" and good enough); the volume at the time was hundreds of gigabytes and close to a billion messages per day.
The comparison. The competitors were ActiveMQ v5.4 (with its default KahaDB store) and RabbitMQ v2.4, known for performance. Two Linux machines, each with 8 2GHz cores, 16GB of memory, and 6 disks in RAID 10, on a 1Gb link; one was the broker, the other the producer or consumer, and all systems were set to flush asynchronously.
The authors add a caveat of their own: the point of the experiment is not that the other systems are inferior — both have more features than Kafka — but to show how much performance a specialized system can buy.
Kafka's real contribution is not "a faster queue" but promoting the log to a first-class system abstraction. Once messages stop vanishing on consumption and progress is held by the consumer, the line between "queue" and "storage" blurs: one ordered stream of facts, read at the tail by live services, read half a day late by the warehouse, and replayed from the beginning by a newly launched system. That untangles the worst knot in a large company — N data sources each wiring up to M downstream consumers now all feed one artery first. The two loose ends in the paper became its main battlegrounds afterwards: cross-broker replication supplied durability, and stream processing grew into Kafka Streams.
① In one line: redesign messaging as "partitions + an append-only log," where the broker records no consumption progress, deletes nothing on read, and the consumer holds its own offset.
② The pain: activity logs are orders of magnitude larger than business data; enterprise messaging could not take the volume (per-message acks, no batching, weak distribution, degradation under backlog), and log aggregators were hourly and offline.
③ Storage: a partition is a chain of ~1GB segment files appended at the end; no message ID — addressing is by byte-position offset, which removes the random-access index, and the broker keeps only a tiny "first offset per segment" table in memory.
④ Stateless broker: progress goes to the consumer and deletion becomes expiry by time (typically 7 days); the bonus is rewind-and-replay, which only a pull model makes easy.
⑤ Where the speed comes from: no application-level cache, living off the page cache; sendfile zero copy; batching on both send and fetch. Coordination uses consumer groups, with the partition as the smallest unit of parallelism and no master — consumers rebalance among themselves via ZooKeeper.
⑥ Results: a single producer at 50,000 msg/sec (batch 1) and 400,000 (batch 50), consumption at 22,000 msg/sec (over 4× the competitors); 9 bytes of per-message overhead versus ActiveMQ's 144. At LinkedIn: hundreds of GB and close to a billion messages a day, end to end in about 10 seconds.
⑦ Guarantees and limits: at-least-once, ordered within a partition; the paper's version has no replication, the producer does not wait for acks, expiry-based deletion loses data for slow consumers, and ZooKeeper-based rebalancing is a source of churn.
⑧ Impact: it turned the log into the central abstraction of a company's data pipeline, and became the de facto foundation of stream processing and event-driven architecture.