Stream Processing

In one sentence

This chapter is about stream processing: treat data as a never-ending, unbounded stream of events, process each event as it arrives, and keep derived results (indexes, caches, live metrics, alerts) continuously close to "now." It shares last chapter's worldview—immutable data + derivation—only swapping "one bounded batch" for "an unbounded continuous stream." Two pillars hold up the whole chapter: the log-based message broker (e.g. Kafka)—a durable, ordered, replayable event log; and event sourcing / stream-table duality—taking "every change" rather than "the latest state" as the source of truth. One counterintuitive line: you'd think the hard part of streaming is "going fast"—it's actually time: the mismatch between event time and processing time is the real deep water of this chapter.

Glossary

Where it sits

This chapter is the second piece of Part III, "Derived Data," right after batch processing (Ch10). Ch10 handles bounded, static batches; this chapter turns to unbounded, real-time streams. The two aren't rivals but two faces of one worldview—both treat data as immutable input and derive results from it; the only difference is "batch vs stream." It builds on Ch10's batch processing and cashes in setups from Ch5 (replication logs) and Ch7 (transactions & exactly-once); it leads into the final Ch12, where the author reorganizes the whole data system around streams. It maps to the real-world Kafka / Flink / Spark Streaming ecosystem, real-time warehouses, and fraud and monitoring pipelines.

What problem does it solve?

Suppose you're building fraud detection for e-commerce: 50,000 transactions pour in per second, and between the user tapping "pay" and the bank replying—under 200 milliseconds—you must judge whether this one is fraudulent. Suppose Ops also wants a live dashboard showing this-minute GMV and per-city order heat. What these have in common: the value of the result decays fast with time. "Run tonight, out by morning" batch is useless here—by the time you finish, the money's stolen and the sale is over.

Yet you can't just shrink batch's "recompute all data from scratch every hour" down to "every second"—that would crush your machines and database. The real problem is three interlocking challenges: ① how to deliver an endless flow of events reliably and in order from producer to processor, and, after the processor crashes and restarts, resume from where it left off—or even replay history; ② how to propagate a change in one place (one row edited) in real time and correctly to every derived system (search, cache, reports) so they don't contradict each other; ③ how to give a trustworthy aggregate over "one time window" in a world where events arrive out of order and late. Solve none of these and you're stuck with either hours of latency or a pile of fragile real-time scripts that drop messages and never reconcile.

The core, explained

1. Events & messaging: from "gone once consumed" to "a replayable ledger"

Streaming starts with how to get events from producer to consumer. The naïve way—have consumers poll a database for new data—is clumsy. Dedicated message brokers emerged, historically in two schools; understanding their difference is the key to why Kafka matters:

• Traditional message queues (AMQP / JMS style, e.g. RabbitMQ, ActiveMQ): like shipping a parcel—a message is delivered to a consumer, who signs for it (ack), and then the broker deletes it. To scale throughput, multiple consumers often share a queue (load balancing). The catch: once consumed, a message is gone—no replay; and under concurrent consumption, processing order easily scrambles. It excels at "task dispatch" (each message need only be handled once by some worker, e.g. send an email).
• Log-based brokers (e.g. Kafka, Kinesis): like a newspaper's bound archive—all events are append-only, pinned in order onto one durable log, and not deleted after being read (retained by time / capacity, e.g. 7 days). Each consumer keeps its own offset as a bookmark. This grants two killer features: ①replayable—rewind the offset to re-read history (debug, backfill a newly added downstream, rerun after fixing a bug); ②ordered—strictly ordered within a partition. To scale, the log is partitioned by key—ordered within each partition, parallel across partitions—exactly the Ch6 partitioning playbook.

The point of the log-based broker: it has both a database's durability and a message queue's low latency—fusing "delivering messages" and "storing data" into one thing. That's the technical bedrock that lets a "stream" serve as a trustworthy source of truth. The cost: one partition can only be read in order by a single consumer, so when you want 100 consumers to share one batch of messages in parallel and don't care about order, a traditional queue is actually more flexible.

2. Databases & streams: change data capture (CDC) and event sourcing

Wiring the "stream" abstraction to a database unlocks the chapter's most useful class of applications: keeping all systems in sync. In practice you almost never use just one database—beyond the primary there's a search engine (Elasticsearch), cache (Redis), warehouse, feature store, all of which must track the primary. The old way is application-level dual writes (update the DB, then manually update search), but dual writes easily fail to reconcile under concurrency or partial failure. The cleaner way is to turn database changes into a stream:

Change data capture (CDC): every insert/update/delete on the primary is captured as an event (typically by reading its replication log / WAL) and pushed onto a stream. Every downstream subscribes and applies it in order, ending up eventually identical to the primary—essentially generalizing Ch5's leader–follower replication across heterogeneous systems. Tools like Debezium do exactly this.

Event sourcing: a more radical step—don't treat "current state" as the source of truth; treat "every event that led to the state" as the source of truth. A shopping cart stores not "2 items now" but the immutable event chain add X, add Y, remove X; current state is computed by replay. Benefits: full audit (every step traced, accountable), time-travel to any moment, and many different views derived from one event stream (the same order events can drive inventory, revenue, and recommendations).

3. Stream-table duality: turning the database inside out

Behind CDC and event sourcing is one deep observation the author calls the duality of streams and tables:

• A table is a "snapshot" of a stream: a database table's current contents equal the result of replaying and summing all its change events (changelog) since creation. Your bank balance is the sum of all transactions.
• A stream is a table's "changelog": conversely, keep recording every change to a table and you get a stream; subscribe to it, apply each change, and any downstream can maintain a copy consistent with the source.

This is no word game but an architectural worldview: turning the database inside out—a traditional database hides the "changelog" internally and exposes only "current state"; streaming reverses it, promoting the changelog itself to a first-class citizen (right there in Kafka) and demoting "current state" to a rebuildable, multiply-copied materialized view. Search indexes, caches, and reports all become "different projections of one source-of-truth log"—inherently consistent, rebuildable from the log when broken. This is Kleppmann's famous "turning the database inside out," and the intellectual core of the final Ch12.

4. The trouble with time: event time vs processing time, windows & watermarks

This is the acknowledged brain-bender—and the true test—of the chapter. In batch, "time" isn't a problem (all data is there, compute freely); in streaming, the moment you want "how many transactions in the past minute," you hit a philosophical knot: which "minute"?

• Event time: when the event actually happened (the second the user tapped pay on their phone).
• Processing time: when the event was processed by the server.

The two often disagree: a user places an order in the subway, has no signal, and 5 minutes later—leaving the station—their phone finally sends the event. By processing time, it's wrongly counted into the window 5 minutes later. Processing time is simple but distorted by system load and network jitter; event time is "correct" but hits a deadlock: you can never be sure all events of a given window have arrived—a straggler could always be on the way.

The remedy is the watermark: the system maintains a heuristic line, "events before event time T have, I believe, mostly arrived," and once the watermark passes a window's end boundary, it emits that window's result. What about late events arriving afterward? Three attitudes: drop them outright (and count/alert), issue a correction (retract the old result and re-emit), or allow a grace period and drop only stragglers beyond it. There's no free lunch here—wait longer to miss fewer and you sacrifice latency; emit sooner and you tolerate the occasional straggler.

Cutting windows has its own varieties, four common ones: tumbling—end-to-end, non-overlapping ("each whole minute"); hopping—fixed length but sliding by a smaller step, so they overlap ("every minute, count the past 5 minutes"); sliding—at any instant look "back N minutes"; session—no fixed length, cut by "a user's burst of activity, broken only after some minutes of silence."

5. Joins on streams & fault tolerance: three joins and exactly-once

As in batch, streams often need joins, but because data is "in motion," the shapes are subtler—three kinds: ①stream-stream join (windowed)—pair two streams within a time window, e.g. match "search events" with "click events" for click-through rate; the hard part is both sides move, so you must buffer a window of state waiting for the other; ②stream-table join (enrichment)—use a (slowly changing) table to enrich each event in a stream, e.g. attach a user profile to each click; implementations often turn the table into a stream via CDC too, maintaining a local copy; ③table-table join (maintaining a materialized view)—merge two changelog streams into a continuously updated joined view.

Fault tolerance and exactly-once is streaming's ultimate exam. A failed batch job just reruns the whole batch (input's still there); a stream is unbounded, never "finishes"—so after a crash-restart, how do you guarantee "each event's effect takes hold exactly once, no dupes, no drops"? Three mainstream routes: ①microbatching (Spark Streaming)—slice the stream into one-second mini-batches and degrade to batch for fault tolerance; simple, but latency is bounded by batch length; ②checkpoint + idempotence (Flink)—periodically snapshot operator state; on crash, roll back to the last checkpoint and replay, with idempotent output, achieving "effectively-once"; ③atomic commit—bind "advance read progress" and "write result" into one transaction that succeeds or fails together (e.g. Kafka transactions). Note: exactly-once means "the external effect takes hold exactly once," not "physically processed only once"—underneath it's still usually replay + dedup / idempotence.

Key trade-offs & choices

Streaming choices trade off replayability, ordering, real-timeness, and correctness cost. Three tables pull out the most-tested, most-useful comparisons.

Table 1 · Traditional message queue vs log-based broker (Kafka)

Table 2 · Event time vs processing time (and late-event handling)

Table 3 · Choosing among the three stream joins

Why it matters / in real systems

Stream processing is the skeleton of today's real-time data stack. Apache Kafka (open-sourced by LinkedIn in 2011) elevated "the log" into infrastructure and became a de facto standard for event streams; atop it grew Kafka Streams, Apache Flink, Spark Streaming, Storm, Samza. CDC tools (Debezium) wire legacy databases into streams; real-time warehouses, real-time fraud detection, recommendation features, monitoring alerts, and IoT telemetry are all, at heart, this chapter's ideas. The author's "turning the database inside out" (unbundling / stream-table duality) directly frames Ch12's direction. In interviews, "why can Kafka be both durable and low-latency," "how is exactly-once actually implemented," and "event time vs processing time and watermarks" are high-frequency questions for data / backend roles—the answers are all here.

Pitfalls & debates

• Reading exactly-once as "physically processed once": the biggest misconception. It means "the effect on the external world takes hold exactly once," still typically implemented via replay + idempotence / transactional dedup—a message may be processed many times, only the effect isn't duplicated.
• Blindly using processing time: windowing by processing time for convenience means "per-minute metrics" get distorted by system load / consumer lag. Real business semantics (billing, fraud) demand event time + watermarks.
• Believing late events can be "never dropped": with event time there's always a "how long to wait before closing" trade-off—wait as long as you like and a later straggler can still appear. Engineering can only pick a watermark policy and accept occasional drops / after-the-fact corrections; there's no having both.
• Dual writes instead of CDC: writing to the database and search index at once from the app looks direct but easily fails to reconcile under concurrency / partial failure. Using CDC to turn "one change" into a stream propagated one-way is the reliable approach.
• The Lambda debate: running both batch + stream (Lambda) was once popular but drew criticism for maintaining two sets of logic whose results often disagree; Kappa (stream + replay only) is cleaner but demands maturity and manageable cost for "recomputing very long history." The trade-off remains live, not one-sided.
• Dated in places: written in 2017, since when Flink's event-time semantics, Kafka's exactly-once transactions, and "unified batch & stream" (Flink, lakehouse Iceberg / Delta / Hudi) have matured greatly; but the worldview of "an immutable event log + derived views" remains the bedrock of real-time systems.

Takeaways