Books Deep-Read · DDIA · Chapter 4
Designing Data-Intensive Applications · Ch 4 · Martin Kleppmann · 2017
The app on your phone updates almost every week — yet the post you made last year, the order you placed three years ago, still has to show up correctly today. Hidden in there is a quiet problem: the code changes constantly, but old data has to stay readable. This chapter is about exactly that: data lives as living objects in memory, but the moment it goes to disk or onto the network it must be "flattened" into a stream of bytes — and when the code gets upgraded and the shape of the data wants to change too, how do you keep the new and old generations able to read each other's writing?
Think of "storing and sending data" as shipping a parcel: the stuff spread on your desk (objects in memory) can't just go in the mailbox — you first pack it into a box, and that step is called encoding; the recipient unpacks and restores it, called decoding. The catch: sender and receiver aren't using the same instruction manual. Your app already updated and slipped something new into the box; their app is still the old version, and its manual has no entry for it. How do you keep the old version from being stumped on arrival? That is the whole drama of this chapter.
Because big systems can't just stop and swap everything at once. Upgrading hundreds or thousands of servers can only be done a few at a time (a rolling upgrade) — so for a stretch of time, new-version and old-version code are running side by side. Data the old code wrote must be readable by the new code (call it backward compatibility, honoring the past); the reverse is trickier: data the new code writes must also be readable by the old code, even with fields it has never seen (call it forward compatibility, honoring the future). Data often outlives the code — a record filed away five years ago is still sitting in the database, read by several later generations of code. Get compatibility wrong and every upgrade becomes a buried landmine.
The clever formats (Google's Protocol Buffers, Facebook's Thrift) use one plain, effective move: they identify fields not by name but by a number tag — like airport baggage, where what's read is the number on the tag, not what your suitcase looks like. The stored bytes contain only "tag N: this value," with no long field names. So: to add a new field, hand out a fresh, unused number; old code that hits a tag it doesn't recognize just shrugs and skips it, never crashing. Adding fields without collisions rests entirely on that tag. The one iron rule: once a tag number is handed out, it can never be changed or reassigned to someone else.
Data flows from one piece of code to another by just three routes, each of which must pass the compatibility test: ① Into a database — the code that writes it and the code that later reads it may be several versions apart; ② Calling a service — a phone app sends a request to a backend, and the two sides upgrade independently; ③ Through a message queue — one system drops a message into a pipe and another picks it up later, so send and receive aren't even at the same moment. As long as the two ends might be on different versions, the encoding format has to carry the burden of keeping them mutually intelligible.
Data must be packed into bytes (encoded) before it leaves memory and unpacked (decoded) when it comes back. The hard part is that code is rolling-upgraded and new and old versions run at once, so the format must let new code read old data and old code survive new data. The way through: identify fields by number, not name — add fields with fresh numbers, never reuse an old one, and upgrades stop being landmines.
Want the concrete formats, how the bytes lay out, how real systems use them? → Switch to the deep read
Chapters 2 and 3 cover how data is modeled and stored on disk; Chapter 4 asks a question closer to daily engineering: data has to be translated back and forth between an in-memory representation (objects, structs, lists) and a byte sequence (files, the network), so how do you do that translation (encoding / serialization) such that code and data can each evolve independently? The soul of the chapter is two terms — backward compatibility: new code can read old data; forward compatibility: old code can read data written by newer code. Nail both and you dare to do a rolling upgrade on a live system instead of stopping every node to swap code at once.
.proto, Thrift's .thrift), from which a tool generates read/write code for each language.This chapter closes out Part I, "Foundations of Data Systems." It follows Chapter 2 (data models) and Chapter 3 (storage engines), taking "what the data looks like and how it's stored" to an unavoidable conclusion — the moment data has to cross a process or cross time, it must be encoded into bytes — and it is also the bridge into Part II, "Distributed Data": replication, partitioning, multi-node communication are all, at bottom, data flowing over a network, and flowing means encoding first. In practice it maps to decisions you make daily: JSON or Protobuf for this API? Will adding a field to events on the queue break downstream consumers? Can this database table's structure be changed safely?
Applications keep changing — requirements shift, and the shape of the data follows: add a field, split a field, change a type. DDIA calls this mutability of data shape schema evolution. The trouble is that a change never syncs instantly to every corner of the system:
So "all four combinations of new/old code and new/old data existing at once" becomes the norm. For the system to keep running through that mess, the encoding format of all data must be backward- and forward-compatible at the same time. DDIA's pointed phrase: "data outlives code." A record written five years ago still lies quietly in the store, while the code that reads it has been rewritten countless times. Treat compatibility as anything less than a first-class citizen and every upgrade plants a landmine for the future.
In-memory data is full of pointers / references — an object holds the address of another object, and those addresses are meaningless outside the process. So to hit disk or the network, it must be translated into a self-contained byte sequence — that step is encoding, a.k.a. serialization. Nearly every language ships its own (Java's Serializable, Python's pickle, Ruby's Marshal), usable in one line — but DDIA urges you not to use it for durable storage or cross-system communication: ① it's tied to the language — a Java-serialized blob is basically unreadable elsewhere; ② it's a security hazard — deserialization can be coaxed into instantiating arbitrary classes, a classic remote-code-execution hole; ③ its versioning is poor — the forward/backward compatibility this chapter prizes is exactly what it handles badly; ④ performance and size are often unimpressive too. Verdict: built-in serialization buys convenience now and pain later.
To cross languages you turn to standard formats. JSON, XML, CSV win on being human-readable and universally supported — the de facto standard for the web and config. But the sour parts are real: numbers are ambiguous — JSON can't distinguish integers from floats and has no notion of precision, so integers beyond 2^53 (like Twitter's snowflake IDs) silently lose precision in languages that parse them as IEEE double floats (Twitter's API returns each ID both as a number id and as a string id_str for this very reason); no binary strings — binary must be Base64'd, bloating it by 33%; schema optional — field types rest on a verbal agreement between the two sides and easily drift apart. They are good enough and universal, but once you care about size, type precision, and enforced compatibility rules, it's time for binary + schema formats.
Thrift (Facebook) and Protocol Buffers (Google) share one idea: pin down a schema in an IDL first (each field gets a numeric field tag), then generate read/write code per language from it. The key: the encoded bytes store only "tag + type + value," never the field name. A userName="Martin" no longer drags an 8-character key around — just a one-byte tag. So they are far smaller than JSON: DDIA's example record is about 81 bytes in JSON, roughly 34 bytes in Thrift's compact protocol and 33 in Protobuf — more than half smaller.
The size is a by-product; the real payoff is schema evolution. Because fields go by number, not name: adding a field just means a fresh, unused tag number — old code hitting an unknown number skips it using the type info (forward compat); new code reading old data supplies a default for the missing new field (backward compat). Two iron rules: a newly added field must not be required, or old data won't parse; once a tag number is used, never change it and never reuse it — renaming is free (names aren't in the bytes), but changing the number means it's a different field. You may only remove optional fields, and that number is retired forever, never handed to anyone new.
Avro (born in 2009 for Hadoop) takes a different road: the encoded bytes carry no field names, no field numbers, not even type markers — just a string of values laid end to end. What decodes something that compact? Two schemas: the writer's schema used at write time and the reader's schema used at read time. They need not be identical, only compatible; Avro aligns them by field name (schema resolution) — a field the writer has but the reader lacks is ignored (forward compat), and a field the reader has but the writer lacks is filled from the default declared in the reader's schema (backward compat).
The cost: at decode time you must be able to obtain the writer's schema. In practice a large file stores it once at the top (in Hadoop, spread over a million records, the cost is negligible); a database or message stream tags each record with a small version number that looks up the matching schema in a schema registry. Avro's biggest sweetener is being friendly to dynamically generated schemas: add a column to a database table and you can generate a fresh Avro schema straight from the new structure — no need, as with Protobuf, to hand-maintain a unique, never-reused tag number for every column. That's exactly why it's favored in data warehouses and big-data pipelines.
Formats done, DDIA asks: how does data actually flow from one piece of code to another? It groups this into three dataflow modes, each of which must pass compatibility. Via a database: the writing process and the reading process may be different versions, even "writing for your future self"; one pitfall — old code reads a record with new fields, changes something else, and writes it back, and if the parser drops the fields it didn't recognize, it silently erases data, so format and code must both preserve unknown fields. Via a service (REST / RPC): client and server upgrade separately, usually server first, so you need backward-compatible requests and forward-compatible responses; RPC tries to make a network call look like a local one, but the network delays, drops, times out, so idempotency and retries have to catch it. Via async messages (message queue / actors): the sender drops a message into a broker and the receiver picks it up at some later moment — the two ends fully decoupled and time-shifted, and the compatibility requirement is most direct.
Selection here has two layers. First, choosing an encoding format — from "usable in one line" built-ins, to human-readable text, to compact binary with enforced compatibility rules, the costs and benefits step up:
Table 1 · Five classes of encoding format
| Format | Size / perf | Schema & compat | Cross-language | Fits |
|---|---|---|---|---|
| Language built-in (pickle/Serializable) | mediocre | poor versioning, security holes | No (locked to language) | same-language, throwaway cache; not for storage / comms |
| JSON / XML | large, carries names | schema optional; ambiguous numbers, no binary | Yes | web APIs, config, human-readable cases |
| CSV | smallish | untyped, ambiguous, no nesting | Yes | simple tabular import/export |
| Thrift / Protobuf | small (~1/2 of JSON) | schema required; field numbers drive evolution | Yes (generated code) | microservice RPC, high-frequency internal comms |
| Avro | smallest (no tags, no types) | writer/reader schemas aligned | Yes | big data / warehouse, dynamically generated schemas |
Second, choosing a dataflow — the same data via database, service, or message differs in the direction and difficulty of compatibility:
Table 2 · Compatibility focus of the three dataflow modes
| Via database | Via service (REST / RPC) | Via async message | |
|---|---|---|---|
| Who talks to whom | writer ↔ reader (incl. future self) | client ↔ server | sender ↔ receiver (through broker) |
| Timing | across time, data outlives code | synchronous request-response | async, decoupled, buffered/redelivered |
| Compat focus | both ways; read-modify-write keeps unknown fields | request backward, response forward | both ways, ends upgrade independently |
| Typical tech | databases + the encodings above | REST, gRPC, SOAP, Finagle | Kafka, RabbitMQ, actor frameworks |
| Signature pitfall | old code writes back, erasing new fields | network delays / fails, need idempotent retries | duplicate / reordered messages, consumer must cope |
A practical rule of thumb: for external, human-read, rarely-changing interfaces, JSON is plenty; for internal, high-frequency service-to-service comms where size and enforced compatibility matter, reach for Protobuf / gRPC; for big-data pipelines whose schema tracks the table structure automatically, Avro is the least fuss.
This chapter is the foundation of the microservices and big-data era. Any system split into many services that deploy independently relies on this: when team A upgrades its service it can't wait for team B to change too — and that's possible only because the interface's encoding is backward- and forward-compatible. gRPC uses Protobuf; Kafka + Avro + Schema Registry govern event streams; REST + JSON carry open APIs — the infrastructure you use daily is a direct product of this chapter's ideas. Those claims have public engineering practice to back them:
① In one line: data must be encoded to bytes leaving memory and decoded entering it; the crux is a format that supports schema evolution, achieving backward (new reads old) + forward (old reads new) compatibility.
② Why it's non-negotiable: big systems use rolling upgrades, so new/old code and new/old data inevitably coexist, and data outlives code — bad compatibility makes every upgrade a landmine.
③ Don't use language built-in serialization (pickle / Serializable): locked to the language, security holes, poor versioning — fit only for a throwaway cache.
④ Text formats (JSON / XML / CSV) are universal and readable, but numbers are ambiguous (big ints lose precision), no binary, schema optional.
⑤ Thrift / Protobuf: identify fields by number, not name; add fields with new numbers, never reuse an old one, never make new fields required — about half the size of JSON, compatibility resting on the tag.
⑥ Avro: data carries no tags and no types, decoded by aligning writer and reader schemas by name; most compact, and friendly to dynamically generated schemas — the darling of big-data pipelines.
⑦ Three dataflows: via database (across time, guard read-modify-write), via service (REST / RPC, request backward · response forward), via async message (Kafka / actors, send-receive decoupled).
⑧ In practice: gRPC=Protobuf, Kafka+Avro+Schema Registry, REST+JSON; independently deployable microservices and safely evolvable event streams all rest on this chapter's compatibility rules.