Storage

In one sentence

You think storage means "pick a database". It is actually a three-layer stack: physical raw ingredients at the bottom (disks, memory, network), storage systems in the middle (file, block, object), and only then the abstractions on top (warehouse, lake, lakehouse). The chapter's lesson: every layer trades latency against cost, scalability and durability. The cloud-era default is "object storage + columnar files + on-demand compute", but "fully separated compute and storage" is only half true; real systems blend separation with local caching.

Glossary

Where it sits

The second chapter of Part II, "The Data Engineering Lifecycle in Depth". It follows Chapter 5 (where data comes from) and precedes Chapter 7 (ingestion) and Chapter 8 (queries and transformation). The authors put it before ingestion on purpose: storage is the floor under the whole lifecycle, read and written at every stage. In practice it maps to choices like a data lake on S3 or GCS, a Snowflake or BigQuery warehouse, and message retention in Kafka.

What problem this chapter solves

Data engineers rarely build storage from scratch, but they choose it all the time, and a bad choice often shows its cost years later. Picture a shop that adds 1 TB of clickstream logs a day and keeps three years: about 1 PB. On a local SSD cluster the bill and the operations are unbearable; parked entirely in an archive tier, every analyst query waits hours. Typical consequences of not understanding storage:

• Paying for performance you do not need. Data nobody has touched in three years sits on the most expensive hot tier.
• Using the wrong access pattern. Treating object storage as a database and updating single records constantly; each change rewrites a file, slow and costly.
• Getting locked in by the architecture. With compute and storage welded together, more data means buying more compute too.

The core ideas, explained

1. See the three layers: ingredients → systems → abstractions

The chapter is built on a layered picture. At the bottom are the raw ingredients: disks, SSDs, memory, network, CPU, plus techniques like serialization, compression and caching. In the middle are storage systems built from them: file, block and object storage, HDFS, caches, streaming storage. On top are the storage abstractions data engineers touch directly: warehouse, lake, lakehouse. The authors' warning: do not shop only at the top layer; its performance and cost come entirely from the layers below.

2. Ingredients: latencies differ by orders of magnitude, and caching answers that gap

The gaps between ingredients are not percentages but orders of magnitude: a memory access is around 100 nanoseconds, an SSD read about 0.1 ms, a hard-disk seek a few milliseconds, the first byte from object storage typically tens of milliseconds, and an archive-tier retrieval is measured in hours. Prices run roughly in reverse. Hence the book's cache hierarchy: the hottest data on the fastest, priciest tier, colder data further down.

Two often-overlooked ingredients are serialization and compression. The same data stored as CSV or as columnar Parquet can differ by an order of magnitude in bytes read; compression trades CPU for disk and network, which in the cloud, where the network is often the bottleneck, usually pays off.

3. Row versus columnar: one table, two layouts

Row-oriented storage keeps all fields of a record together, good for transactional access like "fetch everything about one order". Columnar storage keeps the values of one column together, good for analytical access like "sum or filter one column". Analytical queries usually touch a few of dozens of columns, so columnar reads only those; values in one column share a type and repeat a lot, so they compress far better. The cost: changing one row touches many column files, so single-row writes and updates are slow. Columnar formats are made for "write in bulk, read a lot".

4. Three storage systems: file, block, object, with the weight on object

File storage is the familiar directory tree, with random reads and writes and in-place edits; clouds offer managed network file systems. Block storage is the raw disk an operating system sees; databases and VM boot disks sit on it. Cloud virtualized block storage can be snapshotted and attached to instances, while an instance's local disk is fastest but its data disappears when the instance stops.

Object storage (S3, GCS, Azure Blob) gets the most attention in the chapter. Data is stored as objects under keys in flat buckets, and objects are immutable: you cannot change part of one, only overwrite or delete it whole. That restriction buys practically unlimited capacity, enormous parallel throughput (hundreds or thousands of tasks reading different objects at once), high durability from replication (S3 advertises a design durability of eleven nines), and pay-per-use pricing. Its weaknesses: high per-request latency and a poor fit for frequent small updates. S3 switched to strong read-after-write consistency at the end of 2020; before that, reading right after a write could return a stale version, a historical trap the book calls out. Object stores also offer storage classes such as standard, infrequent access and archive: the colder, the cheaper, but slower to retrieve and with retrieval fees.

The other systems each have their niche. HDFS splits large files into blocks replicated across cluster nodes and runs computation next to the data (it descends from Google's GFS). Caches (Memcached, Redis) keep hot data in memory. Streaming storage (Kafka) keeps events for a retention period and allows replay. And warehouses no longer lean on traditional indexes but on partitioning and clustering: splitting by date and placing similar values together so queries skip most irrelevant blocks (for example Snowflake's micro-partitions).

5. Storage abstractions: warehouse, lake, lakehouse

A data warehouse is a strictly structured central store optimized for analytics; cloud warehouses already put their storage on object storage and scale compute on demand. A data lake dumps raw files into object storage and decides structure later; the lesson of first-generation lakes is that they easily become "data swamps", with no transactions, no schema management, and whole partitions rewritten to change one record. The lakehouse merges the two: data remains open columnar files on object storage, but a table format (Delta Lake, Apache Iceberg, Apache Hudi) adds a metadata layer recording which files make up which version of a table, enabling transactions, updates and deletes, schema evolution and time travel. The book also describes a stream-to-batch storage architecture: data written to streaming storage is also landed in batch storage for long-term querying.

6. The big trends: separation, temperature, when to apply schema

Separating compute from storage. The Hadoop era preached colocation: compute where the data lives. The cloud reversed it: data sits in object storage, compute clusters spin up for a job and are torn down afterwards (ephemeral clusters), and storage and compute scale and bill independently. But the authors stress that real systems are hybrids: cloud warehouses cache hot data on compute nodes' local SSDs, and engines like Spark spill intermediate results to local disk. Separation buys elasticity; local caching wins back speed.

Data lifecycle and retention. Classify data as hot, warm or cold by access frequency and let object-storage lifecycle rules demote it automatically. Set retention by compliance needs too: what must be deleted has to be deleted, not merely moved somewhere cheaper.

When to apply schema. Schema on write (the warehouse way) guarantees quality but is rigid; schema on read (the lake way) is flexible but pushes the parsing burden onto every reader. The book also touches on data catalogs (so people can find and understand data), data sharing (granting another cloud warehouse direct read access with no copy), zero-copy cloning (copying a table copies only metadata), and the isolation-versus-cost trade-off of single-tenant versus multitenant storage.

Key trade-offs and choices

Table 1 · Three storage systems

Table 2 · Storage abstractions

Table 3 · Temperature tiers

Rule of thumb: first ask about the access pattern (small random updates or big bulk scans), then how hot the data is and how long it must be kept, and finally whether several engines must share the same data. If they must, lean toward a lakehouse on an open table format.

Why it matters / in real systems

For an interview prompt like "1 TB of logs a day, kept three years, analysts must be able to query it", a strong answer lands like this: raw data written as Parquet to object storage, partitioned by date; a table format managing transactions and schema; the last 30 days accelerated by a warehouse or query-engine cache; anything older than a year demoted to archive by lifecycle rules; compute clusters scaling with query load. Each step maps to one trade-off in this chapter.

Pitfalls and debates

• "Object storage is just a very big disk." It cannot edit in place and each request is slow; used as a file system with frequent small writes, it is slow and expensive.
• "Separating compute and storage means no local disks." Real systems rely on local caches to regain speed; separation is a hybrid, not an absolute.
• "Columnar is always faster." For single-row lookups and frequent updates, row storage fits better; columnar wins on big scans of few columns.
• "Small files don't matter." Millions of kilobyte-sized files explode metadata and request counts; lakehouses need regular compaction.
• "Archive is cheap, so dump everything there." Retrieval fees, minimum storage durations and hour-long waits can make one urgent query cost more than a year of storage.
• "Schema on read means no schema to manage." The structure problem is only postponed, and handed to every reader.
• Limits of its time. The book came out in 2022, early in the lakehouse and table-format contest; open table formats have since gained broader support across the major cloud warehouses, but the basic "separation plus caching" trade-off has not changed.

Key takeaways