Book Deep-Read · Fundamentals of Data Engineering · Chapter 7
Fundamentals of Data Engineering · Ch 7 · Ingestion · Reis & Housley · 2022
The sales dashboard your company opens every morning draws its numbers from the order system, the payment system, app click tracking, and spreadsheets sent over by partners — dozens of places. This chapter is about moving data out of all those "other people's houses" and safely into your own data platform. That step is called ingestion. It looks like simple hauling, yet it is where the whole pipeline breaks most often.
We talk about "batch" and "real-time streaming" as if they were two different things. But think about it: orders keep happening one after another and never stop. A "daily batch" is just us dropping a gate across a river that never stops flowing, once every 24 hours. Batch and stream are not two kinds of data; they are how often you choose to open the gate.
Most teams default to moving data in scheduled batches, and only build real-time streams when the business truly needs to see things within seconds; and if a ready-made tool can do the hauling, they buy rather than build. One honest cost: streaming makes data fresher, but someone has to watch it around the clock and handle the late and duplicate arrivals, so cost and complexity rise sharply.
Ingestion moves data from other people's houses into yours: batch and stream differ only in how often the gate opens; moving only what changed, landing it as-is, and planning for late and duplicate arrivals matter more than chasing "real time".
Want bounded vs unbounded data, push, pull and poll, snapshot vs incremental extraction, the ETL vs ELT debate, late, out-of-order and duplicate events and dead-letter queues in streaming ingestion, plus evidence from Debezium and Airbyte? → Switch to the deep read
You think ingestion is just "copy data from A to B". In fact it is the most fragile link in the data lifecycle: the source is not yours, and it changes, stutters and resends. The idea this chapter builds is that all data is unbounded by nature, and a batch is just a boundary we draw on it. Designing ingestion is not about picking a tool; it is about first answering a string of questions — how big, how often, push or pull, can it be replayed, what happens to late and duplicate records — and then trading off batch against streaming and ETL against ELT based on real needs.
The third chapter of Part II, "The Lifecycle in Depth". It follows Chapter 5 (where data comes from) and Chapter 6 (where it is kept) and leads into Chapter 8 on queries and transformation. Chapter 5 covers source systems themselves; this chapter covers the road from the source into storage. In practice: CDC replication, Kafka ingestion, managed connectors like Fivetran or Airbyte, and the nightly export job.
Picture an e-commerce company: an order database of 500 million rows with about 2 million inserts or updates a day; app events peaking at 50,000 per second; and a dozen SaaS systems (CRM, ad platforms) reachable only by API. Analysts want "yesterday's numbers before 9 a.m."; the fraud team wants "suspicious orders within seconds". Typical outcomes when ingestion is done naively:
The authors define ingestion as moving data from one place to another, and separate it from integration: ingestion is the haul from source to storage, while integration combines data from several sources into a new dataset.
The authors list questions to answer before designing: who will use the ingested data, and can it be reused? What are the source and destination, how much data and how often? What format is it in, and can downstream use it as-is? If it is a stream, does it need transforming in flight? The most fundamental pair of ideas is bounded versus unbounded data: bounded data has a clear start and end ("orders on September 1"), unbounded data keeps coming. The book's stance is blunt — data is unbounded until someone draws a boundary on it. Business events are a stream by nature; a "daily batch" is a slice we cut for convenience.
So ingestion frequency is not a batch-or-stream switch but a continuum: yearly, daily, hourly, every few minutes (micro-batch), and near real time. The authors note that even when data is ingested as a stream, downstream often processes it in batches (say, retraining a model hourly) — judge the frequency of each stage separately.
Synchronous vs asynchronous ingestion. In a synchronous pipeline the source, ingestion and destination are tightly coupled; if a late step fails, hours of earlier work may have to be rerun. An asynchronous pipeline moves individual events through a buffer (such as Kafka), so each stage fails and retries on its own.
Serialization and deserialization. The source encodes data into bytes and the destination decodes them; both sides must agree on format and version, or the data "arrives but can't be read".
Throughput and scalability. The pipeline must absorb peaks and backlogs: when a downstream system comes back after a few hours down, the backlog arrives like a flood. The authors favor managed services, most of which scale elastically.
Reliability and durability. Reliability means the pipeline runs as expected and recovers from failure; durability means data is not lost. The key insight: a lot of source data is gone forever if you fail to capture it (events that only live in memory, for instance), so weigh spending on redundancy and self-healing against "what happens if we lose this" rather than chasing zero risk.
Payload. Know the data's kind (tabular, image, text), shape, size, schema and types — above all schema evolution: upstream adding fields, changing types and renaming columns is normal. Counter it by detecting schema changes automatically, or by using a schema registry that versions each schema so readers and writers parse by version; more fundamentally, open a line of communication with the upstream team.
Push, pull, poll. In push, the source sends data out; in pull, the ingestion system fetches it; in poll, it checks periodically whether anything changed and pulls only if so. CDC and event streams tend to push; scheduled exports tend to pull.
Snapshot vs incremental extraction. Copying everything each time (a snapshot) is simplest but expensive; extracting only what changed since last time (differential, or incremental) saves resources but requires the source to reliably say "what changed". For the 500-million-row order table above, with 2 million changes a day, incremental extraction moves only about 0.4% of a full copy.
File-based export. The source system exports files itself and pushes them to object storage or SFTP. The source keeps control of security and access, with no direct database connection opened up.
ETL vs ELT. ETL extracts, transforms in flight, then loads into the warehouse; ELT loads raw data as-is into a warehouse or lake and transforms it there with SQL. Once cloud warehouses made storage cheap and compute elastic, ELT became mainstream: the raw data is kept, and transformation logic can be rewritten and rerun. The authors add two warnings. First, if ELT does only the "EL" and nobody does the "T", the lake turns into a swamp. Second, ETL is not dead — when data must be masked first, shrunk first, or the destination is poor at transformation, transforming in flight still makes sense.
Inserts, updates and batch size. Columnar warehouses love large bulk writes and hate row-by-row inserts and frequent small updates — each one may rewrite a file. So ingestion should batch, sizing batches to the destination. For data migration (say, moving a whole database to the cloud), the advice is to trial a small sample and reconcile schema differences first; the hard part is usually not moving the data but repointing every upstream and downstream pipeline at the new system.
Streaming swaps the question "how much to move" for "when, in what order, and how many times". The book walks through what to plan for:
The second half of the chapter is a toolbox. The most important items:
updated_at column for "rows changed since last time" — simple, but it sees only each row's latest state, missing intermediate changes and deletes. Continuous CDC reads the database's change log (MySQL binlog, PostgreSQL WAL) and turns every insert, update and delete into an event on a streaming platform — no lost intermediate states, and much less query load on the source. The cost: you need log access, and must manage how long the source retains its log.Finally, the authors stress that ingestion is a people problem: upstream software engineers decide what data looks like and when it changes; bringing them in and agreeing on change notices beats any tool. On the DataOps side, monitor pipeline uptime, latency and data volume — a sudden 50% drop often means something upstream quietly broke — and run data-quality tests.
Table 1 · Batch vs streaming ingestion
| Mode | Good for | Cost |
|---|---|---|
| Daily / hourly batch | Daily reports, financial reconciliation, offline training; most analytics | Freshness measured in hours; full snapshots load the source |
| Micro-batch (minutes) | Operations dashboards, near-real-time rollups | More frequent scheduling, more small files to compact |
| Streaming (seconds) | Fraud, real-time recommendations, monitoring and alerting | Must handle late, out-of-order and duplicate events; 24/7 on-call; highest cost and complexity |
Table 2 · Extraction methods
| Method | Upside | Cost |
|---|---|---|
| Full snapshot | Simplest, self-consistent | Slow and costly at volume; rescans the source repeatedly |
| Batch CDC (by update time) | Only changes, easy to build | Misses intermediate states and hard deletes; relies on the source maintaining timestamps |
| Log-based CDC | Captures every change, low latency, little query load on the source | Needs log access and extra components (e.g. Debezium + Kafka); heavier operations |
Table 3 · ETL vs ELT
| Pattern | When to pick it | Cost |
|---|---|---|
| ETL | Data must be masked or filtered before landing; limited destination compute; large size reduction | Raw data isn't kept, so changed logic is hard to recompute |
| ELT | Cloud warehouse or lakehouse; transformation logic changes often; recomputation needed | The raw layer keeps growing — ungoverned, it becomes a swamp; warehouse compute bills rise |
The test: first ask how fresh downstream really needs the data — the authors' position is to adopt true streaming only when its business value clearly outweighs its cost. Then ask whether the source can reliably tell you what changed. Finally ask whether you need the raw data kept for recomputation — if so, lean toward ELT.
For the interview question "sync a production database into the warehouse", a strong answer lands like this: log-based CDC writes MySQL changes into Kafka, partitioned by primary key so each row stays in order; consumers do idempotent upserts by key and bulk-load the warehouse; raw change events also land in object storage, replayable; SaaS data goes through managed connectors; per-table latency and daily volume are monitored with alerts on anomalies; schema changes go through a registry with notice to downstream. Each step maps to one consideration in this chapter.
Evidence from open source
① Ingestion moves data from source systems into storage; it is the most fragile link in the lifecycle because you don't own the source.
② Data is unbounded by nature; a batch is a boundary we draw. Frequency is a continuum from yearly to per event, and each stage can differ.
③ Before designing, answer: volume, frequency, sync or async, push, pull or poll, reliability needs, and how the schema will change.
④ Prefer incremental over full; timestamp-based batch CDC misses deletes and intermediate states, log-based CDC misses nothing.
⑤ ELT is the cloud-warehouse default — raw kept, recomputable — but without transformation and governance it becomes a swamp; ETL still fits masking and similar cases.
⑥ Streaming ingestion must plan for late data, disorder, duplicates, replay, retention, message size and dead-letter queues; "exactly once" comes from at-least-once plus idempotency.
⑦ Columnar warehouses hate small row-by-row writes, so batch your loads; buy managed connectors instead of hand-writing integrations.
⑧ Monitor latency and data volume, and agree on change notices with upstream engineers — the people problem matters more than the tool.