Fundamentals of Data Engineering Ch 2 · The Data Engineering Lifecycle

What is this chapter about?

Your banking app says "You spent $340 on dining this month, 12% more than last month." Behind that one line, every card swipe was collected, stored, sorted by category and finally summed into a number on your screen. This chapter splits that journey into five stops and tells you which questions to ask at each stop before you build anything.

An analogy

Think of a restaurant kitchen. Produce comes from farms (data is born in business systems); a buyer brings it through the back door (moving the data in); the freezer, fridge and pantry keep it in the right place (storage); cooks wash, chop and cook (shaping it into something usable); waiters carry plates to tables (delivering it to people reading reports and to prediction models). And a few jobs belong to no single station but matter at every one: hygiene, who gets into the kitchen, shift schedules, bookkeeping.

Why the old world was hard

Buying pans before writing the menu. Many teams pick trendy tools first and only then look for something to do with them.
The farm changes without warning. An upstream system changes a format, the whole line quietly goes wrong, and nobody notices until the boss sees a strange report days later.
Dishes nobody eats. A dashboard built with great effort that nobody opens.

How it works, intuitively

The back door is where things jam. Where the produce comes from, how often, whether it spoils or arrives twice: the kitchen can't control any of it, so it has to ask up front.
One delivery a day, or on demand? Most restaurants are fine with one big shopping run each morning. Only when "ordered now, needed now" truly matters is it worth paying more, and worrying more, for on-demand delivery.
Do they deliver, or do you go pick up? The farm can send produce to you, or you can drive over on a schedule. There's also a clever trick: instead of searching the whole warehouse, read the farm's ledger and move only what was newly written down.
Hot and cold by how often you use it. What you use every day stays within reach (costly but fast); what you use now and then goes in the freezer; what you touch once every few years goes to a far-away warehouse (cheap, but you wait to get it back).
The table isn't the end. A finished conclusion can go back to the front desk, like writing "this guest loves spicy food" into the ordering system so the waiter suggests it next time.
Jobs every station shares. Who may enter the kitchen, whether each ingredient can be traced, who is on call when something breaks, which step waits on which: these run through everything, and missing one causes trouble.

What it buys you / how to choose

With this five-stop map, a team first asks "who are we feeding, and how fresh does it need to be?" and only then picks tools, not the other way round. Start with the simple one-delivery-a-day approach; upgrade to on-demand only when the business can say "every minute late costs us money." The honest cost: on-demand really is fresher, but it is pricier, harder to maintain, and more likely to break at 3 a.m.

One line to remember

Data passes five stops on its way to being used: born, brought in, stored, prepared, served. Ask the right questions at each stop before choosing tools, and keep a few cross-cutting jobs underneath it all.

Want the book's key questions for each stage, how to weigh batch against streaming, push vs pull and change data capture (CDC), storage temperatures, and what each of the six undercurrents covers? → Switch to the deep read

The one-liner

You'd think the lifecycle is a tool list ("A for ingestion, B for storage, C for transformation"). What the chapter actually gives you is five sets of questions to settle before building anything: where data comes from, how it gets in, where it lives, how it is shaped, and who uses it, plus six undercurrents running underneath the whole thing. Its most counterintuitive conclusion: batch is still the default; real-time streaming must earn its cost with a business case.

Glossary

Batch: collect data over a period or up to a size, then process it all at once, e.g. every night at 2 a.m.
Streaming: process data record by record, or in tiny chunks, as soon as it is produced; latency is usually seconds.
Schema: the agreed structure of data, such as which columns a table has and what type each one is.
OLTP database: the database behind business transactions (orders, transfers), built for fast reads and writes of single records.
Object storage: S3-style storage that holds huge numbers of files cheaply, billed by capacity.
Metadata: data about data, such as what a table means, who owns it, when it was updated, where it came from.
DAG (directed acyclic graph): a dependency graph of tasks; an arrow means "B runs only after A finishes," and nothing loops back.
SLA (service-level agreement): a promise to consumers, e.g. "data ready by 8 a.m. every day."

Where it sits

Chapter 2 of Part I, "Foundation and Building Blocks." Chapter 1 drew the lifecycle map; this chapter walks each stage and explains the six undercurrents. Chapter 3 builds architecture principles on top, and Part II (Chapters 5–9) gives generation, storage, ingestion, transformation and serving a chapter each. In practice it answers: "before building a data pipeline, what do I need to think through first?"

What problem this chapter tackles

Tool names change every year, yet pipelines keep failing in the same ways: an upstream field changes silently, data arrives late or twice, a real-time pipeline gets built for a daily report, a dashboard goes unused, someone deletes data that should have been kept. None of these are bugs in a tool. A stage asked the wrong questions, or an undercurrent had no owner.

The chapter first separates two ideas: the data lifecycle is the full life of data from creation to disposal; the data engineering lifecycle is the part of it data engineers own. Then it gives each stage a checklist. The authors' judgment from experience: source systems and ingestion are the biggest bottlenecks in the whole lifecycle, because they sit outside your control.

Core ideas, explained

1. Generation: you don't own the source, but you must understand it

A source system is where data is born: application databases, app event tracking, third-party APIs, IoT devices. The data engineer usually doesn't own it, yet pays for every change it makes. The book's evaluation questions fall into a few groups:

How is data persisted, and how fast is it produced? Events per second, GB per hour. Picture a store database writing 2,000 orders per second: that is about 170 million rows a day, and it sets the scale of every later stage.
What is the quality? Duplicates, late-arriving data, how often errors show up.
How does the schema evolve? A relational database with a fixed schema, or a schemaless document store whose structure changes with application code? When upstream adds a column or changes a type, how will you find out?
Will reading it hurt it? A heavy query on a production OLTP database can slow down real customers placing orders.
Do you get snapshots or changes? A periodic full copy of the current state, or a stream of "who changed what" events.

The book's advice: build a relationship with the owners of source systems, so you hear about upstream changes before they happen, not after a report breaks.

2. Storage: not a stage, but the floor under every stage

In the book's diagram, storage sits beneath the other stages: ingestion lands data, transformation reads and writes it, serving queries it. Questions to ask: can its read and write speeds keep up with upstream and downstream? Will it become the bottleneck? How does it scale if data grows tenfold? Is it storage only (object storage) or can you query it directly (a cloud warehouse)? Can it record metadata and lineage?

One very practical split in the book is "temperature" by access frequency:

Three storage temperatures: access frequency decides the tier, trading retrieval latency for storage cost.

Temperature matters because of cost: the same petabyte kept entirely hot versus tiered hot and cold can differ by an order of magnitude on the bill. The catch is that cold data takes time to retrieve and carries retrieval fees, so data you thought was cold but actually gets queried often ends up costing more in the wrong tier.

3. Ingestion: the stage most likely to jam

Ingestion moves data from source systems into storage. It is the most fragile link: a source that stops responding, a sudden drop in quality, an upstream format change, any one of them can break the whole downstream line. Two pairs of choices matter here.

Batch vs streaming. The book offers an important lens: at the moment it is produced, almost all data is a continuous stream of events; batch is simply a convenient way of processing that stream in large chunks. Streaming ingestion gets data downstream almost in real time, but the book keeps asking first: what business value does real time give that batch doesn't? How much more money and operations time will it cost? Will the streaming pipeline hold up when infrastructure fails? The conclusion: batch remains the sensible default for most cases; adopt true real time only after you find a business use case that justifies the trade-offs.

Push vs pull. In push, the source system sends data to you; in pull, you fetch it from the source on a schedule. Change data capture (CDC) comes in both flavours: periodically querying "which rows changed since last time?" by timestamp is pull; reading the database's own change log (such as MySQL's binlog) and forwarding each commit as it happens is closer to push.

Two ingestion choices: pull (scheduled queries) or push (forwarding the change log), and batch or stream.

4. Transformation: turning raw data into something useful

Transformation is where data starts to create value. The book lists several layers: type mapping (turning strings into dates), cleaning and standardizing (deduplicating, unifying units), modeling (reshaping into analysis-friendly structures), business logic (what exactly counts as an "active user"), and, for machine learning, featurization: extracting the inputs a model can use from raw data.

The book stresses two points. First, business logic is the biggest driver of transformation, and the part most likely to go wrong and most in need of an owner: three reports with three definitions of "monthly active users" destroy trust faster than any outage. Second, every transformation should justify its cost and return: whose time does this step save, which decision does it support? Keep transformations simple and isolated from one another.

5. Serving: data only has value when it gets used

Serving is the last stop. The book has a sharp warning: beware data vanity projects, lakes and warehouses built with great fanfare that nobody uses. Serving comes in three kinds:

Analytics: BI reports (the past), operational analytics (the present, like live order volume), and embedded analytics (dashboards built into the product for customers). The last one demands care with multitenancy: one customer must never see another customer's data.
Machine learning: feeding training and inference, often through a feature store. The book suggests asking first: is the data good enough, discoverable, representative of ground truth, free of obvious bias? Rushing into ML before analytics is solid usually doubles the effort for half the result.
Reverse ETL: writing processed results back into business systems, e.g. putting a churn-risk score into the CRM or an ad platform so front-line teams use it directly.

6. The six undercurrents: what keeps every stage reliable

Undercurrents are not a stage of the lifecycle; they are work every stage must do:

Security: the principle of least privilege, granting only the access a job needs and revoking it afterwards. The book says plainly that people and organizations are the biggest security weakness, more often than technical flaws.
Data management: following the DAMA framework, it covers data governance (discoverability, metadata, accountability), data quality (accurate, complete, timely), lineage (where data came from and what happened to it), master data management, and data lifecycle management: laws like GDPR and CCPA require deleting personal data on request, and "deleting it everywhere" is an engineering problem.
DataOps: bringing Agile and DevOps practice to data, on three pillars: automation, observability and monitoring, and incident response. The goal is to catch a data error before the business does.
Data architecture: understanding business needs and making design trade-offs between cost and simplicity (expanded in Chapter 3).
Orchestration: scheduling tasks as a DAG, managing dependencies, retrying and alerting on failure. It is not cron: cron only knows the clock, an orchestrator knows dependencies, so if upstream hasn't finished, downstream shouldn't start. Apache Airflow is the classic example.
Software engineering: data-processing code, infrastructure as code, pipelines as code, testing and review. However managed the tools become, the ability to write maintainable code doesn't go away.

Orchestration knows dependencies; cron only knows time. A DAG stops a failure where it happens instead of passing bad data downstream.

Key trade-offs and choices

Table 1 · Time granularity of ingestion and processing: what to use when

Mode	Typical latency	Good for	Cost
Batch	hours to a day	daily reports, financial reconciliation, daily model retraining	stale data; one failure reruns the whole batch
Micro-batch	minutes	near-real-time dashboards, near-real-time warehouses	scheduling overhead; still not true real time
Streaming	under seconds	fraud blocking, real-time recommendations, operational alerts	cost, operations, out-of-order and duplicate handling are hardest; someone must be on call

The test: ask "if data arrives an hour late, what does the business lose?" If you can't name a concrete loss, use batch.

Table 2 · Push, pull and log-based CDC

Approach	How	Pros	Cost
Pull: full snapshot	export the whole table on a schedule	simplest, complete result	slow and expensive at scale; heaviest load on the source
Pull: incremental query	fetch rows updated "since last time" by timestamp	only moves changes, easy to build	needs a reliable timestamp column; misses hard deletes
Push: log-based CDC	read the database change log, forward each change	seconds of latency; captures deletes; no heavy queries on app tables	needs log access and extra components; schema changes need separate handling
Push: app emits events	the application writes events straight to a message queue	clearest semantics, most real time	requires changing upstream code and the upstream team's cooperation

Storage tiering is a trade-off too: hot is fast but expensive, cold is cheap but slow, with retrieval fees on top. Tier by access frequency, and check regularly whether "cold" data is actually queried often.

Why it matters / in real systems

The chapter's framework maps almost one to one onto the modern data stack: generation is MySQL / PostgreSQL application databases and event tracking; ingestion is CDC tools, managed connectors and Kafka; storage is object storage plus a cloud warehouse or lakehouse; transformation is SQL modeling and Spark; serving is BI, feature stores and reverse ETL tools; orchestration and data-quality tools span all of it. When an architecture review asks "is this pipeline sound?", a good answer walks the stages: source rate, ingestion mode and latency, storage tiers, who owns the transformation logic, who consumes the output, then names who owns each undercurrent.

Open-source evidence

Debezium: the poster child of log-based CDC. The project describes itself as "an open source project that provides a low latency data streaming platform for change data capture (CDC)"; it monitors databases and hands every row-level change downstream as an event, with only committed changes visible. This is "push: log-based CDC" from Table 2 in practice. Source: debezium/debezium README, GitHub
Apache Kafka: the backbone of streaming ingestion. It calls itself "an open-source distributed event streaming platform," used by thousands of companies for high-performance data pipelines, streaming analytics and data integration. Debezium itself is built on Kafka and Kafka Connect. Source: apache/kafka README, GitHub
Great Expectations: data quality written as tests. It describes its core concept, Expectations, as "expressive and extensible unit tests for your data": what DataOps automation and observability look like when applied to data quality. Source: great_expectations README, GitHub

Pitfalls and debates

"Pick tools first, figure out needs later." The chapter runs the other way: ask what consumers need and how fresh, then work back to how each stage should work.
"Real time always beats batch." Streaming costs most in long-term operations: out-of-order data, duplicates and replays all need handling. A streaming pipeline for a report read once a day is the most common over-engineering.
"Incremental pulls are all of CDC." Timestamp pulls miss hard deletes and rely on a trustworthy timestamp column; for full change history, read the log.
"Orchestration is just cron." Cron knows nothing of dependencies or retries; if upstream fails, downstream still runs and bad data flows on.
"Storage is just one stage." It is the floor under every stage, and hot/cold tiering decides the bill.
"Data management is compliance's job." Lineage, quality and the ability to delete must be built into the pipeline; retrofitting them is very expensive.
Era limits and debate. Published in 2022, the book is cautious about "streaming first." Managed streaming services have matured since and the cost barrier keeps falling, so some teams now argue for streaming by default. The book also says little about vector data or preparing data for large language models.

Key takeaways

① The data engineering lifecycle is a subset of the data lifecycle: generation → ingestion → transformation → serving, with storage underneath all of it.

② Each stage is a set of questions to settle first, not a tool name.

③ Source systems and ingestion are the biggest bottlenecks: you don't own them, but you must understand their rate, quality and schema evolution, and keep talking to their owners.

④ Storage is tiered hot, lukewarm and cold by access frequency, trading latency for cost.

⑤ Batch is a stream processed in chunks; batch stays the default, and real time needs a business case that justifies its cost.

⑥ Push vs pull: incremental pulls are simple but miss deletes; log-based CDC is low-latency and captures deletes but costs more to operate.

⑦ Transformation is driven by business logic and each step should justify its cost; serving means analytics, ML and reverse ETL, so watch out for vanity projects nobody uses.

⑧ Six undercurrents: security (least privilege; people are the biggest weakness), data management (quality, lineage, deletability), DataOps, architecture, orchestration (not cron), software engineering.