The Future of Data Engineering

In one sentence

You'd think that as tools become push-button simple, data engineers get replaced. It's the opposite: what gets automated away is the chore of babysitting infrastructure, while the lifecycle itself (generation, storage, ingestion, transformation, serving, plus undercurrents like security, data management, and DataOps) doesn't go anywhere. The chapter is the authors' set of bets on the future: complexity keeps falling; cloud services grow into an interoperable "data operating system"; data engineering becomes more "enterprisey" and management-heavy; job boundaries get redrawn; and the field moves from the batch-centric modern data stack toward a streaming, application-fused live data stack.

Glossary

Where it sits

The book's last chapter, closing Part III right after Chapter 10, "Security and Privacy." The first ten chapters cover how things are done today; this one is the authors' forecast for the next few years built on those principles. In practice it maps to architecture and career questions like "what should our next data platform bet on?"

What problem this chapter tackles

Data tools turn over at a frightening pace. Around 2010 the star was the self-hosted Hadoop cluster; a few years later, Spark; a few years after that, the "modern data stack" assembled from Snowflake or BigQuery plus dbt and Fivetran; and then lakehouses and streaming databases. An engineer who only learns tools has to start over every three to five years. The chapter asks: amid all this churn, what changes, what doesn't, and where should people place their bets? Get it wrong and you lose both ways: you either cling to an outdated stack or chase every new buzzword, and never actually get good at reliably delivering data to the people who use it.

The core ideas, explained

1. The lifecycle isn't going away; the tools are

The authors' first claim is the foundation of the whole book: the lifecycle and its undercurrents are stable abstractions, and tools are just how a given era implements them. In the Hadoop days you still ingested, stored, transformed, and served; move to a cloud warehouse and not one stage disappears. What changes is who performs each step and with what. The authors even argue that data engineering will grow in importance: the more a company depends on data, the more it needs people who keep that chain healthy.

2. The decline of complexity and the rise of easy-to-use tools

Early big data meant buying your own servers, deploying Hadoop, tuning JVM settings, and replacing failed disks at 2 a.m. Only large companies could afford a team like that. Managed services and serverless warehouses absorb that layer of complexity: you pay per byte scanned or per unit of compute time and never think about the cluster, and ingestion comes as ready-made managed connectors. The result is that big data capabilities get democratized; a team of a few people can handle terabytes. The authors' inference: data engineers will move up from low-level operations and spend their time on architecture, data quality, and business value. The work doesn't disappear; it moves up a layer.

3. The cloud-scale data OS and improved interoperability

This is the chapter's most vivid analogy. An operating system gives applications standard services (a file system, process scheduling, device drivers) so each app doesn't have to build its own. The authors see cloud data services growing into the same kind of layer: object storage is the "file system," query engines and processing frameworks are the "applications," and what lets them cooperate is standardized interfaces: common object storage APIs, open file formats like Parquet and Avro, a new generation of metadata catalogs replacing older ones such as the Hive Metastore, and orchestrators that grow from "just run the jobs" into platforms aware of data assets, lineage, and quality. The better the interoperability, the cheaper it is to swap out an engine, and the less data has to be copied again for each new tool.

4. "Enterprisey" data engineering

Among engineers, "enterprisey" is usually an insult: tedious and bureaucratic. The authors flip it. The easier the underlying tools become, the more data management, DataOps, governance, and data quality, the stuff once dismissed as "too enterprise," become the core of a data engineer's job. Once the machines look after themselves, the remaining hard problems are: who owns this table, are the metric definitions consistent, will we notice when something breaks, and who is allowed to see it.

5. Titles and responsibilities will morph

The authors expect job boundaries to keep shifting, with more overlap among data engineering, analytics engineering, ML engineering, and software engineering. Software engineers will be expected to know more about data (streaming, data modeling), while data engineers move closer to the business and the application. Titles may change; the lifecycle work won't shrink.

6. Beyond the modern data stack, toward the live data stack

The modern data stack (MDS) was the dominant recipe around 2020: managed connectors batch-load data into a cloud warehouse, SQL transforms it in place (ELT), and BI dashboards sit on top. It's cheap and plug-and-play, but at heart it's batch: data often lags by hours to a day, and it mostly serves "humans reading reports." The authors' bet for the next stop is the live data stack: data arrives as streams, is transformed on the stream, lands in real-time analytical databases that answer in seconds, and directly powers application features such as recommendations, fraud checks, and dynamic pricing. With it comes the fusion of data and applications: apps aren't just sources of data, they consume analytical results in real time, and a tighter feedback loop forms between applications and ML, with models adjusting continuously to what users just did. The authors admit that streaming data modeling and tool maturity are still catching up.

7. Dark matter data and the spreadsheet

The authors point out a fact engineers tend to overlook: the most widely used data platform in the world is the spreadsheet. A huge share of real decisions runs on spreadsheets scattered everywhere, like dark matter in the universe: invisible, yet enormous in weight. They predict a new class of tools that keeps the spreadsheet's interactive feel while sitting on top of cloud analytical engines, so business users can work directly with massive datasets. The lesson for data engineers: don't serve only the people who write SQL.

Key trade-offs and choices

Table 1 · Modern data stack vs live data stack

Table 2 · Once complexity falls, where to place your effort

The rules of thumb are simple: freshness requirements decide batch vs streaming. Go streaming only when the business must act on the data within minutes; otherwise batch is cheaper and easier to debug. Differentiation decides build vs buy: any layer that isn't your competitive edge, hand to the cloud.

Why it matters / in real systems

Asked in an interview "how would you build a data platform from scratch today?", this chapter gives you the skeleton: lay out the five lifecycle stages and the undercurrents first, then choose tools layer by layer; keep data in object storage with open formats and a unified catalog so you aren't locked into one engine; default to batch and open a streaming path only where there is a real need for seconds-level freshness (fraud, recommendations); and spend the people you save on data quality, metric definitions, and access control. The same goes for a career: learn tools, but hang them on the lifecycle map, so when the next generation arrives it's just a part swap.

Pitfalls and debates

• "Push-button tools will make data engineers obsolete." What disappears is the ops grind; lifecycle and data management work only grows.
• "Real-time is always better than batch." Most reports don't need seconds-level freshness; streaming costs more and is harder to debug, and without a business action behind it, it's waste.
• "Managed means no lock-in." You trade away ops for billing and migration risk; open formats are the hedge.
• "The modern data stack is the end state." The authors are explicit that it is one phase; batch's ceiling is freshness.
• The limits of prediction. This chapter is opinion rather than argument, written in 2022. Since then lakehouses and open table formats have indeed spread, bearing out the interoperability call; but the shifts brought by the large-language-model wave (unstructured data, vector search) are barely touched here, and the "live data stack" has not displaced batch across the board as forecast.

Key takeaways