Book Deep-Read · Fundamentals of Data Engineering · Chapter 4
Fundamentals of Data Engineering · Ch 4 · Reis & Housley · 2022
Every dashboard you see at work sits on top of a chain of tools that move, store and crunch data. There are hundreds of such tools, and every vendor claims to be the fastest and cheapest. This chapter doesn't recommend a brand. It teaches you which questions to ask before you pick one.
Plenty of small teams copy what big tech does, build a whole self-run system, and end up with three people fixing machines all day instead of doing the actual work. Meanwhile some very large companies move out of the cloud into their own data centers and save tens of millions a year. The same choice is smart for one company and a disaster for another.
Small team? Use ready-made managed services as much as possible and spend your energy on the business. Only when you are big enough that the bill itself becomes a major problem should you take some of it in-house. One honest cost: the more you lean on ready-made services, the easier life is, and the harder it becomes to leave that vendor.
Choosing technology isn't picking the coolest tool. It is setting the architecture first, then picking the option that saves your team the most effort, can be swapped out, and whose full bill adds up.
Want total cost of ownership (TCO) vs opportunity cost (TOCO), immutable vs transitory technologies, on-prem vs cloud vs hybrid vs multicloud, how to evaluate open-source and commercial products, and when serverless costs more than servers? → Switch to the deep read
You think technology selection is about which tool is stronger. The chapter opens with a different claim: architecture is strategic; tools are tactical. Decide the architecture first (what and why), then choose tools (how). A tool is judged not by how cool it is but by the value it adds to the business, the effort it saves the team, and what it will cost to replace it later.
The last chapter of Part I, "Foundation and Building Blocks." Chapter 3 set out the principles of good architecture; this chapter turns them into "what do we actually buy and run." From Chapter 5, Part II walks through each lifecycle stage in turn. In practice it maps to technology-selection reviews and the budget argument over "should we build this platform ourselves?"
Data tooling exploded through the 2010s: ingestion, storage, transformation, orchestration and BI each have dozens of options with near-identical marketing. The authors see three typical failures: teams fall in love with a tool and back-fill the architecture; small teams copy big-tech self-built stacks and spend their time on operations rather than output; and people look only at the price tag, ignoring labor, migration and lock-in.
The consequences are concrete. A five-person data team running its own Hadoop or Kafka cluster can lose half its capacity to upgrades, scaling and night-time alerts. A three-year bet on one cloud's proprietary service may reveal itself as wrong only when the data has grown too big to move. The chapter offers a set of selection criteria that apply across the whole lifecycle, so choices can be explained and reversed.
Team size and capabilities come first. Small teams, or teams with thinner engineering skills, should lean on managed services and SaaS and spend scarce people on the business. The authors warn explicitly against cargo-cult engineering: seeing a big company publish its architecture and assuming that copying it yields the same results. They have a hundred people maintaining it; you don't.
Speed to market: a data team's value lies in delivering useful data quickly and continuously. The authors favor technology that is good enough and ships fast over endless evaluation in pursuit of perfection. Interoperability: every tool must connect upstream and downstream, so ask first which standard interfaces and formats it supports, such as JDBC/ODBC database connectivity, REST APIs, or open file formats like Parquet. A tool that interoperates poorly turns the connections themselves into a permanent project.
Total cost of ownership (TCO) = direct costs (servers, subscriptions, cloud bills) + indirect costs (operations staff, training, downtime). An open-source license is free, but if it needs two engineers full time, that is easily several hundred thousand dollars a year in hidden spend at US salaries. There is also CapEx vs OpEx: the cloud turns large up-front capital spend into monthly operating spend, buying the flexibility to scale down or walk away at any time.
The chapter's sharper idea is total opportunity cost of ownership (TOCO): picking one stack means forgoing the benefits of every alternative. The authors call technology that is easy to enter and hard to leave a bear trap. If a choice prevents you from moving to something better in a few years, its true cost is far above its invoice. So ask during evaluation: if this fails in two years, how long and how much to replace it?
They also push Chapter 3's FinOps further: cloud bills are dynamic, so engineering, finance and the business must watch cost continuously, the way you watch latency.
The authors split technology in two. Immutable technologies are the time-tested foundation: object storage, networking, servers, SQL, bash, the C language, the x86 instruction set. Transitory technologies arrive fast and vanish fast; front-end JavaScript frameworks are the classic example. They invoke the Lindy effect: the longer a technology has already been around, the longer it tends to keep being around.
The strategy that follows: choose the best tool for today, but build on immutable technologies and keep the transitory layer replaceable. For example, keep data in open formats on object storage; the query engine on top can be A today and B tomorrow without moving the data. The authors advise re-evaluating tools periodically rather than once and for all.
On-premises: buy and run your own machines, plan capacity for peak in advance, and scale in months. Cloud: rent on demand and reach thousands of cores in minutes. But the authors warn that the cloud is not someone else's data center: a lift and shift of an on-prem architecture is often more expensive. You need to rebuild the cloud way, e.g., object storage instead of always-on clusters, and shut things down when done.
They spend real time on cloud economics: providers slice storage and compute into products with different durability and availability and price them like financial derivatives, each with its own terms. Data goes in free and comes out for a fee; egress charges give data "gravity," and the bigger it is, the harder it is to move. Hybrid cloud commonly means analytics in the cloud and core operational systems on-prem. Multicloud lets you pick each provider's best service and reduce dependence on one vendor, at the price of egress fees and multiplied engineering complexity.
On cloud repatriation (moving back out), the authors are restrained: it makes sense only for companies at very large scale whose core product is storage or networking itself. The examples they name are Dropbox and Cloudflare. For almost everyone else, start in the cloud by default.
The principle: build only where it gives you a competitive advantage; hand the rest of the "undifferentiated heavy lifting" to someone else. The buy side splits three ways:
Monolith: one system does many jobs. Simple and easy to reason about, but upgrading one part touches everything. The authors single out the worst form, the distributed monolith: many machines and services on the surface, but shared dependencies that force them to upgrade together, typically many jobs crammed onto one Hadoop cluster and pinned to one version. Modular: snap replaceable components together through open formats (like Parquet) and standard interfaces, then wire them into pipelines with an orchestrator (like Airflow). The authors lean modular overall because it makes each component a two-way door; the cost is that integration and orchestration complexity lands on your team.
Serverless vs servers: serverless bills per invocation and suits small, infrequent, discrete workloads: zero operations, zero cost when idle. But its unit price is far above an always-on machine of equal power, so at high volume the bill overtakes. It also caps execution time, memory and language runtimes. The authors' suggested order: use serverless where you can; once you outgrow it, move to containers plus orchestration (e.g., Kubernetes); manage servers directly last. The key is to model usage first and find the crossover point.
The benchmark wars: the authors warn that vendor benchmarks are often rigged: absurdly small datasets used to test "big data" performance, a tuned product compared with a rival on default settings, speed compared but never cost. The remedy is to test with your own data and your own queries: caveat emptor. Finally, Chapter 2's six undercurrents (security, data management, DataOps, data architecture, orchestration, software engineering) are also a selection checklist: a tool with a crude permission model and no metadata or monitoring hooks loses points however fast it is.
Table 1 · Where it runs: on-prem, cloud, hybrid, multicloud
| Location | Good for | Cost |
|---|---|---|
| On-prem | Stable, large workloads with a mature ops team; compliance that keeps data in your building | Buy for peak in advance; scaling takes months; all hardware and staff on you |
| Cloud | The default for most companies; spiky load; fast experimentation | Lifting an on-prem design is pricier; egress fees make big data hard to move |
| Hybrid | Core systems already on-prem, analytics heading to the cloud | Networking, permissions and ops for two environments at once |
| Multicloud | Need each provider's best service, or must avoid a single vendor | Cross-cloud egress and engineering complexity multiply |
Table 2 · Build, open source, commercial open source, proprietary
| Option | Good for | Cost |
|---|---|---|
| Build | The capability is your competitive advantage | Development, ops and hiring are all yours; highest opportunity cost |
| Self-hosted OSS | You have ops skills and want maximum control and no license fee | License is free, labor isn't; if the project stalls, you inherit it |
| Managed COSS | Managed convenience while keeping an open-source exit | Managed and open versions may diverge; subscription grows with usage |
| Proprietary / cloud-native | Small teams, fastest launch, deep use of one cloud | Deepest lock-in; vendor sets pricing and roadmap |
Table 3 · Compute shape: serverless, containers, servers
| Shape | Good for | Cost |
|---|---|---|
| Serverless | Infrequent, bursty, discrete small tasks; light event-driven processing | High unit price, overtakes at volume; limits on duration, memory, runtime |
| Containers + orchestration | Past serverless limits; need portability and elasticity | Kubernetes is an operational discipline in its own right |
| Servers directly | Steady heavy load; special hardware or deep tuning | Capacity planning, patching and failures are all yours |
Rule of thumb: first ask "is this capability our competitive edge?", then "does anyone on the team own its operations long term?", and finally estimate usage to find the cost crossover. Three noes means buy the managed service.
This chapter is the question list for a technology-selection review. Asked "why Snowflake instead of our own Spark cluster?", a good answer doesn't quote benchmarks. It states how many people the team has, how tight the deadline is, what replacing the tool in two years would cost, whether the data sits in an open format or a vendor format, and which side of the cost crossover usage falls on. The modern data stack (managed ingestion like Fivetran, Snowflake or BigQuery as the warehouse, dbt for transformation, Airflow for orchestration) is essentially the modular practice of "small teams buy as much as possible and connect through open interfaces."
Industry and open-source evidence
① Architecture is strategic, tools are tactical: architecture first, then tools; judge by business value, team effort and the cost of replacing.
② Small teams favor managed services and SaaS; beware cargo-cult copying of big tech; good enough and fast beats perfect.
③ TCO = direct costs + indirect costs like operations staff; TOCO reminds you that one path forecloses others, so avoid the easy-in, hard-out "bear trap."
④ Build on immutable technology (object storage, SQL, open formats) and keep the transitory layer replaceable.
⑤ Default to the cloud, but use it the cloud way; egress fees give data gravity; repatriation fits only big companies whose core business is storage or networking.
⑥ Build only for competitive advantage; OSS, COSS and proprietary trade control for ease, subscriptions for backing, and lock-in for speed.
⑦ Modularity keeps each component reversible; beware the distributed monolith; serverless → containers → servers, guided by the cost crossover.
⑧ Trust only benchmarks on your own data; the undercurrents (security, data management, DataOps, orchestration) belong on the selection checklist too.