Security and Privacy

In one sentence

You'd think data security comes down to stronger technology: better encryption, a pricier firewall. In fact most breaches come from people and process: credentials committed to code, admin rights for everyone, a storage bucket set to public. The chapter builds three ideas. Security is an undercurrent of data engineering that runs through every stage of the lifecycle. Defenses come in three layers, people → process → technology, and the weakest link is usually people. And what works is a daily habit, not a compliance performance, built on least privilege, collecting less sensitive data, prompt patching, and real backups.

Glossary

Where it sits

The first chapter of Part III, "Security, Privacy, and the Future." The first nine chapters walked the lifecycle (generation → storage → ingestion → transformation → serving), and Chapter 2 put security first among the six undercurrents. This chapter pulls that cross-cutting thread out on its own and leads into Chapter 11, "The Future of Data Engineering." In practice it maps to warehouse permission design, secret management, cloud account governance, and compliance regimes like GDPR.

What problem this chapter solves

Picture an e-commerce company with tens of billions in annual sales. Its warehouse holds orders, phone numbers, and delivery addresses for 200 million users. All 40 analysts and data scientists are warehouse admins, because it's easier. Airflow jobs and notebooks contain database passwords in plain text. A bucket of export files is publicly readable so partners can download it easily. Backups? "The cloud provider must have some." It all runs fine until one day:

• One phish takes everything. Fool any single person out of their account and the attacker gets the whole warehouse.
• One code repo is one key. If the repo is made public or forked, the passwords leak with it.
• One setting is one breach. Automated scanners find public buckets. No "hacking" required.
• One ransomware hit is one shutdown. Without independent backups, the only choices are paying up or losing the data.

On top of the direct damage comes regulation. The EU's GDPR caps fines for serious violations at 4% of global annual turnover or €20 million, whichever is higher. California has the CCPA, and health data falls under HIPAA. The authors' warning: data engineers handle exactly the data these laws protect, so you can't assume security is someone else's problem.

The core ideas, in depth

The authors organize the chapter around people, processes, and technology. The order is itself the point: no technical defense stops an employee who tells a stranger their password.

1. People: the power of negative thinking, always be paranoid

The power of negative thinking. The authors want you to rehearse failure while designing every pipeline: if this data leaked, what would happen, and who could get it? From that comes the chapter's most practical rule: the best way to protect sensitive data is not to ingest it in the first place. If a retention analysis doesn't need national ID numbers, block or mask them at ingestion rather than collecting them and figuring out how to hide them later.

Always be paranoid. When someone asks for credentials or a data export by email, phone, or chat, verify through a separate trusted channel, even if the request seems to come from an executive. Social engineering (tricking people's trust rather than exploiting system flaws) works precisely because it walks around every technical defense.

2. Process: security theater vs. security habit, active security

Security theater is what the authors argue against: polished documents for the compliance audit, a security pledge signed once a year, a ticked checklist, and nothing actually done differently. A security habit builds security into the everyday: check permissions in code review, check configs before release, review incidents regularly.

Active security: don't copy a static checklist. Keep researching the threats aimed at your organization: which systems you run, where your people tend to get phished, what has recently happened to similar companies. Then adjust.

3. Process: the principle of least privilege

The principle of least privilege: every person and every system gets only the permissions needed for the work at hand right now, and those are revoked when done. The authors call out two common violations: handing out admin roles because it's easier, and doing everyday work as root. The principle applies to data too. An analyst who can query order aggregates doesn't thereby get to see phone numbers. Grant specific columns or rows, or a masked view.

What if someone really needs sensitive data for a while? The book's answer is a break-glass process, like smashing the case to reach the fire axe: request, approval, a time limit, a full audit trail, and automatic revocation when time is up.

4. Process: shared responsibility in the cloud, always back up

The shared responsibility model: the cloud provider secures the cloud itself (data centers, hardware, the virtualization layer). You secure what's in the cloud: account permissions, network rules, whether buckets are public, whether data is encrypted. The authors stress that many cloud data breaches come from customers' own misconfigurations, not from the provider being compromised.

Always back up your data. Data gets lost to accidental deletes and hardware failure, and it gets encrypted by ransomware. The authors call for regular backups and regular restore drills. A backup you've never restored from doesn't count.

5. Process: an example security policy

The book offers a minimal policy you can adopt as is, in three parts. Protect credentials: use single sign-on (SSO) and multi-factor authentication (MFA, which asks for a second factor such as a phone code on top of the password) wherever possible. Use cloud IAM roles instead of long-lived keys. Keep secrets in a secret manager, and never embed them in code. Protect devices: central device management, remote wipe for lost laptops, automatic screen lock. Software updates: keep browsers, operating systems, and dependencies current.

6. Technology: patches, encryption, monitoring, network

Patch and update. Operating systems, databases, and open-source dependencies all ship vulnerabilities. Putting off upgrades leaves a known door open.

Encryption is not a magic bullet. Encryption at rest and in transit is now the baseline, on by default in most cloud services. But the authors point out that it only covers a stolen disk or a tapped link. It does nothing against someone who logs in with valid credentials. When an employee account is stolen, the data is decrypted for the attacker as a matter of course.

Logging, monitoring, and alerting. The authors name four kinds of signal. Access: who logged in when, from where, and what they queried. Resources: abnormal spikes in disk, CPU, or memory, which may mean cryptomining or data exfiltration. Billing: a sudden jump in the cloud bill is often the first sign of an intrusion. Excess permissions: permissions and accounts that go unused for long stretches should be reclaimed.

Network access. Don't expose databases directly to the public internet. Shrink the attack surface with IP allowlists, private networks, and firewall rules, and open admin ports like SSH only to specific sources.

Low-level security and internal research. Engineers often spot a system's weak points before the security team does. The authors encourage a culture where people report a suspicious config or a vulnerability instead of quietly working around it.

7. Privacy: more than not getting robbed

Privacy asks whether data should be used this way at all. Beyond collecting less, the book's earlier chapters add two requirements. Mask or tokenize PII (tokenization swaps the real value for a meaningless stand-in). And build in the ability to honor regulatory requests such as "delete this user's data." When copies are scattered across raw, cleaned, backup, and downstream layers, deleting everything is much harder than it sounds.

Key trade-offs and choices

Table 1 · Security theater vs. security habit

Table 2 · What each control stops, and what it doesn't

None of these is enough alone. Combined, they make defense in depth. Granularity is a trade-off too. Database-level grants are the easiest and the coarsest. Column-level, row-level, and dynamic masking fit least privilege best, at the price of a growing pile of policies that someone has to maintain. A common practice is to tier data by sensitivity (public, internal, confidential, regulated: PII, payments, health) and govern each tier more tightly than the one below.

Why it matters / real systems

Asked in an interview to "design a data platform that stores user behavior and PII," a strong answer lands like this. Detect and tokenize phone and ID numbers at ingestion. Partition the warehouse by sensitivity tier: analysts get masked views by default, and row-level detail goes through time-boxed elevation. Jobs run as IAM roles rather than with long-lived keys, passwords live in a secret manager, and CI scans commits so keys never land in the repo. Buckets are private by default and databases sit on private networks. Alerts watch for unusual logins, unusually large exports, and billing spikes. Backups live in a separate account and get a restore drill every quarter. Mainstream platforms have the matching features: Snowflake and BigQuery support column-level access control and dynamic masking, and AWS Lake Formation offers table-, column-, and row-level grants.

Pitfalls and debates

• "We're on the cloud, so security is the provider's job." Under shared responsibility, configs, permissions, and data are yours.
• "Encryption is on, so we're safe." Encryption doesn't stop someone logging in with valid credentials. It's the baseline, not the finish line.
• "Collect everything now, decide later." Every extra PII column adds attack surface and compliance duties. Not collecting is the cheapest protection.
• "Least privilege slows the team down." Fair, which is why it needs break-glass and self-service requests, not a retreat to admin-for-all.
• "We passed the audit, so we're secure." Compliance is the floor. Perfect checklists without the habits are exactly security theater.
• "We have backups." A backup never restored can't be trusted, and backups in the same account as production can get ransomed along with it.
• Limitations. The chapter is deliberately high level, closer to a checklist from a data engineer's point of view than a deep dive into threat modeling or cryptography. The book came out in 2022, and practices like zero-trust networking and data security posture management (DSPM) have spread since. The people → process → technology ordering still holds.

Takeaways