Data Integrity

In one sentence

This chapter inverts the problem: data integrity is the means, data availability is the goal — users don't care how many replicas you hold or whether your checksums match, only whether the correct data is there when they need it. The things that can destroy it come in 24 combinations, and no single measure covers them, so Google's answer is defense in depth with three lines: soft deletion, backups and restores, and out-of-band validation. The two lines that sting most: replication is not recoverability, and a backup whose restore was never verified is not a backup.

Glossary

Where it sits

Chapter 26 of the SRE book, in the data-reliability cluster of Part III (Practices), right after Chapter 25 on data processing pipelines — fittingly, since pipelines are one of the leading ways data gets destroyed in bulk. The authors, Raymond Blum and Rhandeep Singh, spent years on Gmail and Google Drive, systems where losing one item counts as an incident. In the real world it maps to anything that holds user data for a long time: mail, cloud storage, SaaS, payment ledgers, warehouses.

What the chapter is trying to solve

Everyone knows how to write an availability SLO: 99.9% of requests succeed. But here is the awkward question: if the request succeeded and the response was empty, was that available? A system that has just emptied a user's mailbox can show green across the dashboard — latency normal, error rate zero. Availability metrics measure whether the service is up, not whether the data is right.

Traditional operations answers this with "we have backups". The chapter exists to expose the three holes in that sentence: the backup may never have been verified (when GitLab lost its primary in 2017, none of the 5 backup and replication techniques it had deployed actually worked); the backup may not keep up with the damage (of the roughly 600,000 audio tracks Google Music deleted by mistake in 2012, some 161,000 were gone before they were ever backed up); and nobody may notice at all, so that by the time you look, every generation inside the retention window is already poisoned.

The cloud amplifies all three at once: high update rates, continuous delivery changing code daily, services calling services, users who expect 24×7 with no maintenance window, and data that accretes for a decade or more. The cost of not solving it isn't "slower". It's the user's things are gone, permanently.

The core, properly

Redefine it: integrity is the means, availability is the goal

The textbook definition — data stays accurate and consistent over its lifetime — is useless for engineering, because it never tells you how much is enough or what to spend. This chapter swaps the yardstick for data availability from the user's point of view: can the service put the correct data in the user's hands when they need it? Two consequences follow immediately:

• Every checksum matches and every replica is present, but the application reads back something wrong → still a failure. Integrity was only ever the means.
• Nothing is actually lost, but restoring takes three days → indistinguishable from loss to the user. Recovery time is part of availability.

So the SLO should read "data is correctly readable at all times, and any corruption event is recovered within X hours", not "we back up nightly". And scale turns small probabilities into daily work: a system holding 10 billion objects, each with a one-in-a-million annual chance of silent corruption, still produces about 10,000 bad objects a year for someone to deal with.

Why there is no silver bullet: 24 combinations

The book decomposes "how data goes bad" along three orthogonal axes:

• Root cause (6): user action, operator error, application bug, infrastructure defect, hardware failure, site disaster.
• Scope (2): wide (the whole dataset) vs narrow (one user, one row).
• Rate (2): big bang (everything at once — at least you find out immediately) vs slow and creeping (a trickle a day, discovered long after the retention window has rolled).

Multiplied out: 6 × 2 × 2 = 24 combinations. The insight isn't the number, it's that every single defense covers only a few cells. Replicas cover hardware and site disasters, soft deletion covers accidental deletes, backups cover bulk corruption — but backups do not cover creeping loss: by the time you notice, the bad data has been copied into every generation inside the retention window. The only thing that covers that cell is noticing early. Which is why the answer has to be defense in depth, not "buy a better backup product".

Line one: soft deletion

The mechanism is plain: a delete request doesn't delete. It stamps the record "deleted" with a timestamp, reads filter it out, and real removal happens when the window expires. What users see is a trash can and revision history.

Why does it rank first? Because it catches the most frequent class of incident: accidental user deletes, malicious wipes after an account hijack, one ops script run against the wrong target. These are far more common than a data center burning down, and recovery costs essentially nothing — flip a flag, done in seconds, orders of magnitude faster than dredging a backup. On frequency × cost-to-recover, this is the highest-return layer you can build.

Two engineering details. Window length is a genuine trade-off: too short and you can't save people (users often notice days later), too long and you pay storage while colliding head-on with the demand for real deletion; the common industry answer is weeks (Gmail's 30-day trash is exactly this thinking). And lazy deletion: actually scrubbing data out of every replica, cache, index and backup can take weeks to converge at scale — meaning deletion is inherently a process, not an instant.

Be clear about what it does not cover: soft deletion handles deletion. It does nothing about data being mangled — an app bug that miscomputes a field and writes a dirty value leaves nothing in the trash can.

Line two: backups and restores — the backup has no value, the restore does

This is the heart of the chapter, and it has three counterintuitive points.

(1) Replication is not backup. The book is blunt: "Replication and redundancy are not recoverability." A database that auto-syncs replicas guarantees that one bad delete or one corrupted row is pushed to every copy, usually before you can react. Put the magnitudes side by side and it's obvious: replication propagates in milliseconds to seconds, while humans notice in minutes to days. That gap of several orders of magnitude is the whole reason replicas cannot save you.

(2) A backup is not an archive. The book's test is a single question: can it be loaded back into the application? In its own words, "the most important difference between backups and archives is that backups can be loaded back into an application, while archives cannot." Backups exist so you can load and keep running, so format, schema and dependency versions must line up with the current system; archives exist for compliance and audit and may simply not go back in. Mistaking one for the other is a fixture of incident reviews.

(3) Derive the backup strategy from the recovery requirement. Don't start with "how often do we back up". Start with three questions: how much data can we lose (RPO) → sets frequency and whether you replay transaction logs; how long can we be down (RTO) → sets medium and location; which of the 24 cells must this cover → sets isolation. A snapshot inside the same cluster cannot survive "the delete was synced over", so there has to be a copy that is offline, heterogeneous, and remote. Google's choice at this layer was tape — absurdly slow, but physically disconnected from the live system, so no online mistake can reach it. That is what saved Gmail in 2011.

Be honest about cost: full backups are simplest but storage and time grow linearly with the data; incrementals and differentials save space but restores must be replayed in order, and the longer the chain, the more likely it breaks. Real systems land on "periodic full + incrementals + keep N generations", and how many generations is decided by how late you might plausibly notice corruption — which walks you right back to the creeping-loss cell.

Line three: out-of-band validation

The first two lines are both after the fact: they let you get data back, provided you know it went bad. The third line exists purely for the knowing. You run a set of validators independent of the main service that continuously reconcile the data — are references dangling (does this track's metadata still point at real audio?), do cross-store ledgers balance, do replicas agree, do business invariants hold?

Why it must exist: the faster and more automated the system, the less time it takes to finish propagating the damage — while the time humans take to notice has not improved at all. A validator's value is compressing detection latency from "wait for a user to complain" (days) to "the next validation pass" (minutes to hours) — which decides whether a clean backup still exists inside the retention window. Google Music is the cautionary case: the problem surfaced only because an engineer investigated one track that wouldn't play, and by then roughly 161,000 tracks were past saving.

Three practice notes: it must be out of band — an independent read path and deployment, never sharing the possibly-buggy code of the system under test, or the same bug fools service and validator alike; the validator itself must be monitored — one that silently stopped is worse than none, because it supplies false confidence; and alerts must be actionable — "N records don't reconcile" is useless, you need which records and since which version.

Make recovery real: rehearse it, don't trust it

The step most often skipped: assume any recovery procedure that has not been rehearsed is broken. SRE treats recovery as a product feature under continuous test — periodic end-to-end exercises (Google's DiRT disaster recovery testing is exactly this idea) whose acceptance criterion is "the data was genuinely loaded back into the application and business validation passed", not "the backup job returned success". The chapter ties this back to a few general SRE principles: beginner's mind (don't assume, go look), trust but verify, hope is not a strategy, and defense in depth itself.

Trade-offs and how to choose

Table 1 · The three lines: what each catches, how fast, at what cost

Table 2 · Replication / backup / archive: three things routinely conflated

Table 3 · Choosing medium and location (derived from the recovery requirement)

Do the arithmetic on recovery: RTO is often physics, not policy. Pulling 100 TB back from a remote region at a saturated 10 Gbps takes roughly 22 hours. If your RTO says "4 hours", no amount of extra storage spend fixes that — the architecture has to change (parallel per-shard restore, a nearer copy, hot data first). An RTO nobody has done the arithmetic on is a wish.

Why it matters / in real systems

This chapter is required reading for data systems because it turns a platitude everybody can say — "we have backups" — into an auditable checklist: how many days is the soft-delete window? When was the last full restore rehearsal, and how long did it take? Which invariants do the validators cover, and when did they last complete a successful pass? That framing is now everywhere in the industry: object storage versioning and object lock are essentially "soft deletion plus a tamper-proof backup", warehouse time travel is soft deletion in another guise, data observability products are the commercialisation of the third line, and restore rehearsals are a hard compliance requirement for banks and cloud providers. In an architecture review or an interview it hands you three sentences that cut straight through: multi-AZ is not a backup, backup success rate is not restore success rate, and an unmonitored validator is no validator at all.

Pitfalls and open questions

• Treating replication, RAID or multi-AZ as backup: misconception number one. They defend against hardware and sites; against bad deletes and bad writes they are not merely useless but actively accelerating.
• Monitoring backup success but not restore success: a job returning success says nothing about whether the data goes back in. GitLab's lesson was precisely "silent failure, and the notification never went out either".
• Rehearsing only on a small database: real restores are bound by bandwidth and ordering dependencies, which a toy dataset never exposes. Rehearse against realistic volumes, or at least publish the arithmetic.
• A validator that quietly stopped, or believing validation can replace backups: the first supplies false confidence, the second confuses the division of labour — validation compresses detection latency and recovers nothing.
• The real tension between soft deletion and the right to be forgotten: soft deletion turns deleting into a process lasting weeks, while privacy law wants demonstrable, complete removal. There is no clean answer; write the window and the lazy-deletion convergence time explicitly into the design and the privacy statement. The book predates GDPR, and its age shows most here.
• The 2016 threat model has no ransomware in it: an attacker's standard play today is to delete your backups first and encrypt your data second. The most important post-publication addition is immutability — object lock / WORM backup buckets, plus backup credentials outside the production identity system. The "offline tape" instinct was right; today's equivalent is "immutable plus privilege-isolated".
• Don't treat "24 combinations" as doctrine: its value is prompting you to audit coverage cell by cell, not taxonomic precision.

Takeaways