BOOK DEEP READ · CONTINUOUS DELIVERY · CH 12

Managing Data

Continuous Delivery · Ch 12 · Jez Humble & David Farley · 2010

What is this chapter about?

Behind every app you use sits a database holding your orders, your messages, your balance. Earlier chapters covered how a new version goes live and how you take it back if it goes wrong. This chapter is about the part you cannot take back: a new version usually needs to change how the data is stored, and data, unlike code, has no "previous version" to swap back to.

An analogy

Think of the database as the house numbers on a street. Shipping a new app version is like changing a shop's sign — hang the wrong one and you take it down again, five minutes' work. Changing the database is like renumbering the whole street: the courier's address book, everyone's ID cards, the delivery platform's records all still point at the old numbers. Rip every old number plate off overnight and the next day the whole street's parcels go astray — and the parcels already sent to the new numbers cannot be recalled.

Why the old way was hard

The old way went like this: on release night someone who knew the database logged in and typed a few commands by hand off a scrap of paper. And when it went wrong? There was usually exactly one answer — restore the whole database from last night's backup. That sounds safe, but it means wiping out every order, payment and message anyone made since last night. So teams preferred not to change anything, hoarding six months of changes into one big release, then spending a nervous Saturday night at 3 a.m. pushing it.

What fixes it

First, write every database change as a numbered work order, stored in version control next to the code. The database remembers "I am at number 27", and on release it compares and applies 28 through 31 in order. Who changed what is on the record, and the same work orders have already been rehearsed many times in test environments before they reach production.

Second, work orders only add, never demolish. To retire a field, add the new one, copy the data across, and leave the old one in place — so if you need to go back, everything is still there.

Third, and this is the key move: keep two number plates up for a while. There is no longer a single moment of switchover. Instead it becomes several small steps — hang the new plates first and let old and new coexist for a period; once everyone's address book has been updated, take the old plates down at your leisure. Any one step going wrong means stopping there and stepping back, not tearing up the whole street.

And then there is test data

Plenty of teams take the shortcut of copying the entire live database into their test environment. It looks like the most realistic option and it is the most damaging one: it is huge and slow, it is full of real people's private information, and anyone can change it — someone deletes a record today and tomorrow somebody else's test fails for no visible reason. The sturdier approach is to let each test create the handful of records it needs and clear them away afterwards.

The cost you cannot dodge

Keeping two number plates up means a simple change now takes several separate releases, and for a while the code has to serve both the old and the new shape — more steps, more fuss, in exchange for being able to stop and step back at every one of them.

The one-line takeaway

Code can be swapped back to the previous version; data cannot. So: write every database change as a numbered, versioned script that has been rehearsed across every environment; only add, never demolish; and split a structural change into several steps where old and new coexist, so every step can stop and step back.

Want the mechanics — how migrations get numbered, how the six steps of expand-contract actually run, what data each pipeline stage should use? → Switch to the deep read

In one line

You think the hard part of releasing is the code; in fact the thing you cannot take back is the data. This chapter takes the "build once, deploy anywhere, roll back at will" machinery of the previous eleven chapters and runs it past the database, where half the assumptions collapse: applications are stateless and can be swapped wholesale, but a database carries state, can only be changed in place and incrementally, and time runs in one direction. The answer comes in three parts — put the database under version control and migrate it with numbered incremental scripts, make migrations non-destructive and decouple them from application deployment, and give every pipeline stage the smallest test data that fits its purpose.

Glossary

Schema: the database's definition of what the data looks like — which tables exist, what columns they have, of what type, under what constraints.
DDL / DML: statements that change structure (CREATE / ALTER / DROP) are DDL; statements that change data (INSERT / UPDATE / DELETE) are DML.
Migration: the small script that takes the database from one version to the next, usually shipped as a pair — a forward up and an undoing down.
Reference data: the static lookup data an application cannot run without (countries, currencies, status codes). It belongs to the structure and is versioned along with the schema.
Lock: what a database takes to hold other readers or writers off while it keeps things correct. Some ALTER TABLE statements hold one for a long time, which amounts to an outage.
Backfill: filling in historical rows for a newly added column — normally run in batches, and often hours of work over a few million rows.
Dual write: writing the same data to both the old and the new place during a transition, so both stay current.
Backward / forward compatibility: the new application version can read what the old schema wrote (backward); the old application version still runs against the new schema (forward). You only have a rollback if both hold at once.
Blue-green deployment: keep an identical spare environment, install the new version on the idle one, verify, then switch traffic (Ch10). The catch is precisely that the two environments usually share one database.
Fake: an in-memory stand-in used in place of the real database so unit tests run in milliseconds.
RTO / RPO: the two hard numbers of disaster recovery — how fast service must be restored, and how much data loss is tolerable. Roll back with a backup and what you lose is your RPO.

Where it sits

This chapter belongs to Part III of Continuous Delivery, "The Delivery Ecosystem", directly after Ch11 on infrastructure and environments. It answers the gap Ch10 left wide open: blue-green and canary make code releases revocable, but what about the database the two environments share? It is also the last piece of Ch2's "everything under version control" — code went in, environments went in, and the database has to go in too. Ch13 (components and dependencies) follows. In today's terms it maps onto Flyway, Liquibase, Rails and Django migrations, gh-ost, and every zero-downtime schema-change practice in use.

The problem it solves

First, see why this problem is unlike the previous eleven chapters. Applications are stateless; databases are not, and that single difference destroys three properties you have come to rely on:

"Build once, deploy anywhere" does not hold. The same application artifact drops unchanged into test and production; a database cannot. Production holds real data, so you can only change it in place and incrementally, and every environment starts from a different state.
"Roll back = redeploy the previous artifact" does not hold. Swapping back to the old application is a lossless, seconds-long operation. Taking the database back is a dilemma: run the reverse script (but the data in the dropped column is already gone), or restore a backup (which throws away every write since the backup point — that is your RPO, discarded).
"Releasable at all times" gets stuck on structural change. Running ALTER TABLE against a table of a few hundred million rows traditionally rebuilds the whole table under a lock — tens of minutes to hours, with writes blocked throughout. So "add a field" ends up queued for the twice-yearly downtime window.

What happens if you do not fix it is concrete: the database becomes the one step in the whole pipeline still done by hand, from one person's memory, off a scrap of paper. The previous eleven chapters automated build, test and environments, and release night is still a DBA typing SQL into production that nobody else will ever see. So the team stops releasing often and falls back to large, infrequent, high-risk batches — the very thing the book's first chapter set out to dismantle.

Fig 1 · The asymmetry between code and data: rolling back an application only swaps in a package that still sits in the artifact repository, while "rolling back" a database costs you either the column you just dropped or the data you just accepted.

The core, explained

1. Put the database under version control: initialize, then migrate incrementally

The starting point extends Ch2's rule: everything about the database must be rebuildable from version control. That splits in two. Initialization: scripts that create the database, the schema and the reference data, so anyone on any machine gets an empty but working database with one command. Incremental change: from then on, every structural change is not "go in and edit it" but a new numbered migration script committed to version control.

The mechanism is plain, and it is exactly what pulls the database into the pipeline: the database stores a version table recording which number it is at. The deployment tool reads that number (say 27), compares it with the scripts in version control (up to 31), applies 28 through 31 in order, and writes 31 back. Each script ships with a reverse down for downgrades. The tools of the book's era were dbdeploy and LiquiBase; Rails migrations are the best-known implementation of the same idea, as is today's Flyway.

The payoff is not the word "automation" but this: the same batch of migration scripts has already run many times — on developer machines, in the commit stage, in acceptance environments, in staging — before it reaches production. Release night does not execute SQL nobody has seen; it executes a script that already succeeded twenty times today. Database change stops being a one-off event and becomes a rehearsed routine.

Fig 2 · Versioned database migration: a chain of numbered scripts plus a version table inside the database form a replayable, auditable state machine, and deployment does exactly one thing — take me from 27 to 31.

2. Rolling back: three routes, and one hard rule

This is the hardest and most valuable section of the chapter. The book lists three ways to take a database back, and their costs differ enormously.

Route A: back up before migrating, restore if it goes wrong. The plainest option and most teams' default. Two fatal flaws: data loss — restoring to the backup point evaporates every user write since; and slowness — restoring a terabyte-scale database is typically a multi-hour job, far beyond any respectable RTO. It only works when you have a downtime window, a small database, and can accept losing that data.

Route B: write a reverse (down) script. It can undo structure but not information: if 028_up dropped the legacy_addr column, 028_down can add the column back but cannot refill what was in it. Which forces the rule most worth remembering here —

Migrations must be non-destructive. To remove a column, do not drop it in the same migration: first copy its data into a backup table (or simply leave the original column in place, unused), and only after the new structure has run cleanly for several release cycles use a separate migration to actually drop it. Only then does the reverse script have something to restore. The other face of the same rule: a rename is a drop plus an add — renaming addr to address is a destructive change as far as the database is concerned, and the old application version will fail to find the column immediately.

Route C: never create the moment that needs a rollback. That is the most far-sighted passage in the chapter, and the subject of the next section.

3. Decouple deployment from migration: the compatibility window and expand-contract

Here is the answer to the question Ch10 left open. The two sides of a blue-green deployment normally share one database, so at the moment traffic switches, the old and new application versions are necessarily both talking to the same schema. If the deployment is to stay revocable there is only one way out: make the schema compatible with both versions at once. The book puts it as keeping database changes and application changes independent and mutually compatible; the practice was later named ParallelChange / expand-contract by Danilo Sato and Martin Fowler, and is now the default answer for zero-downtime change.

The idea is to break the single-step switchover into a series of small steps that can each stop and step back, preserving a compatibility window in the middle during which both application versions run correctly. Take "split an address from one field into several structured columns":

Fig 3 · Expand-contract: do only additive work first to open a compatibility window, complete dual writing, backfill and the read switch inside it, and make the one irreversible deletion only after production traffic has proven the new path.

The thing to notice about the six steps is that they ship separately: each is an independent, revocable deployment, and consecutive steps may be hours or weeks apart. Steps 1–3 are purely additive, costing only disk and a little write amplification; step 6 is the only irreversible operation, and it happens long after real traffic has exercised the new path. Note too the "nullable, no default" in step 1 — on many databases adding a NOT NULL column with a default to a large table triggers a full table rewrite (PostgreSQL only optimized non-volatile defaults in version 11), turning an apparently harmless migration into a long lock.

The chapter also mentions a heavier decoupling device: wrapping the database in a "public interface" of views or stored procedures, so the application programs against the interface and the interface layer absorbs changes to the underlying tables. It genuinely reduces coupling, at the cost of pushing logic into the database where testing and version management are harder; the book stays measured about it.

4. Shared databases and orchestrated change

Every technique above assumes the database is yours alone. The hardest real-world case is several applications sharing one database — what the book calls changes that need orchestration: alter one column and you must alter three teams' code, and land three releases simultaneously. That contradicts the foundations of continuous delivery (small batches, independent releases), so the chapter's advice is to minimize the need for orchestration: let each application own its data and let cross-application access go through a well-defined interface — a service or a database view — rather than letting others read your tables directly. Where it truly cannot be split, you must open the compatibility window wide enough — wide enough for the slowest team to finish migrating. This is exactly Martin Fowler's integration database anti-pattern, and the direct ancestor of the microservices "a database per service" rule.

5. Test data: three kinds, and why not to clone production

The second half of the chapter turns to a far more everyday problem that is equally chronically mishandled: where test data comes from. First separate three kinds — reference data (currencies, countries, status codes, versioned with the schema); test-specific data (the few records a given test creates to prove its point); and application consistency data (the surrounding records that must exist to satisfy foreign keys and business constraints). Conflate them and you get the giant shared test database that nobody dares delete from.

The single most important anti-pattern: do not copy the whole production database into test environments. Four reasons, each fatal — it is too big (acceptance tests go from minutes to tens of minutes, wrecking Ch7's ten-minute commit stage and Ch8's feedback rhythm); it holds real personal data (a compliance risk); nobody owns it (one person edits a record and somebody else's test turns red the next day); and it rots (production carries historical dirty records that tests assume are valid). The right answer is that each test creates the minimum dataset it needs, as far as possible through the application's own API — so when the schema evolves the test data evolves with it, instead of decaying into a pile of hardcoded INSERT statements.

For decoupling tests from data the chapter offers three approaches of clearly unequal merit: test isolation (each test uses its own key prefix or account and cannot see the others — the recommendation); adaptive tests (query the current state first, then assert against it or build the prerequisites); and test sequencing (test B depends on data left behind by test A) — which the book explicitly argues against: it cannot be parallelized, cannot be rerun individually, and one failure cascades into many.

Fig 4 · The data gradient along the pipeline: the commit stage trades realism for speed with fakes, acceptance trades size for isolation with purpose-built data, capacity needs generated volume, and production runs the same migration scripts as everywhere else.

Trade-offs and how to choose

Table 1 · Getting back when the database goes wrong: what each of the three routes really costs

	Restore a backup	Run the reverse (down) script	Compatibility window + non-destructive migration
Data loss	Yes — every write since the backup point	Loses whatever was in the dropped column, unless copied first	None — the old structure is still there
Time	Often hours at terabyte scale	Minutes, depending on the script	Seconds — only the app rolls back
Downtime	Required	Usually required	None
Prerequisite	A usable backup and a rehearsed restore	A correct down for every migration; nothing destructive	One change split across several releases; code that serves both shapes
Main cost	You pay in users' money and trust	Reverse scripts are almost never tested and may not run when it matters	A longer, fussier process; old columns linger for weeks
When to use	The safety net you always keep — not the primary plan	Low-risk pure structural change; non-production environments	The default, especially for high-traffic systems that cannot stop

Table 2 · Danger rating of common schema changes, with the safe way to do each

Change	Where the danger is	The safe way
Add a nullable column	Essentially harmless, a metadata operation	Just do it. This is what step 1 of expand-contract looks like
Add NOT NULL with a default	May rewrite the whole table under lock — hours at a hundred million rows (classic before PostgreSQL 11)	Add it nullable → backfill in batches → add the NOT NULL constraint
Create an index	Scans the whole table; the default form blocks writes	Build it online or concurrently (e.g. `CREATE INDEX CONCURRENTLY`) — slower, but out of the way
Change a column type	Rewrites data; the old app version may fail to parse immediately	Treat it as adding a new column and run the full expand-contract
Rename a column or table	Destructive by nature — the old app version cannot find it any more	Add the new name → dual-write → switch reads → drop the old name several cycles later
Drop a column or table	Irreversible, and hidden references often survive elsewhere	Stop writing to it, observe for several cycles, confirm zero access, keep a copy before dropping
Bulk data move	One `UPDATE` across a whole table blows up the transaction log and holds a long lock	Backfill in batches (say 1,000–10,000 rows each, pausable and resumable) rather than one statement

Table 3 · Keeping tests from trampling each other: three ways to decouple

	Test isolation	Adaptive tests	Test sequencing
How	Each test uses its own keys or accounts and touches only its own records	Query the current state, then assert against it or build what is missing	Later tests depend on data left behind by earlier ones
Parallelizable	Yes	Yes	No
Rerun one test alone	Yes	Yes	No — the suite must run in order
Diagnosing failures	Clean — it can only be this test's own problem	Reasonably clean	One failure cascades; you hunt for who polluted the data
Cost	You must design key generation and cleanup	Assertions get convoluted and easily too loose	Looks cheapest, is the most expensive
The book's view	Preferred	Usable, especially against legacy systems	Argued against

Table 4 · Who owns the database: dedicated vs shared, and the price of orchestration

	Each application owns its database	Many applications share one (integration database)
Parties to coordinate for one column	One	Every team reading that table, all releasing together
Release rhythm	Independent per team; daily is possible	Held back by the slowest party; you regress to big batches
How long the compatibility window stays open	Your call — usually days to weeks	Until the slowest team has migrated; possibly months
Cost	Cross-application queries go through interfaces; some data is duplicated	The foundation of continuous delivery is removed — neither small batches nor independent releases survive
The book's advice	The default: own your data, expose a defined interface	Split it if you can; if you cannot, put the orchestration cost on the table explicitly

Beyond the four tables, the chapter's real test is a single question you can ask on the spot: if you decided ten minutes after your last production release to take it back, what would the database side have to do? If the answer is "nothing, just redeploy the old application", you are on this chapter's path. If it is "wake the DBA to restore a backup", then however advanced your deployment tooling, your releases are still irreversible.

Why it matters / in real systems

This is the chapter the following fifteen years validated most thoroughly. Flyway and Liquibase made "numbered migration scripts plus a version table in the database" an industry standard, and Rails, Django and Entity Framework built it into the framework. gh-ost, pt-online-schema-change and pg_repack exist to solve precisely the obstacle the chapter named — a long lock while altering a large table — and expand-contract is now the first line on any zero-downtime release checklist. Conversely, the two traps the chapter warned about are still with us: the shared database is still the hardest cut in any microservices split, and "copy production for test data" is still most teams' opening move.

EVIDENCE FROM INDUSTRY · NOT INFERENCE — THESE ARE THE COMPANIES' OWN PUBLISHED WORDS

Stripe published the four-step pattern they use for large online migrations: dual-write to the old and new tables, backfill historical data so the two hold identical information, move every read path to the new table, and only then stop writing to the old one and remove the outdated data — all without downtime and invisibly to users. It is almost a line-by-line realization of this chapter's compatibility window and non-destructive migration, which is what we now call expand-contract.Stripe, "Online migrations at scale" ↗
GitHub explained its motivation when open-sourcing gh-ost in 2016: continuous production change means continuous schema change to MySQL tables, while existing tools relied on triggers, imposed the migration's write load on the master, and could not truly be paused. gh-ost instead reads the binary log and applies changes asynchronously to a ghost table, so a migration can be paused, reconfigured on the fly and audited, and can even be trialled on a replica so the result can be compared. The chapter's demand that structural change be rehearsable and non-disruptive, turned into a tool.GitHub Blog, "gh-ost: GitHub's online schema migration tool for MySQL", 2016 ↗
Google supplied the formal proof of this chapter's intuition in its paper on F1, the distributed database behind AdWords: where servers cannot switch schema simultaneously, many common structural changes cause data corruption outright; the remedy is replacing one change with a sequence of intermediate schema states, which is guaranteed safe so long as no server is more than one version behind. "Split a structural change into mutually compatible steps" is not merely field experience — it is a provable necessity.Rae et al., "Online, Asynchronous Schema Change in F1", VLDB 2013 ↗
The other side · GitHub, 2018-10-21: maintenance on network equipment severed connectivity between the US East and West Coast sites for just 43 seconds, during which the East Coast primary accepted writes that were never replicated. Because that data could not simply be discarded or rolled back, GitHub chose to fail forward and reconcile, and service stayed degraded for 24 hours and 11 minutes, much of it read-only. Code rolls back in seconds; a fork in the data is repaid in hours of reconciliation — the real price of this chapter's asymmetry.GitHub Blog, "October 21 post-incident analysis", 2018 ↗
Braintree (PayPal's payments platform) published its classification of PostgreSQL structural changes, spelling out which DDL statements take an ACCESS EXCLUSIVE lock and therefore amount to an outage under load, and hardened the judgement into the open-source pg_ha_migrations gem so that unsafe migrations are caught at code review. The premise: their main payment-processing services accept no scheduled downtime at all.Coleman, "PostgreSQL at Scale: Database Schema Changes Without Downtime" ↗ braintree/pg_ha_migrations ↗

In interviews and architecture reviews the questions from this chapter are predictable, and the answers that score are never tool names: are your database changes in version control, or typed out on release night? If your last release had to be rolled back, what happens on the database side? How would you add a NOT NULL field to a table of a hundred million rows? Is your test environment's data a copy of production? That last one is especially revealing — teams who answer yes almost invariably also have slow, flaky acceptance tests.

Pitfalls and disputes

Mistaking "we wrote a down script" for "we can roll back". Reverse scripts are almost never executed in the pipeline and therefore never tested; the moment things go wrong is the first time you run one. More fundamentally, they cannot restore information — the column comes back empty. Reversibility comes from non-destructive migrations and compatibility windows, not from down scripts.
Editing migration scripts like ordinary code. A published migration is immutable history: change number 028 directly and environments that already ran it diverge permanently from those that did not. Corrections can only be appended as number 032.
Testing only whether a migration runs, never how long it takes. An ALTER that finishes instantly against the 200-row table on a developer machine may hold a lock for two hours against 200 million rows in production. Migration duration is itself a property that must be verified against production-like volumes — one reason capacity environments need realistic amounts of data.
Forgetting that rolling back the application demands forward compatibility. Everyone remembers the new version must read old data, and forgets the reverse: the old application version must keep running against the new schema, or a rollback becomes a site-wide outage. Those "redundant" intermediate steps in expand-contract are what buy exactly this.
Assuming anonymized production data settles the matter. Anonymization solves privacy and none of the other three problems: still too big, still owned by nobody, still carrying historical dirt. It is a compliance floor, not a test-data strategy.
Limits of its time. The book predates managed cloud databases, mature change-data-capture tooling and online-change tools like gh-ost, so it says little about altering large tables without locking them; and its assumed relational world does not cover schemaless stores — though those merely move the schema out of the database and into application code, leaving the compatibility-window problem intact and rather better hidden.
The dispute: should databases have down scripts at all? One camp says yes, something is better than nothing. The other — more common today, and the direction Flyway's defaults lean — argues for forward-only: when something breaks, ship another forward migration, because the forward path is tested and the backward one is not. The practical test is simple: if your migrations were non-destructive to begin with, fixing forward almost always works; and if they were not, a down script will not save you either.

Key points

1. In one line: code is stateless and can be swapped back wholesale; data cannot. The reversibility of a release ultimately hinges on the database side.

2. The database must go under version control in two halves: initialization scripts (create the schema, load reference data) and numbered incremental migrations; a version table inside the database records where it is, and deployment applies only the gap.

3. The rule to remember: migrations must be non-destructive. Keep a copy before dropping a column, and note that a rename is a drop plus an add — destructive as far as the database is concerned.

4. What the three rollback routes really cost: restoring a backup loses data and takes hours; a down script recovers structure but not content, and has almost never been tested; a compatibility window loses nothing, takes seconds and needs no downtime — choose the third by default.

5. The core technique is expand-contract: add column → dual-write → backfill → switch reads → stop old writes → drop old column, shipped as six separate releases. Inside the compatibility window both application versions run, so every step can stop and step back, and the one irreversible deletion waits until the risk is gone. It is also the only correct answer when a blue-green deployment shares a database.

6. Learn the dangerous-change list: adding NOT NULL with a default, creating an index, changing a type, renaming, bulk UPDATE over a whole table — any of them can mean an hours-long lock at a hundred million rows. The safe form is always "make it additive, backfill in batches, contract afterwards".

7. Test data comes in three kinds (reference, test-specific, consistency), and you should not clone production — too big, private, unowned, and rotting. Let each test build its minimum dataset through the application's API, and decouple with test isolation rather than test sequencing. Along the pipeline there is a gradient: fakes protect the commit stage's ten-minute budget, purpose-built data buys isolation in acceptance, generated volume buys realism in capacity testing.

8. A shared database is the invisible ceiling on continuous delivery: it converts independent releases into multi-party orchestrated batches. The default should be each application owning its data, with defined interfaces between them.