BOOK DEEP READ · CONTINUOUS DELIVERY · CH 10
Continuous Delivery · Ch 10 · Jez Humble & David Farley · 2010
The apps on your phone and the sites you open every day quietly swap themselves for a new version every week, sometimes every day. This chapter is about how you swap them — how a new version reaches thousands of machines without disturbing a single person using it, and the more important half: when it turns out to be wrong, how you get back within two minutes.
Shipping used to look like closing a restaurant for a refit: hang the "closed" sign at ten at night, a crew works until three in the morning, and you open the doors next day and pray. This chapter teaches two other moves.
The first is blue-green: don't close. Fit out an identical restaurant next door, stock it, do a trial run, then change the sign at the corner — customers walk into the new place on their own. If something is off, change the sign back. Ten seconds, and the old place is still standing.
The second is canary (named for the bird miners carried underground: the bird goes quiet first, and everyone leaves). Serve the new dish to two tables out of a hundred and watch them. Fine? Widen it. Not fine? Pull it — at worst, two tables ever tasted it.
The hard part was never pushing the new version out. It was that nobody had rehearsed it. A release was a forty-page manual typed in by one person at 2am; the test environments followed one set of steps and production followed another — so production was the one performance in the whole run that never had a dress rehearsal.
The back door was worse. The last page of the manual usually said "to revert, do the above in reverse", and nobody had ever actually run that. Its first execution was on the worst night of the year. A lot of famous outages have that shape: the new version wasn't the disaster — being unable to get back was.
One: the moves you make going live must be the moves you rehearse daily. Test, staging and production all get the same automated script installing the same package; only a few lines of configuration differ. You do it right on the hundred-and-first try because you did it a hundred times before.
Two: being able to go back matters more than being able to go forward. Rollback shouldn't be a contingency sleeping in a document; it should be an ordinary function you use constantly. A path you walk every day is the one you can trust in the rain.
Three: don't swap everything at once. Swap a sliver, watch the dashboards, then widen — turning the release from an on/off switch into a dial you can turn slowly.
Code goes back in seconds. Data doesn't. Whatever the new version already wrote in its new format, the old version may simply be unable to read — so the real difficulty was never putting the program back, it's making one pile of data legible to both versions at once.
Turn releasing from "a few times a year, everyone up all night" into "any time, one button to undo": the same script deploys every environment, new versions go out to a sliver first, and the road back gets walked every single day.
Want the actual cutover mechanics, how much traffic a canary really takes, and why rollback always jams on the data? → Switch to the deep read
You think the hard part of a release is pushing the new version out; it is actually getting back when the push was wrong. This chapter converts releasing from a one-off, high-risk event into something repeatable, rehearsed and reversible: one automated process deploys every environment including production, one artifact is promoted upward, rollback is a first-class feature — and then blue-green and canary releasing break the single all-or-nothing cutover into observable, revocable steps.
This is the closing chapter of Part II, "The Deployment Pipeline". Ch5, Ch7 and Ch8 were all about falsifying a release candidate as cheaply as possible; this chapter is the last mile: how the build that went green actually gets into production, and how you get back out. It hands the two biggest unsolved problems to Part III — managing environments (Ch11) and evolving data alongside code (Ch12). Its real-world descendants are everything in use today: blue-green, canary, rolling upgrades, Spinnaker / Argo Rollouts, GitOps.
Picture a typical enterprise release around 2010: a forty-page runbook, a four-to-six-hour downtime window starting 22:00 on a Friday, a room full of people on an all-night conference bridge, one release a quarter — carrying thousands of changes in a single shipment. Three structural defects, and they reinforce each other:
Left alone this becomes a self-reinforcing spiral: rarer releases → more risk per release → heavier process → rarer still. Every technique in the chapter exists to break that spiral and make releasing frequent, boring and reversible. The price tag is public: on 2 July 2019 a single Cloudflare WAF rule skipped the usual gradual rollout and went global within seconds; 27 minutes of worldwide errors, with traffic down 82% at the trough (see §6).
The chapter's most counterintuitive opening claim: the release strategy should be started when the project starts, owned jointly by development and operations, and updated every iteration. It answers who deploys, who approves a promotion; the steps for deployment and rollback in equal detail; how configuration is versioned; how data is migrated and what happens when migration fails; which metrics you watch afterwards. The accompanying hard requirement: get "deploy to a production-like environment" working in the first iteration — drag the most postponable thing to the front, then repeat it every iteration after.
This is Ch5's "build your binaries only once" cashed in at the last mile: the binary is built once in the commit stage and its bytes never change again; it only gets promoted between environments. The one thing that varies is configuration — which must itself be versioned and promoted along with it. Nobody edits a line by hand on a production box.
Why be so strict? Because only if production and staging share one deployment script does any earlier testing count. Two side benefits: the script is exercised dozens of times a day, so it becomes well-tested code in its own right; and which version, approved by whom, landed where is recorded as a matter of course.
The chapter's hardest claim: a release plan with a rollback procedure nobody has actually rehearsed is not a plan. Two roads are offered. Road one: redeploy the last known good version using the same automation — assuming deployment is fully automated, the old artifact is still in the repository, and the data is backwards compatible. A small companion rule earns its keep: don't delete the old files, move them, so the previous version's scene is preserved. Road two: switch back with blue-green (next section) — flip the route, done in seconds.
And the wall is data: once the new version has written anything in its new format, the old version may not be able to read it, and rollback is physically blocked. So the real question is not "how do I revert the code" but how do I make one pile of data legible to both versions: when changing a schema, do additive work first (add a nullable column), let both versions coexist, and only subtract after the new version has been stable for days and you're sure you won't go back. This expand-then-contract shape is developed in Ch12 and became the industry-standard pattern.
Hence a piece of discipline: emergency fixes go through the same pipeline as everything else. One hand-made change on a production box forks production from version control permanently, and from then on nobody knows what is actually running. Under pressure, rolling back is usually faster and safer than a hotfix — judgement at 3am is the least reliable tool you own.
Blue-green: keep two identical production environments. Right now "blue" is live with 100% of traffic; the new version is deployed to the idle "green", smoke-tested and warmed up until you're sure it's alive — all invisible to users. Then you change the route once and traffic moves to green in an instant. Rollback is changing it back, equally instant.
Blue-green compresses downtime to the instant of the cutover and gives you the fastest rollback in the book. Be honest about the costs: double the infrastructure (in 2010 that meant buying twice the physical machines, the main reason blue-green stayed rare; in the cloud it is far cheaper); the database is usually shared — the book's answer is to put blue into read-only mode at the cutover and copy across, or to keep the schema compatible with both versions, which is what people actually do today; plus two practical traps: cutting over via DNS lets TTL stretch "seconds" into tens of minutes, so switch at the load balancer, and existing connections still need graceful draining.
Canary releasing: instead of going to everyone at once, send the new version to a small subset of servers or users — say 2% of traffic — then watch error rates, latency and the key business metrics for a while. Healthy, and you ramp (2% → 5% → 25% → 100%); unhealthy, and you pull the canary. Rollback here barely needs doing: route that 2% back to the baseline, and at worst 2% of users ever saw the bad build.
It unlocks two more things: capacity testing on one real machine (feed the canary a known fraction of live traffic and extrapolate the fleet size you need), and A/B testing. The costs are equally concrete: two versions are live in production simultaneously, possibly for hours or days, so schema, message formats and external protocols must all tolerate both — the same wall rollback hit. And you need to slice metrics by version, or 2% worth of anomaly drowns in 98% worth of healthy traffic on the same dashboard.
Push this all the way and you get continuous deployment: every build that passes the pipeline goes to production automatically, with nobody pressing a button (the book's own example is IMVU — see §6). Keep the distinction straight, though: the book argues for continuous delivery, meaning the software is always releasable and whether to release is a business call. Continuous deployment is one special case of that, not its inevitable destination.
The chapter's most portable idea is that dividing line: deployment is a technical act, release is a business decision, and they can be separated. Blue-green and canary already pry the gap open — the code is sitting in production, it just hasn't been given traffic yet. Prying it fully open (feature toggles that ship code with the feature invisible) is Ch13's material; this chapter goes as far as controlling the traffic percentage.
Finally, a set of unglamorous but very usable details: give a deployment a warm-up period (caches and connection pools need time, so don't slam it with full traffic the second it's up); make scripts fail fast rather than discover a broken environment halfway through; log every deployment activity; make sure the people who actually perform deployments help write the deployment process; and keep server applications headless, so everything can be scripted.
Table 1 · Four ways to go live: downtime, cost, rollback speed and the hard prerequisites of each
| In-place replace | Rolling upgrade (post-book, today's cloud default) | Blue-green | Canary | |
|---|---|---|---|---|
| Downtime | The whole deployment: minutes to hours | None (given multiple instances) | The instant of cutover | None |
| Extra resources | None | Roughly one spare instance | A second production environment | A handful of instances (e.g. 2%) |
| Rollback speed | Another full deployment | Roll the other way; same order as deploying | Flip the route, seconds | Pull the canary, seconds |
| Blast radius | 100% of users | Whatever has been rolled so far | 100% after cutover | = the ramp percentage, tunable |
| Prerequisite | An acceptable downtime window | Old and new can coexist | Database tolerates both; don't cut over via DNS (TTL destroys "seconds") | Coexistence + metrics sliced by version |
| Fits | Internal systems with a natural quiet window | The default for stateless services | Clean cutovers, second-scale rollback | Risky changes, large user base, strong observability |
Table 2 · Three ways out after a bad release (the chapter's default answer is the first)
| Approach | Time scale | Prerequisite | When to pick it |
|---|---|---|---|
| Redeploy the last good version | One full deployment: minutes to tens of minutes | Automated deployment + old artifact still in the repo + backwards-compatible data | The default: stop the bleeding while the cause is still unknown |
| Switch back (blue-green / canary) | Seconds | Old environment not torn down; data tolerates both versions | You already paid for the second environment — collect the benefit |
| Fix forward | Unpredictable: write the fix, then the whole pipeline | You are sure of the root cause and the pipeline is fast | Only when going back is impossible; never as an excuse to bypass the pipeline and edit production |
One overarching criterion sits above all three: ask how often you rehearse rollback before you choose a release mechanism. A plain redeploy that genuinely runs every day beats a blue-green cutover that lives in a document and has never been executed — reliability comes from repetition, not from sophistication.
Nearly every cloud release tool in use today has its prototype in this chapter: Kubernetes rolling updates and readiness probes (the "smoke test plus warm-up" pair), canary and red/black releases in Argo Rollouts and Spinnaker, blue-green with automatic rollback in AWS CodeDeploy. And automated canary analysis in SRE practice — comparing canary against baseline statistically and deciding the ramp automatically — is the industrialised version of this chapter's observation window.
This is a high-frequency interview and design-review topic. Asked "how do you release?", the answer that scores is not a list of tools but five facts: do production and staging run the same deployment script? When was your last rollback and how long did it take? What canary percentage, how long a window, and which metrics? How do schema changes stay reversible? Do emergency fixes go through the pipeline?
1 · This chapter converts releasing from a one-off high-risk event into something repeatable, rehearsed and reversible; the difficulty is not going forward, it is coming back.
2 · A release strategy is written on day one, owned jointly by dev and ops; get "deploy to a production-like environment" working in the first iteration.
3 · The structural rule: one artifact, one script, every environment, production included — all differences live in versioned configuration.
4 · Rollback is first-class: the default road is redeploying the last good version with the same automation; move old files rather than deleting them; emergency fixes still go through the pipeline and never by hand on production.
5 · The real wall is data: code reverts in seconds, data does not. Schema changes must add first and subtract much later so both versions read one dataset — that is what makes rollback real.
6 · Blue-green: two production environments take turns, so release and rollback both become one routing change; the cost is double infrastructure, an unswitchable database, and never cutting over via DNS.
7 · Canary: give 2% of traffic to the new version, pass an observation window, then ramp (2% → 5% → 25% → 100%). Blast radius equals the ramp percentage, and you get capacity testing and A/B for free.
8 · The dividing line: deployment is technical, release is a business decision. The end of the road is continuous deployment (IMVU: a new revision every 9 minutes), but continuous delivery asks only that you can release at any time. Reliability comes from repetition, not sophistication.