BOOKS DEEP-READ · CONTINUOUS DELIVERY · CH 6
Continuous Delivery · Ch 6 · Jez Humble & David Farley · 2010
The app on your phone updates every week or so. Between the code an engineer finished and the working version on your device sits a chain of actions: compile, package, copy to servers, adjust configuration, restart. Chapter 6 of Continuous Delivery is about how that chain should be written as scripts — not a few commands typed from memory, but something written the way you write real product code.
Think of a release as cooking. Manual deployment is a chef improvising: an extra spoon of salt today, a burner left on tomorrow. Scripting it means writing the recipe down and letting a machine follow it. But this chapter goes further: the recipe should not live on a sticky note on the fridge — scribbled ad hoc, one per kitchen, messier with every edit. It should be maintained like a published work: versioned, reviewed, and traceable back to the edition that went wrong.
In August 2012 the US brokerage Knight Capital rolled out new code by hand to eight production servers and missed one. Dormant old code on that machine was woken up by a repurposed switch, fired off bad orders, and in roughly 45 minutes burned more than $460 million — effectively ending the firm. The root cause is not exotic: deployment was done by a person, machine by machine, with nothing guaranteeing the eight were identical.
One, the same script. Development, test, production — all run the same deployment script, and the differences between environments live only in an external configuration file. Once each environment gets its own script, they drift apart, and the thing you rehearsed twenty times in test is not the code that runs in production.
Two, run it as often as you like. A good script does not say "do these 12 steps"; it says "make sure it ends up looking like this" — skip what is already installed, fix what is installed wrong. So when it dies halfway, you just run it again instead of cleaning up by hand first.
Three, layers. Split "install the OS and middleware", "place the application", "apply configuration" and "migrate data" into four separate scripts, because they change at wildly different rates: configuration might change three times a day, the OS once a year. Fused together, changing one config line means reinstalling middleware.
Don't try to script everything at once. The chapter's advice is practical: start with the step that hurts most — whatever part of every release you dread and keep getting wrong — script that, then grow outward from it.
And the cost is real: scripting is not free. These scripts have their own bugs and need their own maintenance; you are now carrying another codebase. It still pays, because errors in manual deployment never show up in the test environment — only in production.
Writing deployment down as a script is not enough. Every environment must share the same script (differences live in configuration), running it twice must change nothing, and it must be treated like production code: in version control, reviewed, refactorable.
Want the two schools of build tools, how idempotence actually works, and the comparison tables? → Switch to the deep read
Chapter 6 handles the pipeline's manual labour: how to write build and deployment as scripts. Its core claim is a single line — build and deployment scripts are production code, not a pile of one-off commands: they belong in version control, they get layered, refactored and tested. Three hard rules follow: every environment shares one deployment script (differences only in external configuration), deployment must be idempotent, and deployment is layered by rate of change.
compile or test.This chapter belongs to Part II, "The Deployment Pipeline", and follows directly from Ch5: Ch5 draws the shape of the gates; this chapter answers what actually executes inside each box. It leans on Ch2 "Configuration Management" (the "same script plus external configuration" rule needs Ch2 to be real) and leads into Ch7 "The Commit Stage" and Ch11 "Managing Infrastructure". The real-world problem it maps to: why nobody on the team dares touch build.xml, the Makefile or deploy.sh after three years of evolution.
Anyone can draw the pipeline diagram. What actually drags teams down is the pile of scripts that executes it. Their usual fate: a three-thousand-line build.xml only its author understands; deploy-dev.sh / deploy-uat.sh / deploy-prod.sh, each separately patched until they no longer resemble one another; and a wiki line saying "before running this, stop service A by hand."
What happens if you don't fix it? The chronic cost: nobody dares restructure the script, so every new requirement arrives as another if; build time climbs from 3 minutes to 25, blowing past the ten-minute budget Ch7 sets for the commit stage — developers stop waiting for the result, and continuous integration is dead in all but name. The acute cost: the Knight Capital 45 minutes from the plain version. The SEC's 2013 enforcement document faulted the firm's deployment process for lacking automation and verification of consistency. "One of eight machines is different" is exactly what scripting plus idempotence exists to eliminate.
Every build tool does the same thing: break the build into targets with dependencies, topologically sort them, execute in order. The disagreement is over what a node is.
Product-oriented, exemplified by Make: a node is a file to be produced. app depends on a.o, a.o depends on a.c; Make compares file timestamps and rebuilds only when the source is newer. Incremental builds come for free — but the test is fragile. Timestamps are one-second resolution on many filesystems, so edits within the same second are silently missed, and a fresh checkout stamps every file with "now". Teams end up in the habit of "when in doubt, clean build", handing back everything incrementality bought them.
Task-oriented, exemplified by Ant / NAnt: a node is a thing to do. You can express whatever you want without forcing it to map onto a file; the cost is that the tool has no idea whether work can be skipped, so the skip logic is yours to write inside each target and the default behaviour is to run everything.
The chapter names a third route: convention over configuration, exemplified by Maven. It fixes the directory layout and lifecycle (src/main/java; compile → test → package → install) so you configure rather than script. The authors' verdict is honestly two-sided: new projects work out of the box and teams share one shape; the moment you need to deviate from the convention, cost rises steeply — usually into writing a plugin, which is harder to read than the script it replaced.
And a fourth: just use a real programming language — Rake (Ruby), Psake (PowerShell), later Gradle. Once build logic needs conditionals, loops and abstraction, expressing it in XML is self-harm. The cost is that an unconstrained language lets the build script grow into a second application — which is why this route requires the discipline in the next section.
This is the soul of the chapter and the part most often skipped. Build and deployment scripts go into version control, in the same repository and branch as the code they serve, through the same review. The reasoning is direct: a given version of the code can only be built and deployed correctly by the matching version of the scripts. Keep them elsewhere — in a CI server's "build configuration" screen, in someone's home directory — and you have permanently lost the ability to go back to the version from three months ago and reproduce it exactly.
The same rule yields concrete discipline: layer and de-duplicate, factoring shared logic into reusable modules instead of copy-paste; always use relative paths, because a script with /opt/app hard-coded runs only on its author's machine; write build output outside the source tree, or "clean" depends on .gitignore and luck; bind artifacts to the revision, putting the commit id in the filename or metadata so a production incident can be traced back to a line of code.
And one that is commonly done backwards: a failing test must fail the build, but must not abort it immediately. Record "something failed" as a flag, let this round of tests finish, and fail at the end — so one build shows you every failure instead of one, then the next, then another ten-minute wait.
This is the most actionable rule in the chapter: developer machines, CI, staging and production all run the same deployment script, and differences between environments exist only as external configuration (per Ch2: configuration is version-controlled too, just managed separately from code).
Why so absolute? Because the deployment script is itself a thing under test. You rehearsed the deployment twenty times in staging; if production runs a different script, those twenty runs never touched the code that actually matters. "Code that executes for the first time on release day" is the risk — and a script per environment is a machine for manufacturing exactly that code.
The cost is real: the script must be parameterised, and parameters breed complexity of their own, so you need a configuration management approach (a config repository, template rendering, secret storage) to carry it. The principle is let environments differ in configuration, never in behaviour — the moment one environment needs "one extra step", ask whether that step could be done in all of them.
Idempotence means: whatever state the target machine is in now, after the script runs it is in the correct state; run it again and nothing changes.
The difference from "record the manual steps as shell" is fundamental. An imperative script says "unpack, create directory, write config, start service"; when step three dies, the machine sits in an intermediate state — re-running fails because the directory already exists, and you go in by hand. A convergent script says "this directory should exist, this config file should have these contents, this service should be running", reading current state first and changing only what disagrees. The result is that "just run it again" becomes a permanently safe action — and that property is the precondition for automation anyone dares leave unattended.
This is the shared design of the Puppet / Chef generation around 2010 (and Ansible shortly after). The chapter pairs it with a related discipline: layer deployment by rate of change — OS and middleware, application binary, configuration, data and schema each get their own script and their own cadence. Fused together, changing one config line means reinstalling the middleware.
The chapter offers an engineer's-taste suggestion that pays off: if RPM / deb / MSI will do, don't invent your own packaging format. They come with dependency declarations, standard install / remove / upgrade actions, and a queryable database of what versions are on a machine — all of which you would otherwise rebuild in scripts, probably worse. The cost is that cross-platform means maintaining several packages, and the package manager's abstraction gets in the way when you need fine control.
Finally, the integrated smoke test: the last step of a deployment script should be a short "is it actually alive" check — process up, port reachable, database connectable. It takes seconds, and it separates two things routinely conflated: "the script exited 0" is not "the service works".
Table 1 · Four families of build tool: what each buys, what each costs
| Product-oriented: Make | Task-oriented: Ant/NAnt | Convention-first: Maven | General language: Rake/Gradle | |
|---|---|---|---|---|
| What a node is | A file to produce | A thing to do | A fixed lifecycle phase | Arbitrary code |
| Incremental test | File timestamps | Written by you, per target | Decided by each plugin | Yours (modern ones hash inputs) |
| Complex logic | Very poor | Poor (branching inside XML) | Requires a plugin | Good |
| Main cost | Timestamps are unreliable, forcing clean builds | Runs everything by default; slow | Cost spikes once you deviate | Grows into a second application |
| When to pick it | File-to-file compile chains such as C/C++ | Legacy projects, simple flows | New Java projects with standard shape | Build logic with genuine complexity |
Table 2 · Deployment scripts: one shared script vs one per environment
| One script + external config (the chapter's position) | One script per environment | |
|---|---|---|
| Has the production code path been tested? | Yes — dozens of runs across every environment | No. It has only ever run in production |
| Adding a new environment | Add one configuration | Copy a script, and gain another source of drift |
| Changing shared logic | Change it in one place | Change N places; miss one and it's an incident |
| When environments really do differ | Must be explicit in config: diffable, reviewable | Buried in the script; nobody can state the difference |
| Cost | Needs a config management approach; parameters multiply | Cheapest short term, most expensive long term |
Table 3 · Three deployment paradigms: imperative, idempotent convergence, immutable replacement
| Imperative step script | Idempotent convergence (Puppet/Chef/Ansible) | Immutable replacement (images / containers, post-book) | |
|---|---|---|---|
| Failure midway | Stuck in an intermediate state; needs a human | Just run it again | Throw it away, take a fresh one |
| Configuration drift | Accumulates | Converged away each run | Cannot occur — machines are replaced, not edited |
| Cost to adopt | Lowest | Medium: learn the declarative model | Higher: needs image builds and orchestration |
| Main cost | Not re-runnable, so automation can't be unattended | The convergence logic has its own bugs; edge cases are hard | Stateful parts (data, caches) still need another answer |
| When to pick it | One-off tasks, scaffolding | Long-lived existing hosts | Stateless services on cloud / containers |
The chapter's three rules are industry default today, only renamed: "build scripts in version control" became CI configuration as code (Jenkinsfile, workflow YAML, sitting in the application repository); "the same script to every environment" became one container image plus per-environment parameters; "idempotence" became the founding property of declarative infrastructure — Terraform's plan / apply and Kubernetes' reconciliation loop are both the loop on the right of Fig 3.
The interview question about why one artifact should travel all the way through rather than be rebuilt per environment has a second half — the script travels all the way through too — and that half is this chapter.
{Fast, Correct} - Choose two, claiming "Bazel rebuilds only what is necessary. With advanced local and distributed caching, optimized dependency analysis and parallel execution, you get fast and incremental builds." — the chapter's "dependency graph plus incrementality" taken to its limit, with Make's timestamps replaced by precise modelling of inputs. bazelbuild/bazel README ↗1. In one line: build and deployment scripts are production code — version-controlled, layered, refactorable, testable. Every other rule in the chapter grows from this one.
2. A build tool is a dependency-graph traversal; the split is whether nodes are files (Make: timestamps, a fragile test) or tasks (Ant: runs everything by default).
3. Convention-first (Maven) gives standard projects a working build for free, at the price of a steep cost once you deviate; a general-purpose DSL is expressive, at the price of growing into a second application.
4. The hardest rule: every environment shares one deployment script, differences only in external configuration — otherwise the code running in production has never been tested.
5. Idempotence: the script describes a target state, not steps; N runs equal one run. This is the precondition for unattended automation.
6. Layer deployment by rate of change: OS and middleware, application, configuration, data — one script and one cadence each.
7. Engineering discipline: relative paths, output outside the source tree, artifacts bound to a revision, failing tests fail the build but only after the round finishes, native packages over homemade ones.
8. The last step of deployment is a smoke test — "exited 0" and "the service works" are different claims.
9. Data is the exception to idempotence: schema migration is inherently non-repeatable and needs versioned migration scripts (Ch12).
10. In today's terms: CI configuration as code, one image across environments, and the declarative reconciliation of Terraform and Kubernetes are all direct descendants of this chapter.