BOOK DEEP READ · CONTINUOUS DELIVERY · CH 11
Continuous Delivery · Ch 11 · Jez Humble & David Farley · 2010
Every app you open runs on thousands of machines. Earlier chapters covered how a new version gets pushed onto those machines. This chapter digs one layer down, to something more basic: where those machines came from, who is allowed to change them, and what happens when one breaks.
Think of the machines as the kitchens of a restaurant chain. The ideal is that headquarters holds one blueprint and every branch is built strictly from it — where the stove goes, which extractor fan, which shelf holds the fire equipment. Opening a new branch means building from the blueprint. If a kitchen burns out, you rebuild from the blueprint, which is faster than repairing it.
The old way looks different: each branch kitchen is put together by a local contractor from experience. They all look roughly alike. Year one is fine. In year three headquarters wants to switch everyone to a new stove, and only then does it emerge that no two kitchens are actually the same — one has the sockets in the wrong place, another had a wall moved years ago for convenience, and nobody wrote the wall down.
A hand-built server is raised rather than built: over three years someone installed a package in a hurry, someone edited one line of config to stop an outage, someone opened a port and forgot to close it. Each change was tiny and none was recorded. Three years later the machine is a living fossil nobody dares touch: it still runs, but nobody can say why it runs, and nobody can produce a second one exactly like it.
Which is where the famous shrug comes from — "well, it works on my machine." Usually nobody is lying. It is simply that the machine you tested on and the machine serving users were never the same thing.
First, write down what a machine should look like: which things are installed, what every setting is, which ports are open. That list lives in the same repository as the code, so every change carries a name and a reason.
Second, only the robot touches the machines; people may not. This is the hardest rule in the chapter: to change a setting in production you change the list and let automation push it down. You do not log in and type.
Third, the robot keeps coming back to check. If something on the list has gone missing it puts it back; if somebody slipped something extra in, it removes it. That way machines cannot drift over time.
A fiercer version grew out of this later: rather than repair a machine that has gone crooked, build a fresh one from the list and throw the old one away — disposable, like paper plates.
Last comes watching: put dashboards on the whole fleet, on a screen the team walks past, so anyone can see at a glance when something has gone red.
Writing everything down as a list is real up-front work, and some ancient systems cannot be written down at all — nobody alive remembers how they were installed — so they stay fossils, tended by hand.
Stop raising machines by hand. Write down what each machine should look like, keep it in version control, and let only automation build, change and correct them. The moment a person logs in and edits something, that machine can never be reproduced again.
Want the actual mechanics — how the definitions are written, how state converges, what virtualization and cloud changed? → Switch to the deep read
You think environments are "an ops problem." In fact the environment is part of the software. This chapter pushes Ch2's "everything under version control" from code out onto the machines: every piece of infrastructure configuration — operating system, middleware, network devices, monitoring — must be modeled, versioned, and changed only through an automated process. There is one test: can you rebuild the whole environment from version control at any time? Anyone logging into a production box to edit a line is a defect, not an operations technique.
This chapter opens Part III, "The Delivery Ecosystem." It pushes the argument of Ch2 (Configuration Management) off application code and onto machines and environments, and it catches the question Ch10 (Deploying and Releasing) left open: blue-green, canary — deploy to what, exactly? It leads into Ch12 (data) and Ch13 (components and dependencies). Its descendants are the tools everyone uses today: Terraform, Ansible, container images, Kubernetes, GitOps.
Picture a typical enterprise around 2010: requesting a new environment means filing a ticket, and weeks pass before machines appear; ops installs them by hand, ticking off a Word document; and over the next three years every emergency config edit, every hastily installed package, every port opened in a hurry lands nowhere near version control. Three consequences reinforce each other:
The price tag is public. On 1 August 2012 a Knight Capital technician deployed a new release by hand to eight servers and missed one of them, with no second person reviewing the work; the firm lost over $460 million in roughly 45 minutes (see §6). One manual operation and one inconsistent machine — that was the entire cause.
The chapter opens on people, not technology, because continuous delivery is most easily misread as "developers bypass ops and push to production themselves," which is backwards. The book lists four real needs that every technique afterwards has to satisfy at the same time:
RTO and lose no more data than RPO allows. The point is that the recovery procedure must have been genuinely rehearsed — a restore plan that has never been run is not a plan.The hardest rule in the chapter: any change to testing and production environments must be made through an automated process, and there is no second path. That means two things — access control: environments are locked so that nobody, including ops itself, can log in and edit; and a change channel: to change something you change the definition in version control and let automation apply it to every environment.
Why so absolute? Because reproducibility is binary: one change that never reached version control makes the entire machine unreproducible — there is no such thing as 95% reproducible. The side benefits are concrete too: changes leave an audit trail automatically, environments can be rebuilt on demand, and the gap between test and production shrinks to a handful of values in a config file.
The chapter splits a machine's life in two. First, provisioning — blank to usable, by one of three routes: by hand (not repeatable, only when nothing else works); automated remote installation (network boot, unattended install from an answer file — the PXE plus Kickstart family); or virtual images (cloned from a prepared template, the fastest).
Second, ongoing management — keeping the machine correct across the years it lives, which is the most valuable technical passage here. With Puppet, CfEngine or Chef you do not write the steps to take; you write the desired state (this package installed, this file's contents exactly this, this service running). The tool compares actual against desired and fixes only the difference. Two properties make it work: idempotence — the same definition applied repeatedly gives the same result, so it is safe to run every 30 minutes; and convergence — each run moves the machine one step closer to the desired state. Drift is therefore corrected continuously.
One criterion from here gets quoted endlessly: rebuilding a machine should be cheaper than repairing it. Once that holds, the operational mindset changes — no forensic debugging of a crooked machine at 3 a.m.; throw it away and rebuild from the definition. This is also the seed of immutable infrastructure.
The section most often skipped, and the one that causes the most real-world trouble. Middleware — web servers, application servers, message queues — must be managed the same way, and the book's three questions work directly during selection or when inheriting a legacy system. How is it configured? Can it be driven from text files in version control — because anything that can only be clicked through a GUI, with no way to export the result as text, cannot enter automation, and that is a hard selection criterion. Where does it keep state? What does it write to local disk (messages, sessions, caches, logs), which decides whether you can destroy and rebuild the box at will. How is it installed, upgraded, and deployed into? Whether all three can be scripted and run unattended.
One layer further out sit the infrastructure services: DNS, firewall rules, router and switch configuration, load balancers, SMTP. These are configuration too, and they belong in version control and under automation just as much — yet in practice they stay manual for years and become the last black box in the pipeline. The book also flags multihomed systems: when a machine sits on several networks, the environment definition has to capture the network topology itself, or "the same configuration" will behave differently on different segments.
Every argument so far hit the same physical obstacle in 2010: machines were expensive and slow to get. Virtualization removed it — environments become things you clone from a baseline image, available in minutes and destroyed after use. The book draws two uses from that:
2 hours serially, split across 10 machines, lands in the low tens of minutes (not exactly a tenth — uneven splits and environment start-up both cost). That difference decides whether acceptance tests can stay in the pipeline at all.One step further is the cloud. The book separates IaaS (rent machines and networks, still yours to manage) from PaaS (hand over code, the platform does the rest), and stresses that the real prize is elasticity on demand: need a hundred machines for an hour of testing, start a hundred and shut them down afterwards. At the same time — and this is to the chapter's credit — it reproduces the criticisms of cloud as they stood: the security and compliance risk of putting data in someone else's hands (in particular which jurisdiction it lands in), dependence on and lock-in to a provider, and the fact that not every workload is cheaper in the cloud. Its conclusion is that one size does not have to fit all: move what needs elasticity, leave the rest, and expect a hybrid.
The closing section is the loop being closed — if you claim an environment is in a given state, you need a way to show that it actually is. The book gives four layers:
Table 1 · Three ways to provision a machine: cost, repeatability, and where each fits
| Manual install | Automated remote install (PXE + Kickstart family) | Virtual image / template (later: container images) | |
|---|---|---|---|
| Up-front cost | Almost none | Medium — build an install server, write answer files | Medium — build image creation and distribution |
| Time per machine | Hours to days, and never quite the same twice | Tens of minutes, unattended | Minutes, sometimes seconds |
| Repeatability | None — depends on who installed it that day | High, guaranteed by answer files and definitions | Highest — identical to the byte |
| Auditability | Hand-written docs, effectively none | Definitions live in version control | Images are versioned, traceable to their inputs |
| Main cost | Unreproducible and unauditable | Still drifts after install; needs continuous correction | Images themselves must be version-managed, or the image library becomes a new pile of snowflakes |
| Use when | Only for legacy gear that cannot be automated | Physical machines, bare-metal starting points | The default; virtualized, cloud and container estates |
Table 2 · Two ways to keep machines correct: convergent (this chapter) vs immutable (what came after)
| Convergent config management Puppet / CfEngine / Chef / Ansible | Immutable infrastructure image + replace (containers / AMIs) | |
|---|---|---|
| How you change it | Edit the desired state; the tool pulls existing machines back | Build a new image and replace the machine; destroy the old one |
| Drift | Happens; corrected by periodic re-runs (say every 30 min) | Cannot occur — machines are never edited |
| Rollback | Revert the definition, wait for the next convergence | Redeploy the previous image, seconds |
| Hard prerequisite | Definitions must be idempotent; the tool must be reliable | All state must live off the machine (database, object store, external logs) |
| Poor fit for | Very large fleets with very frequent change, where the correction window is itself a risk | Stateful legacy systems, heavyweight middleware that resists containerization |
| Choosing today | Physical machines, legacy estates, incremental cleanup of existing boxes | The cloud-native default; often combined — use config management to build the image |
Table 3 · Where environments live: on-premise vs IaaS vs PaaS (criteria and criticisms both from this chapter)
| On-premise | IaaS | PaaS | |
|---|---|---|---|
| You manage | Facility, hardware, OS, middleware, app | OS, middleware, app | The app only |
| Time to a new environment | File a ticket, weeks | Minutes, on and off on demand | Minutes |
| Elasticity | Buy for peak, idle the rest of the time | On demand: a hundred machines for an hour, then gone | Platform scales for you |
| Main cost | High capital outlay, slow to grow | Data residency and compliance, provider dependence; not every workload is cheaper | Deepest lock-in — the app is written to the platform's rules |
| Use when | Hard compliance limits, steady load, existing investment | Most cases, especially elastic test environments | Standard web apps, teams that would rather not touch infrastructure |
Behind all three tables the chapter has exactly one overall test, and you can apply it on the spot: destroy any production server — can you rebuild it identically, within an acceptable time, using only what is in version control? If yes, you have done the work above. If no, then whatever tools you bought and whichever cloud you moved to, you are still raising snowflakes.
Nearly the whole cloud-native toolchain is this chapter delivered: Terraform and CloudFormation put networks, load balancers and DNS — the things that are not servers — into code, exactly as section 4 demanded; Ansible, Puppet and Chef are the direct descendants of desired state plus idempotence plus convergence; container images take image-based provisioning to its limit; and the Kubernetes reconciliation loop — you declare the desired state, controllers keep dragging actual state towards it — is essentially the industrialized version of the convergence curve in Fig 2. GitOps turned "version control is the single source of truth" into a product category.
This is a frequent interview and design-review topic. Asked "how do you manage your infrastructure," the answer that scores is not a list of tools but a few facts: when was the last time a person logged into a production machine and changed something? If a production box is destroyed, can it be rebuilt from version control alone, and how long does that take? Are middleware and firewall rules in version control? When was backup restore last genuinely rehearsed?
1. In one line: the environment is part of the software — every piece of infrastructure configuration modeled, versioned, and created or changed only by automation.
2. The single hard test: can you rebuild any machine identically from version control alone? If not, what you are running is a snowflake.
3. The hardest rule: no manual logins to change configuration. One manual edit diverges the machine from version control permanently, and the next automated run silently overwrites it — a common source of ghost failures.
4. It opens on operations' four needs (documentation and auditing, alerting, service continuity via RTO/RPO, technology ops is comfortable with): automation turns out to be the cheapest way to produce an audit trail, so compliance and automation are not opposed.
5. Three provisioning routes: manual / automated remote install (PXE + Kickstart) / virtual images; ongoing management via declarative desired state, idempotence and periodic convergence. And rebuilding a machine should be cheaper than repairing it — the seed of immutable infrastructure.
6. Do not skip the things that are not servers: middleware, DNS, firewalls, load balancers belong in version control too; when choosing middleware, ask first whether configuration can be driven from text and where it keeps state.
7. Virtualization and cloud are what first made "rebuild beats repair" true, and unlocked parallel testing on disposable environments; the book is honest about the costs: compliance, lock-in, and not always cheaper.
8. The monitoring loop: collect → aggregate → information radiator → behavior-driven checks, so "is this environment correct" has an automated answer. Today's descendants: Terraform, Ansible, container images, the Kubernetes reconciliation loop, GitOps.