BOOKS DEEP-READ · CONTINUOUS DELIVERY · CH 2
Continuous Delivery · Ch 2 · Jez Humble & David Farley · 2010
Almost every company has that one server nobody dares touch. Something important runs on it, but how it was originally set up — and who changed what since — is no longer knowable. The day it dies, the company rebuilds it by archaeology on a few long-timers' memories. Chapter 2 of Continuous Delivery is about making that machine extinct: write down how the whole system is assembled, store it, and record every single change to it.
A coffee shop wants to open a second location. If the first shop's taste lives in the manager's hands — grams of beans by feel, machine pressure tuned once years ago and never written down — the second shop will never taste the same. But if every step is written into a manual anyone can follow, then opening the tenth shop, or rebuilding the first one after a flood, is just following the manual again. Configuration management builds that manual for a software system — and the manual itself goes into the archive, where each edit leaves a record: who, when, what.
Three things. First, changes happened on the machine, not in the manual: someone logged in at midnight, tweaked a setting, fixed the problem — and from then on that tweak lived only inside that one box. Second, over time no two machines were alike any more: the same package installed fine here and misbehaved there, and nobody could say what the difference was. Third, there was no "yesterday, when it was good" to return to — you only discover you don't know what yesterday looked like once you need it back.
Two plain sentences hold the chapter. First: everything that determines what the system looks like goes into the archive — not just the program, but settings, install scripts, third-party parts, even "which operating system and which patches belong on this machine". The acceptance bar is blunt: a brand-new colleague, on a brand-new laptop, should be able to take one copy out of the archive, run one command, and stand the whole system up.
Second: one batch of goods ships to every branch; only the label changes. The package you built is the very same package from testing all the way to going live — never one build per environment. The differences between environments (which database, whether real email goes out, which features are on) are pushed in from outside at install time. That way the thing you tested and the thing users get are the same thing — which is the only reason any of your testing counts.
And one iron rule: from now on, nothing gets changed by hand on a machine. To change something, change the definition in the archive and let automation push it out to every machine.
A broken machine can be thrown away and rebuilt identical; a bad change can be rewound to any past state; when something breaks you can find who changed which setting when. The honest cost: scripting everything, and then maintaining that manual forever, is clearly slower up front — and once the settings multiply, the manual itself grows into another complicated thing that needs careful tending.
There is one question that tells you whether configuration management is real: given the archive and a pile of blank machines, can you rebuild the entire system exactly as it was? If yes, you always have a way back. If no, your system still lives in a few people's memories.
Want the mechanisms, the four-question test, the comparison tables and the diagrams? → switch to the deep read
You think configuration management is basic hygiene — "the code is in Git". It actually asks something far harsher: given a version control repository and a pile of blank machines, can you reproduce the entire system — OS, patches, middleware, application, configuration, data — exactly? Chapter 2 of Continuous Delivery gives a definition and a four-question test, then lays down three rules: put everything under version control; deploy the same binary artifact to every environment, injecting environment differences as external configuration at deploy or run time; and create environments only through automation from the repository, where any manual change counts as a failure. On an irreproducible environment no test result means anything — which is why this chapter is the foundation for the pipeline of Ch 5.
Chapter 2 of Part I ("Foundations"), arriving straight after Ch 1's three release anti-patterns — deploying by hand, only touching a production-like environment once development is finished, and managing production configuration manually. This chapter answers the last two head-on. It is also the book's foundation: Ch 3 (continuous integration) needs a trunk that always builds, which needs this chapter's daily check-in; Ch 5 (the deployment pipeline) needs "the same artifact promoted through the gates", which needs this chapter's environment-independent build; Ch 11 expands environments, Ch 12 data, Ch 13 dependencies. Today's incarnations: Git, Terraform / Ansible, container images and registries, Kubernetes ConfigMaps and Secrets, config services such as Consul / etcd / Apollo, and GitOps.
Start with something concrete: an order service on 30 machines, 3,000 QPS normally and ten times that at peak. Over six months, to rescue two incidents, someone bumped the JVM heap and connection pool by hand on 3 of them and wrote nothing down. Three consequences arrive together: p99 latency (the 99th slowest request out of a hundred) spikes on some machines only and a week of digging finds nothing; staging cannot reproduce production no matter how hard you load it; and when a datacenter failure forces you to rebuild the cluster, all you hold is an out-of-date document. The chapter turns "do we actually have configuration management?" into a test you can answer on the spot:
The verdict is sharp: one unanswered question and you don't have it — and most teams stall on the first. What if you don't fix it? You lose more than efficiency: you lose the very notion of a known-good state. Without it, rollback is just a different kind of guess, capacity tests and canaries stop being comparable, and the recovery time you promised in your DR plan cannot be verified. Google's public SRE material says roughly 70% of outages come from changes to a live system — and configuration changes are precisely the class of change that most often skips review, testing and staged rollout.
The book's definition: configuration management is the process by which all artifacts relevant to your project, and the relationships between them, are stored, retrieved, uniquely identified and modified. Two words deserve pulling out. Uniquely identified: you must be able to say straight away "production is running artifact version X, configuration version Y, on environment definition Z" — and the correspondence between the three cannot live in someone's head. Modified: changes travel a controlled path, rather than whoever has access reaching in. Tools are only means: plenty of teams have the code in Git while the environments and the configuration live in one person's memory plus a few machines. That is not configuration management.
The list is deliberately literal: source code, tests, database scripts, build and deployment scripts, libraries, configuration files, documentation, even the versions of compilers and tools. The acceptance criterion fits in a sentence: a new person on a brand-new machine checks out one revision, runs one command, and can build the system and deploy it to any environment. Two disciplines come with it: check in to trunk at least once a day (small changes → small conflicts, CI that actually means something, and a suspect list that is just this one commit when something breaks), and write meaningful commit messages — when the build goes red, your message is the first thing anyone reads.
The single exception: build outputs do not go into version control. They are regenerated deterministically from source, they are large, and they bloat history. They belong in an artifact repository, uniquely identified by version and written once. That exception is not a compromise — it is the physical basis for a pipeline that promotes the same artifact through every gate.
External libraries are configuration too. A mid-sized Java service easily pulls in hundreds of transitive jars; let any one of those versions float and your "unchanged source" builds a different artifact. The chapter's demand is to pin dependency versions and manage them: in 2010 that meant Maven / Ivy repositories with an internal mirror; today it means lock files (package-lock.json, go.sum) plus an internal artifact proxy. The test is concrete: with the internet cut off, or after an upstream author deletes a version (which has really happened), can you still reproduce last week's artifact?
First separate the moments. Configuration can enter the system at four of them — build time (compiled into the output), package time (a different package per environment), deploy time (injected at install), and run time (read at start-up or pulled dynamically). The chapter's position is unambiguous: build and packaging must be environment-independent, and building a separate binary per environment is to be refused. Not out of tidiness: the moment you build once specially for production, the thing you tested is not the thing you shipped, and every gate result in the pipeline loses its force.
For modelling, the chapter treats configuration as a set of key-value pairs keyed by the triple (application, version, environment) — different versions of an application may need different options, and the same version takes different values per environment. Where do the values live? Files in version control, a relational database, a directory service (LDAP / Active Directory), or a dedicated configuration service — each with its own price (next section).
Three principles from the chapter still hold. Configuration must be testable: run smoke tests right after deployment (a small set of "is it up, can it reach its dependencies?" checks), and validate required settings at start-up so a missing one fails fast instead of exploding on the first real request at 2 a.m. Configuration must be reportable: you need to see, at a glance, what all four environments hold right now — otherwise "it passed in staging but not in production" is never solvable. Configuration must be versioned and audited.
And the warning that gets ignored most: don't make everything configurable. Past a certain number of options, configuration becomes a programming language with no type checker, no IDE and no tests — written by whoever is on the worst night shift. The test: if changing an option requires a programmer to reason about the consequences, it is code — so give it review, tests and staged rollout.
Applications never run in a vacuum. An environment includes machines and capacity, OS and patch level, middleware with its versions and settings, network topology and firewall rules, DNS and certificates, external systems it depends on, and data. The chapter lays down two iron rules:
In 2010 the tools were Puppet and CfEngine; today's images plus infrastructure as code simply take the same rule to its extreme: immutable infrastructure abolishes the act of changing altogether and rebuilds instead. That also answers a common puzzle — why did configuration management seem to get easier after containers? The problem didn't go away; "no manual changes" is now enforced by the environment itself.
The arguable parts are not "should this be in version control" but three concrete choices: when configuration is injected, where the values live, and which posture you take with environments.
Table 1 · When configuration is injected (costs included)
| Moment | How | Upside | Cost | Book's stance |
|---|---|---|---|---|
| Build time | environment values compiled into the output | no external dependency at run time | a binary per environment; tested ≠ shipped; changing a value means rebuilding | against |
| Package time | packaging step emits a different package per environment | works with off-the-shelf build tools | same problem one step later; packages = environments | against |
| Deploy time | inject config files / env vars into the same artifact at install | one artifact all the way; no rebuild to change a value | changing a value means redeploying (seconds today — acceptable) | recommended |
| Run time | read at start-up, or pulled live from a config service | behaviour changes without restart; good for flags and throttles | one more dependency and failure mode; a live change is still a production change — review it, stage it | fine, but treat as change |
Table 2 · Where the values live
| Option | Upside | Cost | Fits |
|---|---|---|---|
| Files in version control | history, diffs and review for free, same tooling as code (today: GitOps) | each change runs the pipeline; credentials cannot sit in plaintext | environment definitions, infrastructure parameters, most app config |
| Environment variables | language- and platform-agnostic, minimal, hard to commit by accident | no history, no types, awkward for nested structure; restart to change | connection strings and credentials for containerised services |
| Relational database | easy to query and report on, one change applies everywhere | versioning and audit are yours to build; becomes a new single point | monoliths already heavily coupled to a database |
| Directory service (LDAP / AD) | already present in the enterprise, centrally managed | weak expressiveness, heavyweight change process | intranet accounts and topology data |
| Dedicated config service | dynamic, stageable, auditable (ZooKeeper / etcd / Consul / Apollo) | must itself be highly available; client caching and fallback need design; easily becomes an ungoverned back door | large estates, many services, dynamic flags |
| Secret manager | encryption, rotation, on-demand delivery, access trails (Vault / KMS) | another dependency to run | all credentials — barely covered by the book; a later addition |
Table 3 · Which posture for environments
| Posture | Time to rebuild one host | Drift risk | Cost |
|---|---|---|---|
| Hand-tended (snowflake) | hours to days, by archaeology | high, and unmeasurable | nothing up front, uncontrollable later |
| Scripted convergence | minutes (Puppet / Chef / Ansible) | medium — only what the manifests cover; anything unlisted still drifts | manifests to write and maintain; scripts must be idempotent |
| Immutable image rebuild | seconds to minutes (container image / AMI) | low — there is no act of changing | needs an image pipeline and artifact store; stateful services need more |
Read the three tables together and the recommended combination is plain: one environment-agnostic artifact, version-controlled environment definitions, configuration injected at deploy time with a little of it tunable at run time — and every change travelling the single route "edit the repository → automation pushes it out".
This chapter is the foundation everything later stands on: without reproducible environments, Ch 3's continuous integration is a green light for decoration and Ch 5's pipeline gates prove nothing — what you tested on environment A was never the thing going to environment B. The most direct incarnation today is GitOps: desired state lives in Git and a controller continuously converges the cluster to it, so manual changes get reverted automatically. The interview staple "where does your production config live, who can change it, how do you roll back, how do you audit it" is this chapter; the sharpest question in an architecture review is "can this be rebuilt from the repository?"
70% and Facebook's practice point the same way.① The thesis: configuration management is not "we use Git" but an answer to one question — given the repository and blank machines, can you reproduce the whole system exactly? The definition's key words: all artifacts and their relationships, uniquely identified and modified through a controlled path.
② The four-question test: can you reproduce any environment? make an incremental change and deploy it? trace who changed what and when? pass an audit, with everyone able to get the information they need? — one gap means no.
③ Keep everything in version control: code, tests, database scripts, build and deploy scripts, dependencies, configuration, docs, even toolchains; the one exception is build output, which goes to an artifact repository, uniquely versioned. Two accompanying disciplines: check in to trunk at least daily; write commit messages that mean something.
④ Dependencies are configuration too — pin them. The test: after losing internet access or an upstream deletion, can you still reproduce last week's artifact?
⑤ The core claim about application configuration: build and packaging must be environment-independent and the same artifact deployed everywhere, with differences injected at deploy time (a little at run time); configuration is keyed by (application, version, environment) and must be testable, reportable, versioned and audited.
⑥ The warning: don't make everything configurable — an option whose consequences need a programmer to reason about is code.
⑦ Two iron rules for environments: created automatically from version-controlled definitions (idempotent scripts), and no manual changes at all, with production locked down until they are impossible. Immutable infrastructure is that rule taken to its limit.
⑧ Evidence: DORA finds config-in-version-control predicts delivery performance even better than code-in-version-control; Facebook changes configuration thousands of times a day with review and canaries attached; the counter-example is AWS in 2017, one mistyped input and roughly four hours of S3 disruption. The honest cost: scripting everything and maintaining it forever is slower up front.