BOOK DEEP READ · CONTINUOUS DELIVERY · CH 13
Continuous Delivery · Ch 13 · Jez Humble & David Farley · 2010
The apps on your phone ship a new version most weeks. But they are not written by one person — they are written by dozens or hundreds of people all editing the same pile of code at the same time. Chapter 13 of Continuous Delivery asks the obvious question: while everyone is mid-surgery, how do you keep the thing ready to ship every single day? The answer has two halves — how to finish a big renovation without closing down, and what to do when the codebase grows too big for one build.
Picture replacing the entire kitchen of a restaurant that stays open for business every day. You cannot hang a "Closed for renovation — back in three months" sign; the customers would be gone. That is the whole chapter: how to swap the kitchen while still serving lunch, and — when the kitchen grows too big for one crew — how to split it into separate stalls without losing track of whether a full table's order still comes out together.
The old way was to go into seclusion: send a team off to work on it alone, and bring the work back three months later. It sounds reasonable and it is a disaster. Everyone else kept editing too, so by the time you come back the two versions no longer line up. Just reconciling them takes several late nights, and when it is done nobody will swear the thing still runs. The longer you stay apart, the worse the reunion — and it does not get twice as bad, it gets many times worse.
If ripping the kitchen out and putting a new one back is what hurts, then don't rip it out. Instead, build a serving hatch in front of the old kitchen: from now on the dining room orders through the hatch and never cares who is cooking behind it. Once the hatch is up, you can calmly assemble the new kitchen behind it while the restaurant stays open and the old kitchen keeps cooking. When the new one can produce every dish, you connect the hatch to it; after a while, once you are sure, you tear the old kitchen out.
The beauty of it: at no point is the place ever "closed for renovation." After every small step, you can open the doors again tomorrow.
When the kitchen is too big for one crew, split it into stalls that stock and cook for themselves. The benefit is concrete: changing one stall's menu no longer disturbs the whole restaurant.
The cost is that nobody is watching whether a full table's order comes out complete anymore. Worse is the wrong-screws problem: stall A's recipe calls for version 2.0 of a particular sauce, stall B's calls for 1.0 — and there is only room for one jar on the shelf. That is what programmers grumble about as dependency hell. The chapter's rule is blunt: the recipe must name the exact version of the sauce, never "whatever the newest jar is" — otherwise a dish you could cook today mysteriously fails tomorrow, and you changed nothing. Then you add a pass counter whose only job is to check that this particular set of stalls, at these particular versions, really does produce one complete table.
Once you split into stalls, the total hours the kitchen spends working usually goes up, not down. What you bought is "changing one stall does not disturb the rest," and somebody still has to pay for all that extra checking. Which is why the chapter keeps saying: don't split until you truly have to.
To keep software that people edit daily always ready to ship: don't go into seclusion for a big renovation — build a serving hatch and swap things out in place. Only split into stalls once the kitchen is genuinely too big, and when you do, pin the exact version of every part and add a gate whose sole job is checking that this combination of versions actually works together.
Want the mechanisms, the notation and the diagrams? → Switch to the deep read
You would assume applications get split into components for "clean architecture" or "reuse." The chapter's primary reason is far more practical: to get builds and feedback back under ten minutes. And it puts the cost on the table in the same breath — the moment you split, you lose the most valuable property continuous delivery has: one commit, one full verification. The whole back half of the chapter is about buying that property back with a dependency graph, an artifact repository, and one integration pipeline.
2.4.1 version convention, whose three parts signal breaking changes / new features / fixes.This chapter belongs to Part III, "The Delivery Ecosystem," after Chapter 11 on infrastructure and Chapter 12 on data. It is where the book pushes "always releasable" down into the structure of the code itself. It follows from Chapter 3 (continuous integration: trunk must stay buildable at all times) and Chapter 7 (the ten-minute commit stage budget), and leads into Chapter 14 on version control and branching. The real-world questions it maps onto: monorepo or many repos, how to pin versions in Maven / Gradle / npm, and who exactly verifies a combination of microservice versions.
Compress everything continuous delivery argues for into one line and you get "trunk is always releasable." By Chapter 13 that line has run into two walls.
The first wall is the large-scale change. Replacing an ORM, swapping a payment gateway, moving from home-grown auth to OAuth — work like this runs two or three weeks, sometimes months, and the code sits half-finished throughout. The traditional move is a feature branch in seclusion, merged at the end. But merge pain grows superlinearly with how long the branch lives: a one-week branch merges in fifteen minutes; a three-month branch takes several people several days, and afterwards nobody will vouch that it runs. That is the release-day hell of Chapter 1, moved into the code.
The second wall is build duration. Chapter 7 budgets ten minutes for the commit stage, because past ten minutes people start batching up their commits and continuous integration quietly dies. But codebases grow: a few hundred thousand lines with a few thousand tests routinely goes from an eight-minute build to a forty-minute one within a year or two. At that point a developer gets three or four trustworthy answers per day and the whole discipline of Chapter 3 collapses.
What happens if you ignore both? They lead to the same place: trunk stops being releasable — either because a long-lived branch is permanently "nearly done," or because a build nobody will wait for teaches everyone to route around the pipeline.
The chapter does not open with a decomposition strategy. It opens with three ways to protect releasability without touching the structure at all.
① Hide new functionality until it is finished. The code can merge to trunk and ship with every release — you simply do not wire up the entry point. As far as users are concerned it is code that can never be reached. "The feature isn't done" stops being a reason not to release.
② Make all changes incrementally. Break one large change into a series of small ones, with a hard test: stopping and releasing after any one of them must be safe. If that is not true, you have not really broken it up yet.
③ Branch by abstraction. The most useful trick in the chapter. The name comes from Paul Hammant, and one of this book's authors, Jez Humble, later wrote it up in his own right.
To replace a large part of a system without creating a version-control branch, five steps: ① introduce an abstraction layer over the part being replaced; ② change the rest of the system to depend only on that abstraction (usually the most laborious step, and the most valuable); ③ write the new implementation beneath the abstraction, alongside the old one; ④ point the abstraction at the new implementation, ramping by user or by traffic if you want; ⑤ once you are confident, delete the old implementation — scaffolding comes down when it is done, and often the abstraction goes with it.
The counterintuitive part: the act of branching has been moved out of version control and into the code. The repository holds one trunk and everybody integrates every day; the coexistence of two implementations is carried by an abstraction layer that compiles, tests and ships. Unlike a long-lived branch, its merge risk is smeared across every commit rather than saved up for one bad week.
The costs are real too. During the transition the codebase carries two implementations, which is harder to read and doubles some test surface. And if the abstraction covers only part of the old implementation's usage, the remaining call sites are back doors around the layer — and the whole scheme fails on switchover day.
Dependencies come in two flavours: build-time (needed to compile — a utility library) and runtime (needed only when running — a database driver). Almost all of the trouble is about versions.
The disease the book names is dependency hell — DLL hell on Windows, JAR hell in the Java world. Its classic shape is the diamond dependency: your application uses components A and B; A requires version 1.0 of some library and B requires 2.0 of the same one, while the class loader (or DLL loader) at runtime can only hold one. When the two versions are not API-compatible there is no clean answer — somebody has to give way, by downgrading, by adapting the code, or by isolating the loading.
The remedy is to manage your libraries. The book accepts two routes: check the binaries straight into a lib/ directory in version control (simple, absolutely reproducible, at the price of a bloated repository and entirely manual upgrades), or use declarative dependencies plus an artifact repository (the Maven / Ivy approach: easy upgrades, at the price of state living outside version control). Whichever you pick, one rule in this chapter is not negotiable: pin your versions, and never use dynamic ones like "give me the latest." This is not fastidiousness — dynamic versions make builds irreproducible: the same source is green today and red tomorrow while you changed nothing, and the cost of diagnosing that is brutal.
The book's definition of a component is roughly: a reasonably large-scale code structure with a well-defined API that could in principle be swapped for another implementation. Bigger than a class, lighter than an independently deployed service.
Its list of good reasons to split is deliberately short: builds and tests take too long; different parts change at very different rates (a reporting module untouched for a year need not rebuild alongside a checkout module that changes three times a week); you genuinely need reuse across products; the team is large enough to need clear ownership boundaries; some part must be deployed or scaled independently. The book explicitly rejects splitting to make the architecture look nicer, and its stance is: live with one pipeline until it stops working, then split off only the pieces you truly need.
After the split, the pipeline changes shape: each component gets its own pipeline, publishing versioned binaries into an artifact repository; downstream components build against upstream artifacts pulled from it; and an integration pipeline picks one specific set of versions (say A-12, B-7, C-3), assembles them and runs acceptance tests. One hard constraint applies: the dependency graph must be acyclic.
Triggering is a genuine problem here. Trigger every downstream the instant an upstream goes green, and a graph of a few dozen components turns one change into a build storm. Never trigger automatically, and a breaking change hides until the day somebody upgrades by hand — by which point dozens of changes have piled up and nobody can say which one broke it.
The chapter's answer to that dilemma is cautious optimism: stop treating "pinned" and "always latest" as a binary choice, and give every edge of the dependency graph one of three states.
One honest caveat: this mechanism rarely exists off the shelf. In 2010 you largely had to build it into your own build tooling, and today it is still mostly assembled from custom logic on a CI platform — it is better remembered as a design idea than as a switch you can flip.
Circular dependencies — A depends on B, and B depends back on A — are, in the book's words, probably the nastiest dependency problem. Most build tools simply refuse them. If you truly must get out of one in the short term, there is the build ladder: build B against the old A, then build the new A against that B, climbing one rung at a time. But the book is blunt that this is a stopgap, and the right fix is to refactor the cycle away — usually by extracting whatever both sides depend on into a third component.
The artifact repository is where Chapter 5's "build your binaries only once" lands in a componentized world. What it must do: hold every binary the pipelines produce, retrieve them by version, remember which commit each artifact came from, and — when an artifact is promoted from test to production — re-label it rather than rebuild it. Conversely, binaries do not belong in version control: they can be regenerated from source, and repositories were not designed for large files.
Table 1 · Three ways to make a large-scale change
| long-lived feature branch | branch by abstraction | hide the feature / toggle | |
|---|---|---|---|
| Fits when | the change would break trunk and the team accepts a stall | replacing an existing large part (ORM, gateway) | adding something not yet finished |
| Trunk releasable | not on the branch; trunk drifts away from it | after every step | always — the entry point is not wired up |
| Merge risk | grows superlinearly with branch age; three months often costs several people several days | smeared across commits, near zero | none |
| Cost | merge hell; integration problems all detonate at once | two implementations during transition; the layer must cover every call site and be removed afterwards | toggles accumulate as debt; needs a cleanup discipline |
Table 2 · Managing library dependencies
| How | Upside | Cost | |
|---|---|---|---|
| Binaries in version control | JARs / DLLs committed into lib/ | absolutely reproducible; check out and build, no external state | bloated repository; upgrades entirely manual; poor fit for many dependencies |
| Declarative deps + artifact repository | Maven / Ivy coordinates resolved from Nexus / Artifactory | easy upgrades; transitive dependencies resolved for you | state outside version control; repository down means no build |
| Pinned versions | write 1.2.3 | reproducible builds — the same source always yields the same result | security fixes and upstream improvements lag; needs a deliberate upgrade habit |
| Dynamic versions | write LATEST / ^1.2 | follows upstream by itself | the chapter rejects this: irreproducible builds, green today and red tomorrow with no change of yours |
Table 3 · Version strategies across components (the chapter's central table)
| build everything at once | static pinning | fluid | cautious optimism | |
|---|---|---|---|---|
| How it works | every component built from latest trunk source in one build | each edge pinned; humans upgrade | each edge always takes the newest green upstream | fluid by default; a failing edge rolls back and freezes |
| Integration lag | zero | large — sometimes weeks between upgrades | zero | near zero |
| Breaking change surfaces | at commit time | on upgrade day, with dozens of changes piled up | the same day | the same day, without blocking everyone else |
| Build cost | highest — full builds, viable only with incremental builds and remote caching | low | medium; upstream churn can cause build storms | medium |
| Seen in practice | Google / Meta / Microsoft monorepos | the npm and Maven ecosystems plus lockfiles | small graphs with high team trust | ahead of its tooling — little off-the-shelf support |
Table 4 · Should this become a component?
| Signal | Split? | Reasoning / cost |
|---|---|---|
| Commit stage exceeds ten minutes | Yes | the chapter's primary reason: slow feedback destroys the discipline of CI outright |
| Parts change at very different rates | Yes | a module untouched for a year need not rebuild with one that changes three times a week |
| Genuine reuse across products | Yes | reuse needs a stable API and an independent version line |
| A part must deploy or scale independently | Yes | this is the road to microservices, and it costs you the full distributed-systems bill |
| "The architecture would look cleaner" | No | explicitly rejected; modularity comes from packages and interfaces without paying for extra pipelines |
| The team is only a few people | No | five components means five pipelines and version coordination, for no matching benefit |
The loudest argument in the industry today — monorepo versus many repos — is just Table 3 in modern dress: one end says "every component builds from the latest source and is verified together," the other says "every edge is pinned and each part evolves on its own." This chapter costed out both ends in 2010, before either word existed.
Three things do need updating. First, incremental builds and remote caching in Bazel and Gradle have partly removed the chapter's primary reason to split — you can now plausibly keep one enormous repository and still get feedback inside ten minutes. Second, the lockfile ecosystem (package-lock.json, Gemfile.lock, go.sum) is the industrialized answer to pinning, and bots like Dependabot and Renovate are the industrialized answer to fluidity; together they land remarkably close to cautious optimism. Third, microservices pushed components all the way to independently deployed units, which turns this chapter's dependency-graph problem into a service-contract problem — consumer-driven contract testing does exactly the job of the integration pipeline.
2 billion lines of code and 86 TB of content, taking about 40,000 commits a day on its in-house Piper system. One of the headline reasons they give is simplified dependency management — with everyone on one version, the diamond problem disappears at the source. R. Potvin & J. Levenberg, "Why Google Stores Billions of Lines of Code in a Single Repository," CACM 2016 ↗300 GB and 3.5 million files, with roughly 8,421 pull requests and 1,760 official builds a day — large enough that they had to build a virtual file system (GVFS) before Git could cope. Building it all together is not free. Brian Harry, "The largest Git repo on the planet," Microsoft DevBlogs 2017 ↗LATEST and ^1.2 look convenient and make builds irreproducible. But pinning everything has its own cost — security fixes lag. The modern answer is lockfiles for reproducibility plus upgrade bots to stay current, a combination the 2010 book did not have.① In one line: this chapter pushes "trunk is always releasable" down into code structure — first how to make big changes without restructuring, then how to buy back "one commit, one full verification" when you must split.
② Two walls: the large-scale change (merge pain grows superlinearly with branch age) and build duration (past ten minutes, CI dies quietly).
③ Three moves that need no restructuring: hide unfinished functionality, make every change incremental, branch by abstraction.
④ Branch by abstraction, five steps: add the layer → route the rest through it → build the new implementation beneath → switch over → delete the old one and the scaffolding. It moves "branching" out of version control and into code.
⑤ Dependency hell's classic shape is the diamond: two paths demand different versions of one library while runtime holds one. The iron rule is pin your versions — dynamic versions make builds irreproducible.
⑥ A component is a reasonably large structure with a well-defined API that could be swapped wholesale. Split for slow builds, differing change rates, reuse, independent deployment; "cleaner architecture" is not a reason, and not splitting is the default.
⑦ The resulting shape: one pipeline per component → artifact repository → one integration pipeline that verifies a version set; the graph must be acyclic, and a build ladder is only a stopgap for cycles that should be refactored away.
⑧ Cautious optimism: edges are static, fluid or guarded; a guarded edge rolls back to the last known good version when upstream breaks it — a binary choice turned into a self-healing switch.
⑨ The artifact repository is where "build binaries only once" lands: store them, retrieve by version, trace back to a commit, and re-label rather than rebuild on promotion. Binaries stay out of version control.
⑩ Modern mapping: monorepo versus many repos is Table 3 today; lockfiles plus upgrade bots ≈ cautious optimism; microservices turn the dependency graph into a contract problem.