BOOKS DEEP-READ · CONTINUOUS DELIVERY · CH 14
Continuous Delivery · Ch 14 · Jez Humble & David Farley · 2010
A piece of software has dozens or hundreds of people editing the same pile of files, and at the end it all has to add up to one thing you can ship. The tool that keeps this from descending into chaos is called version control. This chapter isn't about how to drive the tool. It asks the one question no team escapes: should everyone take their own copy, work apart, and combine at the end — or should everyone fold their work back into the shared copy every day?
Two colleagues work apart for two months. On merge day the tool cheerfully reports no conflicts — not a single line needs a human. Then the code runs, and it's broken.
Why? The tool only checks whether two people edited the same line of text. One of them quietly changed how a shared routine calculates something; the other wrote new logic elsewhere that was counting on the old behaviour. Different files, different lines — of course the tool saw no conflict. But what came out is already wrong. "The merge went smoothly" and "the merge is correct" are two different claims.
The pain of working apart isn't spread evenly over time — it snowballs. Work apart for a week and the two of you probably touched different corners. Work apart for two months and you've both renovated the same set of files; combining goes from "a few spots arguing" to "nobody can say whether the result is still right."
Worse, it makes people timid. You know some piece of code should be tidied up — but tidying it means everyone else suffers extra on merge day. So nobody touches it, and the bad parts keep piling up.
The book is blunt: everyone folds their changes back into the shared copy every single day, so the window in which the two sides can drift is too short for trouble to grow in it.
"But my work isn't finished — won't it leak out?" The whole value of this chapter is in answering that. It offers three moves. Leave the door shut: the code for a new feature can go in early, it just has no button, menu or address that reaches it yet; you open the door when it's done. Break a big change into a string of small ones, each of them complete and shippable on its own. Put up scaffolding: when you need to swap out an old part that everything depends on, first insert an adapter in the middle, hang the old and new parts off it at the same time, move users across a bit at a time, then take down both the old part and the adapter. Renovate one room at a time instead of moving house.
Being apart for a few hours, even a day, is fine. Being apart for weeks or months is the actual enemy. Which is why today's habit of "open a branch, merge it back the same day" is on the same side as this book, not against it.
The price has to be said out loud too: those switches and that scaffolding are themselves debt. They must be torn out once they've done their job — otherwise, the day someone trips over a switch that should have been deleted years ago, the damage is worse than working apart ever was.
Working apart looks like separating risk. It is actually saving risk up and paying for it all at once. The longer you stay apart, the more dangerous rejoining becomes — and the danger often arrives without an alarm. So: merge back daily, and solve the real problem ("my work isn't finished") with closed doors and scaffolding instead of distance.
Want the mechanisms, the bill for each branching model, and the diagrams? → Switch to the deep read
Chapter 14 of Continuous Delivery exists to install one idea: the hard part of version control isn't the tool, it's the branching strategy — because that strategy sets your team's integration rhythm. A branch looks like it isolates risk; what it actually does is defer the pain of integration and compound it with the length of the divergence. So the book's default answer is develop on mainline: use hidden entry points, incremental change and branch by abstraction to land even large changes on the mainline one small step at a time, instead of hiding them on a long-lived branch.
main in Git.This is the second-to-last chapter of Part III, "The Delivery Ecosystem": it follows Ch 13 (components and dependencies) and leads into Ch 15 (managing continuous delivery). It is really the flip side of Ch 3, "Continuous Integration" — Ch 3 lays down the rule that everyone commits to mainline at least daily, and Ch 14 answers the question that inevitably follows: so what about branches? In today's terms this is the git-flow / GitHub Flow / trunk-based argument, and the choice every team eventually has to make: merge when the feature is done, or merge while you build it.
Around 2010 the industry norm was the Subversion / ClearCase / Perforce pattern of long-lived branches plus periodic big merges: every team or every feature took a branch, ran on it for weeks or months, then merged in bulk before release. That period had a name of its own — the integration phase, or code freeze — days to weeks of humans resolving conflicts and fixing regressions. It was treated as a sign of order: branches kept people apart, so they couldn't disturb each other.
What happens if you don't fix it? First, merge cost doesn't grow linearly with divergence — it snowballs. A week apart and the changes mostly land in different files. Two months apart and both sides have refactored the same directories, and the conflict escalates from "a few spots arguing" to "nobody can say whether the result is still correct." Second, it backfires on engineering practice itself: a long-lived branch makes refactoring a bad deal. You know a class should be split, but splitting it means everyone else suffers more at merge time — so nobody touches it.
The book's verdict is direct: branching and continuous integration trade off against each other. The longer you sit on a branch, the less CI's "mainline is green" means to you — it proves other people's code works together, and says nothing at all about yours.
What the tool can shout about is a textual conflict: both sides edited the same stretch of the same file, the tool can't tell which to keep, so it stops and hands it over. That pain is visible and bounded — however bad, you know where it is.
The dangerous one is the semantic conflict: the two sides edited places that don't overlap at all, so the tool merges without a word — and what comes out is broken.
The easy version to spot: branch A renames a shared method and reorders its parameters; branch B adds three new calls against the old signature. Different files, different lines, zero conflicts — and the compile breaks immediately. That's the lucky case, because the compiler shouts. The nastier version compiles and runs, and only the behaviour has quietly changed: A switches a shared method's rounding from half-up to floor, and B's new reconciliation logic was relying on the old behaviour. That class of problem is only caught by tests — and the tests on both branches were green at the time, because each was running in the world as it existed before the merge.
This is exactly why the cost is superlinear: textual conflicts grow roughly with how much each side changed, while semantic conflicts grow with the interaction between the two sets of changes. So the book's prescription is not a better merge tool — it is to shrink the divergence window.
The book classifies branches by why you took one, and states a clear order of preference:
Developing on mainline is not "everyone crammed onto one line stepping on each other." It comes with a set of techniques for keeping mainline releasable at all times. The book gives three:
① Leave the door shut. The code for a new feature goes onto mainline early; there is simply no screen, route or menu entry that reaches it until it's done. The general form of this today is the feature toggle: code on mainline, switch off by default, opened by percentage during rollout. It decouples "feature complete" from "feature visible."
② Break a big change into a string of individually releasable steps. Swapping a database or a framework shouldn't be saved up into one lump: every step keeps mainline building and could ship that day. It costs more thought than one big change — but what it spends is design effort, and what it buys back is the gamble on merge day.
③ Branch by abstraction. The most useful move in the chapter, and the most misleadingly named — it involves no version-control branch at all. To replace a component everything calls: first insert an abstraction between the callers and the old implementation so every call goes through it; write the new implementation underneath, so old and new coexist; shift traffic across a piece at a time; once everything is over, delete the old implementation and the abstraction. Every step of that happens on mainline, and every step is releasable.
The chapter was written in 2010, when Git had just turned five, yet its read on DVCS holds up remarkably well. What changed: committing and sharing came apart — you can commit twenty times locally, tidy that into three clean commits, and push those, polluting nobody in between. And the cost of creating a branch fell to roughly zero, which is what produced fork-and-pull-request as today's default shape of collaboration.
What didn't change: what got cheap was creating branches, not merging them. The cost always sat on the merge side, and it is set by how long you diverged, not by the tool. Git lets you open a hundred branches effortlessly; that fact alone does nothing to make the hundredth merge easier. Once tooling drops the friction of creation to zero, the only thing still standing between you and a long-lived branch is discipline. As for the two enterprise worries about DVCS — no single source of truth and weak auditability — the book's answer is now standard practice: nominate one blessed repository as the sole authority, and let CI and releases recognise only that one.
Table 1 · The bill for each branching model (ordered by the book's preference)
| Model | Integration delay | Main risk | Discipline required | The book's view |
|---|---|---|---|---|
| Develop on mainline | Near zero (hours to a day) | Unfinished code and toggle debt sit on mainline | Merge at least daily; pair with hidden entry points / incremental change / branch by abstraction | The default |
| Branch for release | Zero (all dev on mainline) | A hotfix never merged back → regression next release | Critical fixes only, merged back immediately | Endorsed |
| Branch by team | Medium (instant in-squad, days across squads) | Cross-squad semantic conflicts bank up until merge day | A CI per branch plus one on mainline; forced frequent merge-back | Workable but costly |
| Branch by feature | High (a feature cycle, often weeks) | At odds with CI by definition; refactoring gets suppressed | Lifetime in days; merge mainline into the branch often; refactorings go to mainline | Most reserved |
Table 2 · Four ways to land unfinished work — and when to use which
| Technique | How | Cost | When |
|---|---|---|---|
| Hidden entry point / feature toggle | Code lands on mainline with nothing reaching it, or gated by a configuration switch | Doubles the code paths; toggles need a retirement discipline | New functionality that doesn't disturb existing behaviour |
| Incremental change | Split the big change into a string of individually releasable steps | The change path has to be redesigned; more design effort | Changes that split naturally (adding fields, widening interfaces) |
| Branch by abstraction | Insert a layer → run old and new side by side → shift over → delete the old | A temporary layer and two implementations that must genuinely be removed | Replacing a component everything calls (swapping a library or a store) |
| Componentisation | Split it into its own component, integrated by version number (Ch 13) | Loses "one commit, one full verification"; bought back with pinned versions | Modules with clean boundaries that can ship and evolve independently |
Compressed into one portable test: the enemy isn't branches, it's divergence time. Today's "open a branch, merge it back the same day" habit is on the book's side — it simply implements "commit to mainline daily" as "merge daily." So to judge whether a team really does trunk-based development, don't ask whether they use branches; look at their median branch lifetime and how many branches are alive at once.
In 2010 this chapter swam against the tide — disciplined branching was still taken as a sign of maturity. A decade and a half later it has largely won: the default shape at mainstream internet companies is mainline plus short-lived branches, merged as soon as they're reviewed, and the git-flow model of long-running develop / feature / release lines has retreated to software with explicit version numbers and several released versions under maintenance. In an interview or an architecture review, this chapter converts into three questions: How long does your average branch live? How many are alive at once? Which line do refactorings go to?
Note: the egress proxy on this run blocked the domains hosting these sources, so the citations are given in text only rather than as unverified links.
git branch.1. The hard part of version control isn't the tool, it's that the branching strategy sets the integration rhythm: a branch defers the pain of integration and compounds it.
2. Merging has two layers. Textual conflicts the tool shouts about; semantic conflicts it doesn't — they merge perfectly and are still wrong. They grow with the interaction between the two sets of changes, which is why the cost is superlinear.
3. The prescription is not a better merge tool but a smaller divergence window — the enemy is divergence time, not branches. Long-lived branches also suppress refactoring: splitting a class hurts everyone at merge time, so nobody does it.
4. The order of preference: branch for release (endorsed) > by team (costly) > by feature (at odds with CI by definition) > by environment (rejected — that's configuration management).
5. The default is develop on mainline, with three knives for unfinished work: hidden entry points / feature toggles, incremental change, and branch by abstraction (insert a layer → both live → shift over → delete the old, all on mainline).
6. DVCS changed the cost of creating a branch, not of merging one. To judge whether a team really does trunk-based development, look at median branch lifetime and branches alive at once (DORA: under a day, fewer than three) — not at whether branches exist.
7. Toggles and abstraction layers are temporary debt that must genuinely be deleted — Knight Capital's $460 million is the cautionary tale for that discipline.