BOOKS DEEP-READ · CONTINUOUS DELIVERY · CH 3
Continuous Delivery · Ch 3 · Jez Humble & David Farley · 2010
Software isn't written by one person. It's a dozen or fifty people changing the same pile of stuff at the same time. Which raises the question: if everyone works separately, how does it all come back together? That's what Chapter 3 of Continuous Delivery is about — and its answer is slightly counter-intuitive: don't save it up. Merge every day.
Five people are co-writing a book, a few chapters each. Option one: everyone writes alone for three months, then you spend the last week stitching it together. You can picture how that goes — a character dies in chapter three and is alive again in chapter seven. Nobody can tell you in advance how painful that stitching week will be, because nobody knows how many contradictions are buried in there.
Option two: every day before going home, each writer folds their day's pages into the one shared manuscript, and the whole thing gets read start to finish right then. If it doesn't read straight, everyone stops and fixes it. Now each batch of contradictions is only one day's worth — small enough to settle on the spot. Continuous integration is option two.
So why do people save it up? Because merging is annoying, and saving it up feels more productive — finish the thing in front of you first. But contradictions don't disappear because you're not looking at them; they just grow quietly in the dark. Worse: under option one the manuscript can't be assembled and can't be handed over for most of its life. Only in the short window after stitching is it any good — and you never know how long the stitching will take.
This chapter wants to flip that default: the manuscript should read straight and be deliverable at any moment, breaking occasionally and getting fixed at once. Two rules do it.
First: everyone folds their changes into the single shared version every day, and folding it in triggers an automatic read-through of the whole thing. Second, and this is the part most teams can't manage: the moment that check fails, everyone stops what they're doing and fixes it, and nobody adds anything new until it's fixed. It's the factory line where anyone who spots a defect can pull the cord and halt everything — sounds like lost output, but it's exactly because you can afford to stop that defects never travel all the way to the customer. And if it can't be fixed, there's a fallback: go back to yesterday's version, the one that worked.
Software goes from "broken most of the time" to "working most of the time," and when something does break, the suspects are just the last small step. The honest cost: this takes a whole team holding a discipline for years — it's a habit, not something you get by installing a tool. Once people get used to ignoring that red light, the whole thing decays into a dashboard nobody reads.
Rather than saving up three months and merging once, painfully: merge every day, and check the whole thing right after every merge. If the check fails, everyone stops and fixes it first. The default state of software should be "it works."
Want the actual practices, time budgets, comparison tables and diagrams? → Switch to the deep read
You think continuous integration (CI) means "install Jenkins and run a build on every commit." What this chapter is actually about is a discipline: take "integration" out of that unestimable phase at the end of a project and break it into a small act each developer performs several times a day — every commit triggers a full build plus a comprehensive automated test suite, and the moment it goes red the whole team stops and fixes it. The goal is to flip the default state of the software from "it doesn't work, and occasionally it does" to "it works at all times, and occasionally breaks." The book says it flatly: CI is a practice, not a tool — its effect depends on team discipline, not on what you bought.
main in Git).Chapter 3 of Part I, between configuration management (Ch 2) and testing strategy (Ch 4): the version control and automated build from Ch 2 are exactly this chapter's prerequisites, and the fast automated test suite it demands is what Ch 4 unpacks. The loop it describes — commit, build, commit tests — gets a name in Ch 5: the commit stage, the first gate of the deployment pipeline. Today's equivalents: GitHub Actions / GitLab CI / Jenkins, Google's TAP, GitHub's merge queue, and the trunk-based development that DORA keeps validating.
Start with a concrete scene. A 20-person team builds a trading system, split into 4 squads, each on its own feature branch, agreeing to "merge after this cycle." Three weeks later, merge day, you hit three things at once: semantic conflicts (two squads refactored the same interface — no textual conflict, and it doesn't run once combined); a merge whose duration you genuinely cannot estimate, half a day or a week; and the expensive one — a fundamental design conflict was written down on day 3, and you learn about it on day 21. The cost of a defect rises monotonically with the gap between writing it and finding it, and batch integration deliberately stretches that gap to its maximum.
The book's verdict on the old world is cold: on most projects the default state of the software is "it doesn't work" — it is usable only in the brief window after an occasional integration. What you lose isn't just efficiency, it's predictability: you can't commit to a release date, can't answer "can we ship right now," and can't attach any test result to a specific version.
The book's definition splits in two. The first half is done by machines: every time anybody commits any change, the entire application is rebuilt and a comprehensive suite of automated tests is run against it. The second half is done by people, and it is the real dividing line: if the build or the tests fail, the team stops what it is doing and fixes it immediately.
Why is the second half the soul of it? Because only that keeps the invariant "the trunk works at all times" true. If red is allowed to sit there, the build result decays into a decorative metric and CI's value collapses on the spot. The spirit behind the rule is lean manufacturing's stop the line: anyone who spots a defect can halt the whole production line — apparent lost output, actual guarantee that defects never reach the finished product.
Before you start you need three things. Version control: everything related to the project in one repository, not "code in Git, configuration on somebody's laptop" (that's Ch 2). An automated build: one command builds and tests the entire application — no clicking buttons in an IDE, because a CI server only knows how to type commands. Agreement of the team: everyone accepts that this is the highest-priority activity. The third is the hardest and the most often skipped, which is how you get teams whose CI has been red for weeks and who simply walk around it.
The book adds four prerequisites for keeping it going: check in regularly (at least once a day, more often is better); build a comprehensive automated test suite (otherwise a green build only proves it compiles); keep the build and test process short (past ten minutes people stop waiting for it); and manage your development workspace (local results must match CI, or "works on my machine" eats all the trust).
Turned into actions: pull the latest trunk and merge locally → run the build and commit tests locally → commit only when green → the CI server does a clean checkout from the trunk, rebuilds, and runs the whole commit suite → red stops the line, green earns the artifact the right to move to the next gate.
Three details deserve their own note. Running it locally first is about not turning your own slip into a stoppage for the whole team. CI checks out cleanly from the trunk because your machine always carries residue you're not aware of — uncommitted files, locally installed dependencies — and only a machine starting from zero proves that somebody else taking this code can build it. Don't walk away right after committing: otherwise you're already deep in the next task, fixing the red costs far more, and meanwhile others keep committing on top of it.
Table 1 · The essential practices the book lists, and why
| Practice | Why |
|---|---|
| Don't check in on a broken build | Committing while red makes "who broke it" impossible to determine. Wait for green. |
| Run all commit tests locally first | Keeps your slip on your own machine — or have the CI server run them for you (pretested commit). |
| Wait for commit tests to pass before moving on | Until the feedback arrives you still have the context; that's when fixing is cheapest. |
| Never go home on a broken build | Nobody fixes it overnight and the whole team is locked out next morning — the book's most famous rule. |
| Always be prepared to revert | Reverting is the fallback that always works, provided every historical version of the trunk was once green. |
| Time-box fixing before reverting | Not fixed in ten minutes? Go back to the last green version rather than making the team wait on your debugging. |
| Don't comment out failing tests | That deletes a known constraint and disguises it as green — far more dangerous than red. |
| Own every breakage your change causes | Your change may have broken somebody else's test. That's still your job. |
The book also puts test-driven development (write the test before the implementation) in this section, for a very practical reason: CI's entire power depends on how comprehensive that automated suite is, and TDD is the most reliable way to get the tests actually written.
How long the commit tests take decides whether this discipline survives. The book's threshold is ten minutes: past it, people start batching several changes into one commit; commit frequency drops, the suspect set behind each red light grows, and the whole feedback loop breaks. So the commit stage carries only the fastest, most diagnostic checks, and the slow ones move later.
The book adds several "worth adopting" practices whose common thread is turning verbal agreements into rules that turn the build red. Fail on architectural breaches: if the rule is "the presentation layer must not touch the database directly," write an automated check for it — otherwise that rule only lives on a wiki page. Fail on slow tests: cap the runtime of an individual test, so the suite can't quietly drift from 3 minutes to 20 over a few months. Fail on warnings and style breaches: switching everything on at once drowns people, so the book prescribes ratcheting — record today's warning count as the baseline, go red on anything above it, and the number can only go down.
The chapter was written in 2010, when distributed version control (DVCS) like Git was just taking off. The book's stance: a fine tool, but one that makes not integrating far too easy. DVCS encourages a pile of local branches that never get pushed, and a change that hasn't been merged back into the shared mainline has not been integrated, however many times you committed locally — which is the origin of today's line, "using Git is not the same as doing CI." For distributed teams the answer is equally centralized: one central CI, one shared mainline.
Table 2 · Three branching strategies: how CI's power decays with branch lifetime
| Trunk-based | Short-lived branch + PR | Long-lived feature / team branch | |
|---|---|---|---|
| Branch lifetime | Hours, or straight to trunk | ≤ 1 day | Weeks to months |
| Conflicts surface | Hours after being written | Within a day | Weeks later, and in batches |
| What CI actually tests | The real integrated result | The merged result (needs a merge queue to guarantee it) | Just the branch itself, not integration |
| Cost | Half-built features must hide behind feature toggles; review needs another mechanism | One extra merge and wait | Unestimable merge; semantic conflicts arrive all at once |
| Fits | Disciplined teams with strong test suites; the book's first choice | The pragmatic compromise most teams run today | Only when long isolation is genuinely required (e.g. a major version fork) |
The point: CI's power decays with branch lifetime. A branch that lives a day costs you a day of feedback. A branch that lives three weeks means CI is only testing that branch — "green" says nothing about whether it will be green once integrated.
Table 3 · The build went red — four responses
| Response | Consequence | When |
|---|---|---|
| Fix immediately (ten-minute box) | Shortest path; overrunning holds up the whole team | The default, when the cause is obvious |
| Revert to the last green version | Trunk is usable again at once; the problem becomes one person's task | The standard fallback once ten minutes pass with no lead |
| Comment out the failing test | Anti-pattern: deletes a constraint and disguises it as green | Never; if it's genuinely flaky, quarantine it and file a ticket |
| Leave it red and keep committing | Anti-pattern: nobody owns anything, trunk is unusable for days | Never; this is how CI typically dies |
Table 4 · Where the commit tests actually run
| Locally before committing | Pretested commit / merge queue | Commit straight, CI runs them | |
|---|---|---|---|
| Chance trunk goes red | Low | Near zero (nothing lands unless it passes) | High |
| Developer waiting | A few minutes locally | Queued; tens of minutes when busy | None |
| Infrastructure cost | Low | High: a full build per candidate change | Low |
| Today's equivalent | Local make test / pre-commit hooks | GitHub merge queue, OpenStack Zuul | Only tolerable in small teams with very fast tests |
This chapter is the first slab of foundation under the book's pipeline: without "the trunk is green at all times," every later gate loses its starting point — you can't answer "which version do we run acceptance tests against." It's also the common substrate of today's practice: GitHub Actions / GitLab CI / Jenkins made "every push triggers a build" the default configuration, and merge queues turned "go green locally first" into infrastructure. When an interviewer asks how your CI works, what's being tested is rarely the tool name; it's this chapter: how often you commit, who owns a red build, how long commit-to-feedback takes, and whether anyone has ever commented out a failing test.
4.2 million tests and around 150 million test executions per day; yet Google itself states in print that even with those resources it cannot regression test every code change individually, and must select which tests to run — validating the practice and marking its scaling limit at the same time.J. Micco, "The State of CI Testing @Google", 2017 ↗ · "Taming Google-Scale Continuous Testing", ICSE-SEIP 2017 ↗100% of production web servers in April 2017, with changes going to employees first, then a 2% production canary, then everyone — the "trunk always works" invariant carried all the way into production release.Engineering at Meta, "Rapid release at massive scale", 2017 ↗1.5% of test runs are flaky and about 16% of tests have shown some flakiness — precisely where "don't comment out failing tests" gets broken in practice. Google's answer isn't switching tests off; it's detecting, quarantining and actively managing them.Google Testing Blog, "Flaky Tests at Google", 2016 ↗1 · In one sentence: CI takes "integration" out of that unestimable phase at the end of a project and breaks it into a small act performed several times a day, to flip the software's default state from "it doesn't work" to "it works at all times."
2 · The definition has two halves: the machine half is a full build plus a comprehensive automated test suite on every commit; the human half is stopping the team to fix red immediately — and the second half is the dividing line.
3 · CI is a practice, not a tool: the three prerequisites are version control, a one-command automated build, and agreement of the team — the last being hardest and most often skipped.
4 · Four prerequisites for keeping it alive: check in regularly (at least daily), a comprehensive automated test suite, build plus tests under ten minutes, and local results that match CI.
5 · Of the eight essential practices, three are worth memorizing: never go home on a broken build, revert if it isn't fixed in ten minutes, never comment out a failing test.
6 · The suggested practices share one idea: turn verbal agreements into rules that make the build red — architectural breaches, slow tests, style violations (ratcheted so the number can only improve).
7 · The key trade-off: CI's power decays with branch lifetime; green on a long-lived feature branch says nothing about green after integration, and the realistic compromise is sub-day branches plus a merge queue.
8 · Evidence and position: Google runs ~150M test executions a day and still can't run everything per commit; Meta ships continuously from master to 100% of production; DORA finds ≤3 active branches strongly associated with high performance. This chapter is the first gate of the Ch 5 deployment pipeline.