BOOK DEEP READ · CONTINUOUS DELIVERY · CH 8

Automated Acceptance Testing

Continuous Delivery · Ch 8 · Jez Humble & David Farley · 2010

What is this chapter about?

You tap a button, pay, and a confirmation message arrives — someone has to check that the whole chain actually works. The previous chapter's gate only inspected parts: every small piece looks fine on its own. This chapter is about the next gate: assemble the whole machine, run it end to end, and see whether the thing the user actually wanted got built.

An analogy

Think of the final walkthrough on a new apartment. Testing the concrete mix and measuring the rebar — that's parts inspection. The walkthrough is something else: you open every tap to see if water comes out, flip every switch to see if the light turns on, close every door to see if it locks. Every part can pass and the toilet still won't flush, and then nobody can live there.

Here's the counterintuitive bit: the walkthrough checklist is most valuable not on move-in day, but before construction starts. The moment you write it down, both sides discover that "enclosed balcony" meant two different things. Fixing that now costs one sentence; fixing it after the walls are up means knocking walls down.

Why the old world was hard

This job used to be done by people: a room full of them working through a checklist by hand. The trouble is that every change means, in principle, redoing the entire checklist — you touched the payment button, so who can promise signup still works? A manual pass takes days, so the team ships less often; shipping less often piles up more changes, and more changes make trouble more likely. That's the same loop from two chapters ago.

People also get tired doing repetitive checks. They skip. They think "it was fine last time, it's probably fine now." The most tedious work is exactly the work least suited to humans.

How this chapter fixes it

First, write the checklist as something a machine can run — in the language of the business. "A new customer places an order; once payment succeeds, they should receive a confirmation email." That one sentence is both the requirement and the test. Writing it with the customer, the analyst, the developer and the tester in the room is where the vague bits surface.

Second — and this is the real craft of the chapter — separate "what to check" from "how to click it." The thousands of checklist items only say "log in, place an order, pay." One separate layer underneath is the only place that knows where the login button lives and what it's called. Move the button and you change one line in that bottom layer; the thousands of items above don't change at all. Teams whose acceptance tests grow too expensive to keep almost always skipped this cut.

Third, if you can avoid going through the screen, avoid it. Driving the screen is slow and gets tripped by tiny layout tweaks. Talking to the system directly, behind the screen, does the same job ten times faster and far more steadily.

What you get

The main payoff is that people dare to change the code: rerun the whole business flow, see all green, and you'll press the release button. Its real identity isn't "a bug-finding tool" — it's a safety net that makes people willing to move.

It does have a cost: checks like these are inherently slow and expensive, tens of minutes a round at best, so they must always be the minority — most checking belongs at the cheaper gate before this one.

One line to remember

Acceptance tests don't answer "is the code correct?" — they answer "did the thing the user asked for actually get built?" And whether you can afford to keep them depends entirely on whether you separated what to check from how to click it.

Want the actual layering, why record-and-playback is a trap, and whether thousands of end-to-end tests can really run? → Switch to the deep read

In one sentence

An acceptance test doesn't ask "is the code right?" — it asks "was the value the user wanted actually delivered?" The counterintuitive part: this chapter is nominally about testing and substantially about requirements. Its single largest payoff arrives before any code is written, in the hour where analysts, testers, developers and the customer turn "what does done mean" into executable acceptance criteria. And whether the suite survives three months hinges on exactly one thing: did you split "what to verify" from "how to drive it"? If not, thousands of tests get flattened by every cosmetic UI tweak and the team quietly abandons them. If yes, the same thousands run for a decade.

Glossary

Acceptance test — an automated test written from the user's / business's point of view, run against the whole deployed system, verifying "this feature really works" rather than "this function computes correctly."
Acceptance criteria — the list of conditions that define "done" for a requirement, agreed before coding starts; this chapter argues they should be written in executable form.
Executable specification — something that is simultaneously a human-readable statement of the requirement and a program a machine can run. That's this chapter's definition of an acceptance test.
Given / When / Then — a phrasing for acceptance criteria (from Dan North's BDD): given this starting state, when this happens, then this should be true.
DSL (domain-specific language) — a small custom vocabulary for one business domain, so tests read in business terms, e.g. placeOrder(customerA, 100).
Application driver layer — the one layer of the test suite that knows how the system is actually operated: which API to call, which button to click, which message to send.
Window driver / Page Object — wrapping "what controls this screen has and how to operate them" into a single object, so a UI change touches only that object.
Capture-replay (record-and-playback) — recording a human's clicks on the UI and replaying them as a test.
Deployment test / smoke test — a tiny set of checks run immediately after deployment, answering only "is this environment and application alive?"
Production-like environment — an environment whose configuration, topology and middleware versions match production as closely as possible; acceptance tests run here.
Artifact (binary) — the deployable output built once in the commit stage (a jar, a container image). The book's iron rule is "build once": every later stage uses that same artifact.
Stub — a stand-in for a real collaborator during a test; in this chapter, mostly used to replace third-party external systems.
Test isolation — each test creates its own data and interferes with no other, so tests can run in any order and in parallel.
Flaky test — a test that passes and fails on identical code, usually broken by timing, concurrency or shared state.

Where this sits

This chapter follows Ch7 inside Part II. Ch5 drew the whole conveyor belt from commit to production, Ch7 zoomed into the first slot (the commit stage), and this chapter zooms into the second — the gate an artifact passes through once the commit stage goes green. Upstream, it inherits Ch4's testing quadrants, taking over the "business-facing, supporting the team" square; downstream, Ch9 picks up nonfunctional acceptance (capacity, security) and Ch12 expands the test-data problem this chapter only touches. In today's world it maps to Cucumber / FitNesse / Concordion, to Selenium and its descendants WebDriver, Playwright and Cypress, and to the "as production-like as possible" gamma environment in a cloud pipeline.

The problem this chapter solves

A green commit stage only says "this change didn't obviously break anything." It says nothing about whether the user got what they asked for — unit tests are written by developers for themselves, and they verify that the code behaves the way the developer intended, not that the intention was right. That gap is where the most expensive class of defect lives: the feature is finished, the code has no bugs, and it isn't what the customer wanted.

The old world plugged that gap with manual acceptance testing. Do the arithmetic and you see why it collapses: a mid-sized business system has roughly 150~250 primary scenarios, and a full manual regression pass typically costs 2 testers 2~3 days. At 12 releases a year that's 50~70 person-days; but continuous delivery aims at being releasable at any time, hundreds of times a year — at that frequency manual regression isn't expensive, it's arithmetically impossible. Worse is how it fails: attention on the fifth repetition is nothing like attention on the first. Manual regression is exactly the kind of work that gets less reliable the more you need it to be reliable.

The chapter also has to answer a hard objection, which the authors put on the table in the opening pages: "acceptance tests are too expensive, you can't maintain them." Their reply isn't denial, it's concession — badly written acceptance tests really are unaffordable — followed by a whole chapter on how to write them so they aren't. Which means the actual subject of this chapter is maintainability, not coverage.

The core, explained

Acceptance criteria are a requirements activity, not a testing activity

The chapter's first and most-skipped claim: acceptance criteria must be written before work on the requirement begins, and written jointly by analysts, testers, developers and the customer — they are simultaneously the specification, the sign-off checklist, and the skeleton of the eventual automated test. Why insist on "before"? Because writing them is an intense requirements-clarification exercise. The moment you have to turn "support coupons" into "given a spend-100-get-20-off coupon, when the customer places a 95 order, then the coupon is unavailable and the message says 5 short," the ambiguity surfaces. Caught before work starts, the fix costs a sentence; caught after the feature is built, it costs rework. This chapter's biggest single payoff lands before any test code exists.

Ownership follows from that: the chapter explicitly rejects outsourcing acceptance tests to a separate QA team — if the people writing the code aren't accountable for green, red becomes somebody else's problem and then accumulates. This is the same discipline as Ch3's "stop the line." Testers' value moves accordingly: away from repetitive manual clicking, toward defining acceptance criteria and doing exploratory testing — neither of which a machine can do.

Layering: the craft at the centre of this chapter

If you remember one diagram from this chapter, make it this one. Acceptance tests split into three layers (the fourth is the system itself):

Figure 1 · Layering an acceptance test: the top two layers speak only business and domain language, while UI and API details are locked inside the application driver layer — this one cut decides whether the suite is still alive in three months.

The rule lives in the wall between them: layers 1 and 2 may only use business and domain vocabulary, while "click the element with id btn-submit" may appear only in layer 3, and only once. The payoff is quantifiable: change the login page from "username + password" to "phone number + one-time code" and, without layering, every test that logs in first has to change — typically over 80% of the suite; with layering the change lands on the single login() method in the driver, and not one line above it moves. The maintenance cost of an acceptance suite is almost entirely determined by how many times UI detail was duplicated.

Which layer to drive from: stay below the UI when you can

The chapter's second key judgement reads like a contradiction on first pass: acceptance tests must exercise the whole deployed system, but the entry point should sit below the UI — driving the application through the public API that the UI itself calls.

Figure 2 · Three possible entry points: through the UI is the most faithful but slowest and most brittle; straight into domain objects is fast but has degenerated into a unit test; the chapter's workhorse is the public API just below the UI.

The reason is speed multiplied by brittleness. A UI-driven test typically takes 3~15 seconds (waiting on rendering and async loads); the same scenario driven through the public API usually lands at 100~300 milliseconds — one to two orders of magnitude. Worse is the brittleness: one frontend framework upgrade or layout change can turn a batch of tests red while the business logic is entirely intact.

So who tests the UI itself? The chapter's position: if the UI contains business logic, the first thing to do is push that logic out of the UI, leaving a small number of UI tests over the purely presentational remainder. The advice fits today's frontend era even better than 2010 — modern frontends carry a lot of logic, and the answer is still to make it independently testable rather than stacking thousands of end-to-end tests on a browser.

Why not record-and-playback

The pitch is seductive: no code, click once and you have a test. The chapter firmly rejects it as the mainstay of acceptance testing, on three solid grounds:

It requires the application to exist first. Acceptance criteria are supposed to be defined before work starts and go green as development proceeds; capture-replay can only be done afterwards, which discards this chapter's largest payoff.
It welds UI detail into every script. Recorded scripts are full of coordinates, control ids and page order — precisely what layer 3 in Figure 1 exists to contain — duplicated hundreds of times. One UI change voids all of it.
Nobody can read the output. The script expresses "where I clicked," not "what the business requires," so it can't serve as an executable specification and can't tell you which business rule broke.

The chapter does allow one honest exception: for a legacy system with no tests at all that you must start changing, weaving a coarse net with capture-replay is a reasonable stopgap — but it's scaffolding, not foundation.

How the acceptance stage runs: artifact, environment, smoke, parallelism

The chapter is quite prescriptive about the shape of this stage:

Figure 3 · The shape of the acceptance stage: the single artifact from the commit stage, deployed by the same scripts used for production into a production-like environment; smoke tests confirm the environment is alive, then acceptance tests run sharded in parallel.

Use the same artifact the commit stage produced (Ch5's "build once"). Rebuilding for acceptance means you aren't testing the thing that will ship.
Deploy to a production-like environment using exactly the same deployment scripts as production. This has an underrated side effect: your deployment process is rehearsed automatically many times a day, so by the time you go to production those scripts have run thousands of times.
Run deployment (smoke) tests first. A large share of acceptance-stage failures aren't application defects at all — they're misconfiguration, a service that didn't start, a dependency that isn't reachable. A handful of trivial checks confirming "the environment is alive" classifies those failures instantly and saves a long misdiagnosis.
Red stops the line, and the whole team owns it. This is harder to hold than at the commit stage: acceptance tests are slow and failures are noisy, so it degrades easily into "leave it red, we'll look later" — and a red left hanging spreads into everyone ignoring it.

On duration, the chapter gives no hard number like the commit stage's ten minutes, only a relative constraint: the acceptance stage may be slow, but it must be short enough to run many times a day. The engineering answer is parallelism — shard by scenario across machines, so the total is the slowest shard rather than the sum of all shards. How far that road goes is answered by the public LMAX numbers below.

State and test data: the dirtiest part of acceptance testing

A unit test can pretend the world doesn't exist; an acceptance test can't — it runs against a real system with a database and history. State is the source of every weird acceptance failure: the username already exists on the second run, an order left behind by the previous test pollutes a total, two parallel tests fight over the same account. The chapter offers three postures and states a preference:

Test isolation (preferred) — each test creates the data it needs under a unique key (e.g. user-7f3a91). The cost is needing a way to create data; the benefit is that tests can run in any order and in parallel — and parallelism is the only way out of "slow," so these are really one decision.
Adaptive tests — assert relative change rather than absolute values ("this customer's order count is one higher than before"). Suits environments that carry existing data; the cost is weaker assertions and more convoluted test code.
Test sequencing — a fixed order where each test consumes the state left by the last. Not recommended: it kills parallelism outright, and one failure cascades red through everything downstream.

The headline anti-pattern is a shared "big fat test database" for the whole team: fast to start with, then rotting — nobody remembers which row belongs to whom, nobody dares delete anything, and eventually it becomes a legacy asset of its own. Ch12 covers test data properly.

Asynchrony, time, and external systems

Running against a real system means facing three real-system problems:

Asynchrony — the confirmation email doesn't come back synchronously. The right approach is polling with a timeout — recheck at short intervals until the condition holds or the timeout fires — not fixed sleep calls sprinkled everywhere. A fixed wait is either wasted time (slowing the whole suite) or too short (a flaky test); it loses both ways.
Time — logic like "expires after 30 days" or "settles at month end" can't be tested by actually waiting. Make "what time is it" something the system exposes as controllable, so a test can move the clock.
External systems — third-party payment, credit and logistics providers are typically slow, unreliable, or have no test instance, so the chapter stubs them here. But keep honest books: whatever you stub isn't being verified, so a separate set of integration tests must exercise the real thing in another stage or at a lower frequency.

Key trade-offs

Table 1 · Which layer should a given acceptance test drive?

Entry point	Per test	Covers	Brittleness	When to pick it
Through the UI	`3~15` s	Whole chain incl. frontend logic and rendering	Highest: layout, ids or timing change and it's red	Keep a few primary paths when the UI genuinely holds logic
Public API below the UI	`100~300` ms	The complete business chain except the UI	Low	The chapter's workhorse entry point
Message / event interface	tens of ms ~ s	Async and event-driven paths	Medium: must handle waiting and out-of-order	The natural entry for event-driven systems
Domain objects directly	ms	One slice of logic, no integration	Minimal	This is a unit test — it belongs to the commit stage (Ch7)

Table 2 · How to make acceptance criteria executable

Approach	Who can read it	Maintenance	Cost / risk
Team-built DSL	Developers + business (with guidance)	Low	You must build and maintain that language; slow to start, cheapest long term
Given/When/Then tools (Cucumber, FitNesse)	Business reads it directly	Medium	If the business never actually reads it, you've added a layer of glue nobody looks at
Plain test framework (plus a driver layer)	Developers only	Medium-low	Loses the executable-specification value; degenerates into a regression net
Capture-replay	Nobody	Very high	Welded to UI detail, can't be written first, unreadable; only fit as legacy scaffolding

Table 3 · What to do about state and test data

Strategy	How	Upside	Cost
Test isolation	Each test creates its own data under unique keys	Any order, full parallelism	Needs a data-creation path and a cleanup policy
Adaptive tests	Assert relative change, not absolute values	Works in environments carrying existing data	Weaker assertions, more convoluted code
Test sequencing	Fixed order, each test eats the previous state	Easy to write	No parallelism; one failure cascades — not recommended
Shared "big fat" test DB	One preloaded dataset for the whole team	Fastest start	Cross-contamination, rots over time, nobody dares delete anything

Table 4 · The acceptance stage got too slow — now what?

Move	Effect	Cost / precondition
Shard by scenario across machines	Near-linear speedup, the single most effective move	Requires real test isolation; costs hardware (see the LMAX numbers below)
Move the entry point from UI to API	Commonly one to two orders of magnitude	Gives up UI coverage; keep a few UI tests separately
Stub slow, unstable external systems	Removes the longest waits	Stubs drift from reality; needs integration tests as backstop
Share expensive one-time setup	Avoids repeated environment startup	Sharing is coupling; slides quietly back into ordering dependencies
Delete redundantly covered scenarios	Often surprisingly large wins	Look at data; don't delete on instinct
Just run it nightly instead	Looks like the easy way out	Feedback drops from minutes to days — you're back in the Ch1 world

Table 5 · Which gate does a given check belong to?

What you want to verify	Where	Why
Boundary cases of a pricing function	Commit stage (Ch7)	Pure logic, verifiable in milliseconds without starting the system
Order → payment → confirmation	Acceptance stage	The business value only exists when the whole system is wired together
Is the environment alive?	Deployment (smoke) test	Runs before acceptance so environment faults are triaged at a glance
Capacity / performance / security	Separate nonfunctional stage (Ch9)	Needs baselines and dedicated environments; different cadence
Layout and visual design	A few UI tests plus humans	The worst return on automation investment
"Is this feature any good?"	Exploratory testing by people	Machines can't answer this — it's what testers are freed up to do

One judgement underlies all five tables: every minute of the acceptance stage is expensive, so buy only what genuinely requires the whole system to be wired together to prove. If the commit stage can settle it in milliseconds, keep it out; if a machine can't answer it, don't force one to.

Why it matters / in real systems

This chapter's claims are all default practice today, under other names: the driver layer grew into the industry-standard Page Object pattern; Given/When/Then grew into the Cucumber / SpecFlow / Behave family; Selenium's WebDriver interface became a W3C standard, with Playwright and Cypress travelling the same road; and "production-like environment plus smoke first" became the pre-production environment and post-deployment health check that ship as standard in every cloud pipeline.

What's most worth arguing about today, though, is the fight this chapter started: how many end-to-end acceptance tests should you have? Both sides left public, checkable numbers.

EVIDENCE FROM INDUSTRY · BOTH SIDES PUT NUMBERS ON THE TABLE

For · LMAX (the authors' own proving ground): co-author Dave Farley was Head of Software Development at the LMAX exchange, where this DSL-plus-layered-driver approach was forged. LMAX engineer Adrian Sutton published the scale in 2015: roughly 11,000 end-to-end acceptance tests running in about 50 minutes; after new CI hardware (four Dell FX2s chassis, 480 cores and 4.5 TiB RAM in total, running 61 exchange instances concurrently), about 11,500 tests in under 20 minutes. So "end-to-end suites are inevitably unaffordable" is not a law — the price is a DSL to absorb maintenance cost, and actually buying the machines. A. Sutton, "End to End Tests @ LMAX Update," Symphonious, 2015 ↗
Against · Google: in 2015 the Google Testing Blog ran a post titled, flatly, Just Say No to More End-to-End Tests, on the grounds that they are slow, flaky and very hard to trace to a root cause, recommending a mix of roughly 70% unit / 20% integration / 10% end-to-end. That is a head-on disagreement with this chapter's "acceptance tests are the main safety net." M. Wacker, "Just Say No to More End-to-End Tests," Google Testing Blog, 2015 ↗
Quantifying the brittleness · Google: in 2017 Google published statistics across its roughly 4.2 million tests — about 63,000 had a flaky run within a week (under 2%) — plus one very sharp finding: "overwhelmingly, the larger the test (as measured by binary size, RAM use, or number of libraries built), the more likely it is to be flaky." Acceptance tests sit at the "largest" end of that scale, which is this chapter's maintenance warning in numbers. J. Listfield, "Where do our flaky tests come from?," Google Testing Blog, 2017 ↗
"Production-like" in its modern form · Amazon / AWS: AWS has published the shape of its internal pipelines — the pre-production gamma environment is "as production-like as possible," with the same deployment configuration, the same monitoring and alarms, and the same continuous canary testing as production; a deployment starts as a one-box (one container out of ten, taking about 10% of gamma traffic) before going wide. What this chapter wrote in 2010 — deploy to a production-like environment with the same scripts, and confirm the environment is alive first — is the prototype of exactly that. C. Liguori, "Automating safe, hands-off deployments," Amazon Builders' Library, 2020 ↗

This is a frequent interview and design-review topic. Asked "how do you do end-to-end testing," the answer that scores isn't a list of tools, it's four facts: do you have a separate driver layer (how many tests change when the UI changes)? How long does a full run take, and at what parallelism? What share of failures are environmental, and how are they triaged apart from real defects? And what happens to the flaky ones — quarantine with a deadline, or unlimited retries?

Pitfalls and open arguments

The commonest way to die: writing acceptance tests as UI scripts. Control ids and page order get copied into hundreds of tests, one redesign turns the board red, the team patches for two weeks and then decides to "disable some of them" — from there the decline is irreversible. Survival is decided by layering, not by count.
Outsourcing acceptance tests to a separate QA team. Explicitly rejected here. If the people writing code aren't accountable for green, red becomes someone else's problem; the delivery team as a whole must own the suite, with testers moving to acceptance criteria and exploratory testing.
"BDD theatre." You adopted Cucumber, wrote piles of Given/When/Then, and the business never reads it — then that natural-language layer is glue developers wrote for themselves. The test is simple: in the last three months, did any non-developer actually read or edit those files? If not, a DSL in your programming language is the cheaper choice.
Curing flakiness with unlimited retries. Acceptance tests are inherently flakier than unit tests (Google's data explains why), so "auto-retry three times" becomes many teams' default — and it also masks genuine concurrency and timing defects. The real fix is still quarantine plus a deadline plus tracking the flake rate.
Open argument: how many end-to-end tests. Google says 10%, LMAX runs 11,000, and neither is being silly. Three variables separate them: whether a DSL and driver layer absorb the maintenance cost, whether you can afford the hardware, and whether the domain demands end-to-end correctness (a financial matching engine and a content site legitimately differ). The honest conclusion: without a DSL and layering, Google's advice is right for you; with them, LMAX's road becomes passable.
Dated in places. The book predates containers and cloud — production-like environments were painfully expensive in 2010, whereas today container orchestration spins one up in minutes, so several of the chapter's compromises can be relaxed. It also predates consumer-driven contract testing (e.g. Pact): under today's microservice architectures a substantial share of cross-service end-to-end acceptance tests can be replaced by contract tests, where each service independently verifies "the interface I promised hasn't changed" — far cheaper than standing up dozens of services to run one scenario. That is the most important methodological addition since this chapter.

Takeaways

① Acceptance tests verify that business value was delivered, not that the code is logically correct; they take the user's point of view against the whole deployed system.

② Nominally about testing, substantially about requirements: acceptance criteria must be defined before work starts, jointly by analysts, testers, developers and the customer. The biggest payoff lands before any test code is written.

③ The core craft is layering: acceptance criteria (business language) / test implementation (domain actions) / application driver (the only layer that knows UI and API detail). A UI change touches layer three only; the thousands above don't move.

④ Enter through the public API just below the UI: per-test cost drops from seconds to ~100ms and layout changes stop breaking things. If the UI holds logic, push the logic out of the UI.

⑤ Don't build on record-and-playback: it needs the app to exist first, welds UI detail into every script, and produces something nobody can read. It's only fit as legacy scaffolding.

⑥ The stage's shape: one artifact → deployed by production's own scripts into a production-like environment → smoke first → acceptance in parallel → green promotes; as a bonus your deployment process is rehearsed many times a day.

⑦ The commonest failure at this stage is environmental, not a business defect — the whole value of smoke tests is triaging the two apart at a glance.

⑧ State is the dirtiest part: prefer test isolation (own data, unique keys), because it is what unlocks parallelism — and parallelism is the only cure for slowness.

⑨ Handle asynchrony with polling plus timeout, never fixed sleep; make time controllable; stub external systems but keep honest books and back them with integration tests.

⑩ How much end-to-end testing is genuinely contested: Google recommends 70/20/10 and quantified "bigger means flakier," while LMAX squeezed 11,000 end-to-end tests into under 20 minutes. The dividing line is the DSL and driver layer — without them, this chapter's approach becomes the very disaster it warns about.