Continuous Delivery Ch 9 · Testing Non-Functional Requirements

What is this chapter about?

The moment concert tickets go on sale, the app spins and then tells you the system is busy. The features are all there — the button works, the order flow works — it simply cannot take that many people at once. Chapter 9 of Continuous Delivery is about that family of requirements: fast enough, sturdy enough, not trivially hackable. And about turning them into something a machine checks every day, rather than something you discover on launch day.

An analogy first

A building has two kinds of requirements. One kind is how many rooms there are and whether the kitchen works — you can see at a glance whether it was done. The other kind is how many people the floor can hold, how well it blocks sound, how long it survives a fire. The second kind cannot be added during decoration; it was settled the day the foundation was poured. And to verify it you have to actually put a hundred people on that floor — reading the blueprint is not enough. This chapter is about verifying software's second kind.

Why the old world struggled

The hard part is not the testing. It is that these requirements are usually written as a sentence nobody can judge: the system should be fast, it should support a lot of users. With no way to say whether you met it, the work slides to the very end — a few people clicking around the week before launch, or simply letting real users find out. By the time it fails, the cause usually sits in the deepest structure, and fixing it means redoing the foundation.

The core idea

The chapter's method is two plain steps.

One: pin the sentence down. "Should be fast" becomes "with a thousand people ordering at once, ninety-five out of a hundred see a result within 0.8 seconds, and it holds for a full hour". Now it is not a wish; it is a question with an answer.

Two: take the scripts you already have and run thousands of copies. The previous chapter already produced scripts where a machine walks through the ordering flow by itself. Load testing needs no new suite: run the same scripts in parallel, by the thousand, and watch where the system starts to crawl and where it falls over. The machine runs it on a schedule and turns the light red when it fails — no different in kind from any other test.

The biggest trap

The rehearsal room is not the venue. Few companies pay for a second full set of machines just for load tests, so a team measures one small box and multiplies by thirty. But the small box holds a hundred thousand rows where production holds a billion; on the small box everything sits in cache, in production it has to go to disk. That multiplication is almost always wrong. The book's stance is refreshingly plain: the closer the environment is to production, the more the numbers mean. When it is not close, use it to ask "did we get slower than last week" — never to promise "here is what we can take".

The honest cost: these tests are slow and expensive — tens of minutes per round, and a whole set of machines — so they must not sit in the path of every single commit.

Remember this

Turn "fast enough" and "sturdy enough" into a sentence with numbers that can be judged right or wrong, reuse your existing test scripts to make it a pipeline stage that can go red — and remember that a shrunken environment shows you trends, never promises.

Want the capacity-knee curve, the injector architecture and the trade-off tables? → Switch to the deep read

In one sentence

You think performance testing is a dress rehearsal before launch; Chapter 9 says it should be a stage in your pipeline that can turn red. The claim: turn capacity, performance and security — the "-ilities" — from adjectives into acceptance criteria with thresholds a machine can judge, built by reusing the acceptance tests you already have. The authors go further and reject the name: these are not optional extras but cross-functional requirements that cut across every feature.

Glossary

Cross-functional requirements (CFRs): the book's replacement for "non-functional requirements" — performance, capacity, security, availability. Renamed because they are not un-functional; they constrain every single feature.
Throughput: completed requests per unit of time, e.g. 5,000 per second (TPS / QPS).
Response time: how long one request takes end to end; the server-side portion is often called latency.
Capacity: the maximum throughput the system sustains while response time still meets its constraint — not "the most it can run", but "the most it can run while still usable".
Scalability: whether adding machines buys a proportional increase in throughput.
Load / stress / soak test: respectively "do we meet the target at expected load", "where and how does it break past that", and "does running for hours leak memory or connections".
Percentile: line every request up by duration; p95 is the 95th fastest — 95% of requests were quicker.
Injector (load agent): a machine whose only job is generating requests; one controller coordinates them and aggregates results.
Stub: a stand-in service replacing a third party you cannot hammer for real, such as a payment gateway.
Static analysis / penetration testing: the first scans code for dangerous patterns without running it; the second has someone attack the running system for real.

Where it sits

The chapter belongs to Part II, "The Deployment Pipeline", between Ch8 (Automated Acceptance Testing) and Ch10 (Deploying and Releasing). Ch8 automates "is it correct"; this chapter automates "is it fast enough, sturdy enough, safe enough" — using the same scripts. It builds on the quadrant from Ch4's testing strategy: technology-facing tests that critique the product. The real-world problem it names: why the "performance testing stage" everyone has on their pipeline diagram is the one everybody skips.

What the chapter is solving

These requirements die in two opposite ways.

One is never testing them. Written as "the system should be fast", they carry no criterion, so they queue forever behind features and get tested by real users. The other is over-engineering: sharding, queues and three cache layers for an imagined hundredfold load, trading delivery speed for a peak that never arrives. The book takes the middle path: cross-functional requirements should be analysed and prioritised at inception alongside functional ones, then nailed down by tests.

The cost of skipping is finding out too late. Concretely: an e-commerce site with 20M registered users, 2M daily actives and ~30 requests each averages ~700 QPS, with an 8× promotional peak near 5,600 QPS. It buckles in production, and the cause turns out to be 41 database queries per product page (one list query plus 40 per-item lookups). No parameter tweak fixes that — it is how the code reaches data, and that shape is usually set in week one. Capacity problems are almost always architectural, and the later you find them, the wider the blast radius of the fix.

The core material

1. First, write the requirement so it can be judged

The most actionable rule in the chapter: every cross-functional requirement needs scenario + measure + threshold. "The system should be fast" becomes: with 1,000 concurrent users against a 5M-item catalogue, the checkout endpoint holds p95 < 800ms with an error rate under 0.1%, sustained for 60 minutes. Drop any of the three and it stops working: without a scenario the number is meaningless, without a threshold no machine can call it red or green.

One criterion people skip: the threshold comes from the business, not from "what we measured last time". Otherwise it degrades with the system — every failure raises the bar — and the gate becomes a rubber stamp.

2. Capacity, throughput, response time: three words used interchangeably and wrongly

This is the most valuable page of concepts in the chapter. The three are related by one curve: as concurrency rises, throughput grows roughly linearly until some resource — CPU, connection pool, a lock — caps it; then it flattens and may even fall. Response time turns sharply upward at that same point. Capacity is defined at that knee: the largest throughput you can absorb while response time stays under its threshold.

Throughput (blue) flattening and response time (violet) turning upward happen at the same point; capacity is the maximum throughput while response time still meets its threshold — not "the most it can run".

Why the distinction matters: past the knee, extra requests do not become throughput, they become queue. A system sitting to the right of it looks busy and productive — CPU pinned, throughput intact — while every user waits. That is exactly how "the service is up but nobody can use it" happens.

3. Where capacity problems come from

The book points at a handful of habits in code and architecture. Each is worth a line on a review checklist:

Chatty remoting: using network objects the way you use local ones. For scale: an in-process call costs nanoseconds, a same-datacenter RPC about 0.5–1ms, a cross-continent round trip about 150ms — six orders of magnitude apart. One remote call hidden inside a loop is a disaster.
Database round trips: the 41 queries above are a textbook N+1. Fifty rows at 0.5ms each is 25ms of pure waiting before pool contention.
Locks and serialisation: the physical reason the throughput curve flattens is frequently one global lock — and adding machines does nothing for it.
Session state: keeping user state in application-server memory grows the heap and forces sticky sessions, which kills horizontal scaling.
Allocation and garbage collection: high allocation rates push pauses into the tail of your response times. The average hides it; p99 shows it immediately.

4. Building the tests: reuse acceptance tests, record at the service layer

The cheapest move in the chapter: do not write a second suite — scale up the acceptance tests from Ch8. Those already express how a real user drives the system; load testing only adds "ten thousand of them at once".

With one crucial constraint: record at the service / API layer, not at the UI. Two reasons — UI scripts shatter when a button moves, and the maintenance eats the entire benefit; and browsers are heavy, so a UI-driven injector becomes the bottleneck before the system under test does. The rule still holds today, under a newer name: protocol-level load testing.

Shape of a capacity stage: one controller drives N injectors, which replay service-layer interaction templates against the environment under test; monitoring collects resource metrics and third-party dependencies are stubbed.

5. Environment distortion — the section most often skimmed, and the one to remember

The book concedes reality: very few teams can afford an environment identical to production. But it warns that numbers from a scaled-down one cannot be extrapolated linearly. The distortions come from several places, and every one of them biases you toward optimism:

Data volume: a 100k-row table sits in memory; a billion rows means really walking the index — a B-tree level or two deeper, a disk round trip or two more per query.
Cache hit rate: near 100% on a small dataset, perhaps 80% in production — and that missing 20% lands on the database amplified several times over.
Topology: a "call" on one box is a cross-rack, cross-availability-zone round trip in production.
Shared resources: in test you own the database; in production a dozen services and a pile of batch jobs eat it with you.
Stubs: stubs answer in milliseconds and never time out — whereas a wobbling real dependency is a classic start of a production incident.

The pragmatic conclusion: a scaled-down environment is a regression detector, not a capacity predictor. It is good at "did we get slower than last time" and bad at "how many QPS can we take". Only a production-like environment, or production itself, answers the second.

6. Security is a cross-functional requirement too

The chapter puts security beside capacity for the same reason: both get postponed indefinitely unless they become automated gates. In pipeline terms that is three automatable things — static analysis (run on every commit, scanning for injection patterns, hard-coded secrets and the like), dependency scanning (libraries with known vulnerabilities), and penetration testing (mostly manual, done periodically). The criterion matches capacity testing: if it can produce an unambiguous red or green, make it a gate; if it can only produce a report, do not pretend it is one.

Trade-offs and choices

First choice: which environment you test in. It decides what your numbers are allowed to promise.

Option	Answers	Cost and risk	Fits
Production-like	Absolute capacity: how many QPS, and where it breaks	Near-double the hardware bill; data must be anonymised and refreshed	Finance, pre-peak-season, externally committed SLAs
Scaled-down	Relative trend: did we get slower	Cheap, but the numbers do not extrapolate; treating them as a promise ends badly	Everyday regression — most teams' default
In production (shadow / diverted traffic)	The truth: real data, real dependencies, real jitter	Can hurt real users; writes need isolating; demands solid circuit breakers and rollback	Teams with mature observability and automated guardrails

Written in 2010, the book assumes row two. Row three was considered unacceptable then and is mainstream at large companies now — see the next section.

Second choice: how you generate load.

Approach	Strength	Cost
UI record & replay	Close to the real user journey, quick to start	Extremely brittle; browsers are heavy, so the injector saturates first
Service / API scripts	The chapter's recommendation: stable, light, reuses acceptance tests	Bypasses the front end, so client rendering and asset loading go untested
Production traffic replay	The most realistic load shape, long tail included	Needs a capture and anonymisation pipeline; replaying writes needs isolation

Third choice: which test. Four kinds catch different bugs and should not be conflated. Load tests ask "do we meet the target at expected peak". Stress tests ask "past that, where and how do we break — graceful degradation or avalanche". Soak tests (8–24 hours) catch memory leaks, connection leaks and logs filling the disk. Spike tests catch cold caches and the lag in autoscaling. Running only the first is the most common shortcut — and the other three describe how systems actually go down.

Why it matters / in real systems

The chapter's argument is now the default, only with different vocabulary: we say SLO instead of "threshold", "performance testing as code" instead of "reuse the acceptance tests", "protocol-level load testing" instead of "record at the service layer". JMeter, which the chapter names, is still maintained at the Apache Foundation — and its successors have written "belongs in the pipeline" straight into their product positioning.

Evidence from the field

Apache JMeter — the tool this chapter names, still active — describes itself as "an Open Source Java application designed to measure performance and load test applications", able to "simulate a heavy load on a server, group of servers, network or object to test its strength or to analyze overall performance under different load types".Source: Apache JMeter README · github.com/apache/jmeter
Twitter's Iago pushed "replay real interactions" to its conclusion: "Iago is a load generation tool that replays production or synthetic traffic against a given target", where the source is "a transaction log … typically from logs, such as web server logs, proxy server logs, or network sniffer captures". (The repository was archived in 2021, but the approach outlived it.)Source: Twitter Iago README · github.com/twitter-archive/iago
Grafana's k6 makes this chapter's conclusion its headline feature: "Tests as code. Reuse scripts, modularize logic, version control, and integrate tests with your CI." — precisely what the chapter asked for in 2010.Source: Grafana k6 README · github.com/grafana/k6

One more piece of evidence worth carrying, about which number to use: in the Dynamo paper (SOSP 2007) Amazon argued that internal service SLAs should be stated at the p99.9 percentile rather than the mean — because an average hides the slowest cohort of users entirely, and that cohort is often the one with the most data and the largest orders.

Pitfalls and disputes

Using mean response time as the threshold. A 200ms average happily coexists with 5% of users waiting three seconds. Thresholds belong at p95 / p99, and always paired with error rate — otherwise "fails fast" reads as "performs beautifully".
Multiplying scaled-down numbers into a capacity figure. All five distortions lean optimistic; the error is usually not 20% but several fold.
Putting capacity tests in the commit stage. A round takes tens of minutes and blows straight through Ch7's ten-minute budget, after which everyone starts skipping it. It belongs after acceptance testing, as its own stage, triggered on demand or on a schedule.
"It passed the load test, so it is safe to ship." The load shape came from history; what actually takes systems down is the shape nobody modelled — retry storms, caches expiring together, everything reconnecting at once after an upstream timeout. That is an inherent limit of the method, not sloppy execution.
A limit of its time. In 2010 the premise was a fixed-size datacenter, so capacity was a physical number. Cloud elasticity turned it into a cost question but did not abolish it: autoscaling lags by tens of seconds to minutes, and stateful data layers barely scale at all — which makes spike testing more important now, not less.
One claim later overturned. The chapter assumes load testing happens in a production-like environment, implying it must not happen in production. A decade later the mainstream practice is the opposite: injecting load and failure into production and verifying under real conditions is standard for mature teams. The method is not wrong — its boundary is narrower than the authors assumed.

Takeaways

1. The claim: make capacity, performance and security into gates with thresholds that go red automatically, not a one-off rehearsal before launch.

2. The authors rename them cross-functional requirements: they cut across every feature and belong in the backlog from inception, prioritised alongside functionality.

3. Write each one as scenario + measure + threshold; the threshold comes from the business and must not be raised just because the test failed.

4. Capacity = maximum throughput while response time is still within threshold; throughput flattening and latency taking off happen at the same knee.

5. Usual sources of capacity problems: chatty remoting, N+1 queries, global locks, session state, GC pauses — all architectural, all more expensive the later you find them.

6. The cheapest method: scale up the acceptance test scripts, and record at the service / API layer rather than the UI.

7. A scaled-down environment is a regression detector, not a capacity predictor: data volume, cache hit rate, shared resources and stubs all bias it optimistic.

8. Four test types catch different bugs — load, stress, soak, spike — and running only load tests is the common shortcut.

9. Use p95 / p99 with error rate; averages hide the slowest cohort (Amazon's Dynamo paper set SLAs at p99.9).

10. Today's descendants: SLOs, performance testing as code (k6), production traffic replay (Iago). The single claim overturned is "never test in production".