BOOK DEEP READ · CHAOS ENGINEERING · CH 8
Chaos Engineering: System Resiliency in Practice · Ch 8 · Capital One · 2020
You tap your card for a coffee and, a second or two later, the till beeps. Behind that beep is a bank's entire system. This chapter asks: how does a bank dare to break its own systems on purpose, for practice?
A bank is not a video site. When a video stalls, people grumble. When a bank fails, someone can't pay, and the regulator wants to know why. So "randomly switch off machines in production" sounds, inside a bank, like "cause an incident on purpose." Capital One's answer was not "then we won't." It was a slower road where every step leaves a paper trail.
Nobody puts a new driver on the motorway on day one. First the practice yard, then quiet streets, then the motorway — and at each stage an instructor signs off on what you practised and how it went.
A bank breaking its systems for practice follows the same order: get fluent in the test environment, write down the result of every drill, then use that record to earn the right to move closer to the real thing. The sign-off book is exactly what regulators and auditors want to see.
The bank's old method was the surprise inspection: another team quietly broke something, unannounced, to see whether the people on call would notice and fix it. That measures real skill, but it has two flaws. The tested team feels like it's sitting an exam — fail and you get blamed, so nobody volunteers their weak spots. And it happened only a few times a year while the system changed every week, so each result went stale fast.
Drills go from "a big exam once a year" to "a check-up on every release," so problems are stopped before launch — and once enough records pile up, you have the standing to ask for small, careful drills in the real environment.
One honest cost: a test environment is never the real one — no real crowds, no real peaks — and some failures only appear for real, so practising in testing is the starting line, not the finish.
In a regulated industry, chaos runs on a ledger, not on nerve: make the drill a routine check on every release in the test environment, let each drill leave its own record, and trade those records, step by step, for permission to get closer to production.
Want the mechanisms, numbers and diagrams? → switch to Deep
You'd think heavy regulation means "no chaos engineering." In fact what regulators want — evidence that you can survive failure — is exactly what chaos experiments are best at producing. Capital One's path has three steps: from blind resiliency testing run by an outside team, to chaos experiments whose hypotheses the application team writes itself, to experiments embedded in the CI/CD pipeline so every release runs them and records them automatically. On that path, "start outside production" is not a compromise — it is how you earn permission to enter production.
Chapter 8, the last entry in Part II ("the principles in practice"). The four companies before it each bent Ch3's five principles: Slack yields to a human in the loop, Google scales to company size, Microsoft solves prioritization, LinkedIn measures blast radius in people. This chapter is the most externally constrained bend: regulation pushes principle three, "run in production," to the very last step. Next, Part III turns to the human factors, and Ch16's "continuous verification" is where this chapter's "chaos in the pipeline" ends up. In the real world it speaks to banks, payments, insurance and healthcare — teams who must answer to someone outside when things break.
Start with the constraints. A bank outage is not just a UX problem: a card authorization has to come back within a second or two, and a failure means someone standing at a till unable to pay. Major disruptions must be reported to regulators, and every production change goes through change management with an approval record. When the book was written Capital One was moving its business wholesale onto the public cloud — its dependencies suddenly became more numerous and newer, and experience alone could no longer say what happens when a given component breaks.
Here Ch3's principle three, "run experiments in production," collides directly with policy: in an environment where every change needs approval, deliberately injecting failure looks like causing an incident on purpose.
What happens if you don't solve it? Two failure modes. One is paper drills: a disaster recovery failover once or twice a year, a checklist ticked, proof that "we have a plan" but not that "the plan works" — while the system changes weekly and each conclusion quietly expires. The other is going ahead without a mandate: one self-inflicted incident, and chaos engineering becomes a forbidden phrase at that company.
So the question is: under the constraint that every step must be explainable to outsiders, how do you get chaos engineering started and walk it, step by step, towards production?
Capital One's starting point wasn't chaos engineering but blind resiliency testing: a separate team injects failure into an application, and the tested team doesn't know the time or the form in advance. It tests three things — do the alerts fire, does on-call pick up, does the runbook actually work.
It has real value: in an announced drill everyone is already sitting in front of a screen, so you can't measure real detection latency; a blind test can. But its structural flaws are just as clear:
The pivot is who states the hypothesis. Chaos experiments are written by the application team itself: "When the cache tier is unavailable, the success rate of transaction lookups stays within its steady-state range — it just gets slower." Then they verify it in a controlled environment; if the hypothesis is refuted, that's a defect found before launch.
That step changes three things: ownership returns to the people who know the system best; experiments are repeatable rather than one-off ambushes; and every experiment naturally carries a four-part record — hypothesis, injection, observation, conclusion. That last point is exactly what a regulated industry needs most: structured evidence, not a report written after the fact.
The chapter's most actionable step: make chaos experiments a pipeline stage. After a new version deploys to a non-production environment, the pipeline reads a chaos configuration checked in alongside the code (which environment, what to inject, for how long, what counts as a pass), injects the failure automatically and checks the steady-state hypothesis; if it holds, the build is promoted; if not, it's stopped and sent back to development.
This fixes both of blind testing's flaws. Frequency: from a few times a year to every release — the system changes, and the verification changes with it. Evidence: a pipeline already records who ran each step, when, on which version, with what result; write the experiment result into the same record and the auditors' material needs no separate report.
The hidden cost should be said out loud: experiments on a gate must be stable and repeatable. Put a flaky experiment on the gate and within weeks the team will learn to route around it — which is worse than nothing, because the record still says "verified."
The chapter lists specific cautions for designing experiments in a regulated environment. Condensed into four rules:
Tooling. Build or buy? The chapter gives criteria, not an answer: does it cover your platforms and failures at several layers (infrastructure, network, application), can teams self-serve, does it have guardrails and a one-click abort, does it plug into the pipeline, and can it produce records an auditor will accept. In a regulated industry that last one is a hard requirement.
Team. At the start, a central team builds the platform, sets standards and accompanies the first few experiments; at scale, ownership of experiments must return to application teams — the central team can't do them all and doesn't know the systems as well. The central team's end state is platform and coach, not executor.
Evangelism. Technology decides whether it can be done; culture decides whether anyone does it. Describe the defects experiments find as "incidents stopped before launch," let people try it hands-on at game days, and share wins across the company again and again. The biggest obstacle to chaos engineering in a bank is not tooling but the instinctive fear carried by the words "break things on purpose."
First table: the ways a regulated team can start, what each tests, what evidence it leaves and what it costs.
| Approach | What it tests | Evidence | Risk | Cost / limits |
|---|---|---|---|---|
| Annual DR drill | Whether a big failover works end to end | Drill report, sign-off | Low (window, plan) | Too infrequent; results expire as the system changes |
| Blind test | Real detection and response latency | One-off grade | Medium | Feels like an exam; teams resist; hand-run by a central team |
| Non-prod hypothesis experiment | Fallback, timeout, self-healing logic | Structured experiment record | Zero customer impact | Unreal load and data; scale failures invisible |
| Automated chaos in the pipeline | Whether resilience regresses per version | One per version | Zero customer impact | Experiments must be stable, or the gate is bypassed |
| Controlled production experiment | Behavior under real load | Production evidence | Real; needs blast-radius limits and auto-abort | Only on the table once earlier rungs vouch for it |
The two non-production rows aren't "afraid of production"; they bank evidence at zero customer risk — the only currency that buys the last row in a regulated industry.
Second table: how to structure the team.
| Structure | Upside | Cost | When |
|---|---|---|---|
| Central team does it all | Uniform standards, fast start | Becomes a bottleneck; app teams don't learn | The first few experiments |
| Each app team on its own | Knows the system best, scales | Inconsistent standards, messy evidence formats | Once the platform matures |
| Central platform + team-owned experiments | Shared platform, guardrails and logging; experiments owned by those who know best | Needs ongoing evangelism and training | End state for most regulated organizations |
Concrete choices:
Three sentences that can change a review outcome. One: the regulator is not chaos engineering's opponent but a buyer of its evidence. Two: the value of starting outside production is not safety but banking permission. Three: put experiments in the pipeline and the evidence writes itself.
Asked in an interview "how do you do chaos engineering in a regulated industry?": "we can't touch production, so we just do DR drills" is one level. "Start with hypothesis-driven experiments outside production, put the stable ones on a CI/CD gate with results flowing into the audit record, then trade those records for small-blast-radius production experiments" is another — it shows you know the hard part here is permission, not technology.
Industry evidence
1. Core claim: regulators want evidence that you can survive failure, and chaos experiments naturally produce it — regulation isn't a reason not to do chaos, it's a reason to take a different road.
2. Constraint: bank outages must be reported and changes approved, so "experiments in production" looks like causing an incident on purpose; principle three moves to the last step.
3. Starting point, blind testing: an outside team ambushes and measures real detection and response — but it's an exam, one-off, and the knowledge doesn't stick.
4. The pivot: the application team writes its own hypothesis; experiments are repeatable and naturally carry a structured hypothesis–injection–observation–verdict record.
5. The key step: make experiments a CI/CD gate with config checked in alongside the code; frequency goes from a few times a year to every release, and evidence becomes a by-product of the pipeline.
6. Cautions: observability before injection, warn operations and security in advance, write abort and rollback first, grow blast radius slowly and avoid money and customer data.
7. Organization: a central team builds the platform and coaches, ending as platform + coach while experiments belong to app teams; evangelism frames defects as "incidents stopped."
8. Honest limits: non-production can't see scale failures and is not the destination; gate experiments must be stable, or a "verified" record is more dangerous than none.