Chaos Engineering Ch 15 · The Chaos Maturity Model

Chaos Engineering: System Resiliency in Practice · Ch 15 · The Chaos Maturity Model · 2020

What is this chapter about?

Many of the companies behind your streaming, ride-hailing and food-delivery apps deliberately break their own systems a little to find hidden weaknesses. But "how good are we at this?" is a hard question — one person says "we've been doing it for a year", another says "we haven't really started". This chapter offers a map with two axes so a team can say where it stands and where to go next.

An analogy

Picture a gym assessing its members. "How long have you trained?" is not enough. You want two answers: how skilled is the training (jumping around randomly, or following a plan with a coach recording progress), and how widespread is the habit (one visit now and then, or the whole family every week, where skipping needs an excuse). Perfect form once a year does nothing; going daily but flailing does not build strength either.

Why the old way was hard

One number. "We ran 50 drills this year" sounds impressive — yet it might be one small group switching off machines in a test environment.
No idea what to fix next. Some companies spend heavily on a brilliant platform that only one team uses; others have everyone doing drills, but only the shallowest kind. Both are stuck, and the cures are completely different.

The core intuition: two axes, four levels each

Axis one: how skilled. At the bottom you unplug a machine by hand in a test setup and check whether the computers look busy. Moving up, you run in the real environment, the system starts and stops the drill on its own, and you watch what actually matters — can customers still check out? At the top you can even estimate how much money a failure would cost.
Axis two: how widespread. First a few enthusiasts do it quietly; then the company blesses it and lends a few people; later there is a dedicated team and every important service drills regularly; at the top it is simply habit — taught to new hires on day one, and opting out needs an explanation.
Put yourself on the map. Crossing the two axes gives a grid. Find your square, and the next step is to move the lagging axis forward one notch.

What it buys you / how to choose

It turns a vague "are we any good?" debate into a picture people can point at: a strong platform nobody uses means go spread it; everyone doing shallow drills means invest in tooling and safety. One honest limit: the map tells you where you are and which way to go, not what each step costs or whether it is worth it.

One line to remember

Don't ask "do we do it?" — ask "how skilled?" and "how widespread?" Find your square on the grid, then strengthen the weaker leg.

Want the concrete criteria for each level, the cure for four typical positions, and how Google and Netflix climbed the grid? → Switch to Deep

In one sentence

You think maturity is a one-way ladder; it is actually a two-dimensional map. Sophistication measures how effective and safe the experiments are; adoption measures how many systems and people they reach. The chapter's use is not scoring but locating — see which square you are in, then decide which axis gets the next unit of investment.

Glossary

Chaos experiment: injecting a failure (crash, latency, network loss) into a system within controlled bounds, to see whether it still behaves normally.
Steady state: a measurable signal of normal behaviour, such as orders per second, used to tell whether anything went wrong.
Business vs system metrics: the former is what users feel (checkout success, playback starts); the latter is machine health (CPU, memory).
Control / experiment groups: two small slices of real traffic; only the experiment group gets the fault, so the comparison shows whether a difference was caused by it.
Blast radius: the maximum number of users or machines one experiment can affect.
Continuous delivery (CD): a pipeline that automatically builds, tests and deploys every code change.
Shadow / replay traffic: copying or recording production requests and sending them to a non-production environment, to test with realistic load.
Game Day: a scheduled, hands-on failure drill involving several teams.
Maturity model: a framework that splits a capability into levels, used to assess where you are and plan improvement.

Where it sits

Chapter 15 closes Part IV, "Business Factors". The model first appeared in the Netflix chaos team's 2017 O'Reilly report Chaos Engineering, its name borrowed from software engineering's Capability Maturity Model (CMM). Ch13 (ROI) and Ch14 (open sharing) ask "is it worth it?" and "how does it spread?"; this chapter asks "where are we now, and what next?" Ch16, Continuous Verification, is the top of the sophistication axis.

The problem this chapter solves

Two or three years into a chaos program, managers and engineers alike ask: how are we doing? The usual answer is a count — "120 experiments this year." That number hides two very different situations:

All 120 came from one enthusiastic team, manually killing machines in staging and watching CPU. The organisation has barely changed.
The 120 are spread across 40 services, but each one only dares to kill a single machine, nobody watches business metrics, and there is no automatic abort — so anything that goes wrong becomes a real incident.

Same count, opposite problems and cures: the first lacks adoption, the second lacks tooling and guardrails. Without a shared coordinate system, planning collapses into "do a bit more", and money goes the wrong way — more features for a platform nobody uses, or more teams running unsafe experiments.

The core ideas, in depth

1. Why two axes instead of one ladder

The classic CMM is a five-step ladder. The Chaos Maturity Model deliberately splits into two, because the value of chaos engineering is the product of both: an elegant experiment covering one service leaves the organisation no more resilient; broad coverage of experiments that can only prove "losing one machine is fine" teaches little. Crucially, the two axes are driven by different investments: sophistication mostly by engineering (platform, automation, measurement), adoption mostly by organisation (sanction, headcount, training, defaults). Only by separating them can you tell whether the money belongs to the tooling team or to rollout.

Four levels per axis, sixteen squares. Plot yourself first, then see which axis lags; the two purple corners are the most common imbalances (labels are this page's reading).

2. The sophistication axis: how "scientific" the experiments are

Sophistication asks how reliably experiments find real problems, and how safely they avoid causing real incidents. The book describes four levels, which climb along four directions at once: where it runs, who starts and stops it, which metrics it watches, what it injects.

Four sophistication levels: each step brings experiments closer to real production, more automated, and closer to business outcomes; grey text is the kind of fault injected.

Where it runs: from a test environment, to realistic traffic, to production, and finally every stage of development. The closer to production, the more real the findings — and the higher the risk, so the next item must climb in step.
Who starts and stops it: from hands-on operation, to automated execution with a human ready to abort, to automated analysis, to automated abort. Automatic abort is the safety precondition for running in production.
Which metrics: from CPU and memory, to aggregate business metrics, to experiment-versus-control business metrics, and finally to projecting how much revenue a failure would cost.
What it injects: from killing a machine, to network latency, to service-level errors and combinations of failures, up to shifting usage patterns and mutated responses.

3. The adoption axis: how many people and systems it reaches

Adoption asks how legitimate and widespread chaos engineering is inside the organisation. Roughly:

In the shadows: a few early adopters work unsanctioned, cover few systems, and the organisation barely knows.
Investment: officially sanctioned, part-time people assigned, several teams interested, a few critical services experiment now and then.
Adoption: a dedicated team; incident response feeds the framework, turning each incident into a regression experiment so it cannot quietly return; most critical services experiment regularly, with occasional verification of incident response and Game Days.
Cultural expectation: every critical service experiments frequently, most non-critical ones too; it is part of engineer onboarding; participation is the default, and opting out needs a justification.

The last criterion is the one to remember: flipping from opt-in to opt-out. As long as every team must be persuaded to join, adoption has a ceiling; once joining is the default and abstaining needs explaining, coverage is set by rules rather than enthusiasm.

4. Using the map: locate, then fix the weaker leg

The method is plain: assign a level on each axis, plot the square, and see which axis lags. It is a diagnostic, not a ranking — two teams in different squares are not better or worse, they simply need different next steps. Re-plot over time to see whether the trajectory heads up and to the right, or keeps stretching along one axis only.

Key trade-offs and choices

There is one core trade-off: does the next unit of resources go to sophistication or to adoption? Both imbalances waste money, just in different ways.

Four typical positions: symptoms and cures (scenarios illustrative)

Position	Typical symptom	Next step	Cost of not doing it
Bottom-left: shadows × elementary	One or two people kill machines by hand in staging	Win official sanction; deliver one result with business metrics on 1–2 critical services	No result, and the effort leaves with its champion
Bottom-right: gilded island	Platform auto-splits traffic and auto-aborts, but a handful of 200 services use it	Push adoption: cut onboarding cost, fund dedicated rollout, turn postmortems into regression experiments	Platform costs stay; organisational resilience barely moves
Top-left: wide but shallow	Every team runs Game Days, but only kills one machine and eyeballs dashboards	Push sophistication: build auto-abort and business metrics first, then widen fault types	More experiments, more real incidents — until someone shuts the program down
Top-right: high on both	Experiments run in the pipeline by default; results project losses	Sustain: control noise and cost, size experiment frequency by service criticality	The automation decays into a green light nobody reads

Which axis to push first

Situation	Push first	Why / cost
Never run in production	Sophistication (auto-abort, control groups)	Widening adoption without guardrails widens the incident surface; rollout waits for tooling
Tooling can safely run in prod	Adoption	Marginal value is in coverage; needs organisational resources the tooling team cannot supply alone
High adoption, nobody reads results	Sophistication (business metrics, auto analysis)	Shallow conclusions are not actionable; metrics must be aligned with business teams
Need budget from leadership	Top of sophistication (loss projection)	Translating experiments into money is what connects to Ch13's ROI case

Rule of thumb: make experiments safe, then widespread, then automatic. Guardrails are the ticket to wider adoption; adoption is where platform investment pays back.

Why it matters / in real systems

Asked in an interview or architecture review "how far along is your chaos engineering?", the best answer is not a count but two coordinates: "Sophistication is at sophisticated — production runs, control groups compared on business metrics, but aborts are still manual. Adoption is at adoption — a dedicated team, critical services drill regularly, non-critical ones not yet. So next come automatic abort and onboarding." One sentence states the present, the gap and the plan.

Real organisations' trajectories also alternate between the two axes: someone experiments at the edges, it gets sanctioned, a platform appears, and finally it lives in the pipeline as the default.

Industry evidence

Google's DiRT: from a few narrow questions to company-wide coordinated drills. A Google Cloud blog post recounts that DiRT began as "a small set of theoretical tests ... isolated and very narrow in scope", and is now "a coordinated set of events organized across the company", growing to disconnecting entire data centers. Both axes moved. Google Cloud Blog · Jesus Climent · 2019 (Shrinking the time to mitigate production incidents)
Netflix wired injection into continuous delivery. The Chaos Monkey repository states: "This version of Chaos Monkey is fully integrated with Spinnaker, the continuous delivery platform that we use at Netflix." — the "wired into CD" rung of the sophistication axis. Netflix · Chaos Monkey README (github.com/Netflix/chaosmonkey)
Open-source tools ship "in the pipeline" and "steady-state checks" as features. One LitmusChaos use case is to "run chaos as a pipeline stage", with probes declaring steady-state validation; Chaos Mesh's Schedule / Workflow / StatusCheck resources support "recurring experiments". The middle-upper sophistication rungs can now be bought or installed. LitmusChaos README (github.com/litmuschaos/litmus) · Chaos Mesh README (github.com/chaos-mesh/chaos-mesh)

That last item doubles as a warning: tools can lift sophistication overnight, but adoption cannot be bought — which is exactly how gilded islands usually form.

Pitfalls and debates

"Higher is always better — race to the top-right." The model is a map, not an exam. For a 10-person company, projecting revenue loss or plugging into an A/B platform may simply not pay.
"Adoption = number of experiments." Adoption is about coverage, legitimacy and defaults, not counts; one team running 100 experiments is still in the shadows.
"Push one axis and you're done." Sophistication alone yields an island; adoption alone yields more incident risk. Alternate the legs.
Level assignment is subjective. The book gives descriptive traits, not measurable thresholds; two people may rate the same team differently. Great for conversation, dangerous for performance reviews.
The generic maturity-model problem. CMM-style models are often criticised for encouraging levelling-up for its own sake and treating the checklist as the goal. This one can also become a spreadsheet game, losing sight of "find and fix weaknesses".
Of its time. The model comes from Netflix's 2017 context, assuming large-scale microservices and an in-house platform. With managed fault-injection services and open-source platforms now common, the mid-sophistication bar is much lower, and the hard part has largely shifted to adoption.

Takeaways

① Chaos maturity is not a ladder but a sophistication × adoption map.

② Value is the product of both: elegant but unused, or widespread but shallow, and resilience stays flat.

③ Sophistication climbs along four directions: where it runs, who starts/stops it, which metrics, what is injected.

④ Auto-abort is the safety precondition for production; business metrics plus control groups make conclusions credible.

⑤ Adoption's four levels: in the shadows → investment → adoption → cultural expectation.

⑥ The top level's hallmark: opt-in flips to opt-out, and it is part of onboarding.

⑦ Two classic imbalances: the gilded island (push rollout) and wide-but-shallow (push guardrails and tooling).

⑧ Ordering rule: safe, then widespread, then automatic; tooling can be bought, adoption cannot.

⑨ Report status as two coordinates, not one experiment count.