Chaos Engineering Ch 6 · Microsoft: Variation and Prioritization of Experiments

What is this chapter about?

The last two chapters were about how to stage a drill — Slack scripting one for a legacy system, Google scaling one up to the whole company. This chapter steps back and asks something more basic: which failure should you break first?

Any moderately complex production system has more ways to break than you can count, and you only get a few dozen good experiments a year. Where should those few dozen go?

An analogy: which test does a check-up start with?

A hospital can run a thousand different tests. You have one morning.

The smart move is not to pick at random, and not to use the most expensive machine on every organ. It is to ask two things: how likely is this problem at your age and lifestyle, and how bad is it if you have it. Heart and blood pressure go first; tropical parasites go last — not because parasites are harmless, but because the odds on you are tiny. Chaos experiments get ranked by the same ruler.

Why the old way is hard

Many teams start chaos work by installing a tool, then let the tool's feature list become the experiment list: it can kill machines at random, so they keep killing machines.

Two hundred kills, nothing broke. That is reassuring — but it may only mean you already knew nothing would break. The handful of things that could actually take the system down were never touched.

The other half is more awkward: a complete list is impossible to write. There are twenty or thirty dependencies, each can go wrong in several distinct ways, and multiplying gives thousands.

The core intuition: three things

Start from dependencies, not from what the tool can do. Write down everything a single request touches — database, downstream services, cache, plus the easy-to-forget ones: name resolution, certificates, config rollout, time. The test is simple: if it breaks and your request gets slower or fails, it belongs on the list — whether or not anyone drew it on the architecture diagram.
"Slow" matters more than "dead." When a service dies outright, your code usually copes, because it gets an error immediately. The nasty case is a service that is still alive but went from 0.05 seconds to 3 seconds — every waiting slot in the caller fills up, and requests that have nothing to do with it jam too. There is a worse rung still: it answers normally, with wrong data, and no alarm fires at all.
Rank by "how likely × how much it hurts," not by "how interesting." For a failure that has already happened to you, the probability is measured, not guessed. And one counter-intuitive rule: the thing most worth breaking first is often the dependency everyone calls unimportant — "unimportant" is a claim nobody ever verified, and if it is wrong the whole chain goes down with it.

What it buys you / how to choose

You spend a limited number of shots where they can actually teach you something. One more rule saves a lot of effort: do not run an experiment whose answer you already know. If you are sure it will break, do not break it — go fix it; breaking it is just scheduling your own incident. One honest cost: the ranking rests on estimates of probability and impact, and estimates are wrong, so the failures you ranked last will still come for you.

One line to remember

There are infinitely many ways to break and only a few dozen experiments — so the real craft of chaos engineering is not injection but ordering: start from the dependency list, favour the "slow" and "wrong data" modes, rank by likelihood × pain, and skip the ones whose answer you already know.

Want the mechanism, the numbers and the diagrams? → Switch to the deep read

In one line

You think the barrier to chaos engineering is injection capability — can you kill a process, cut a network link, add latency. This chapter moves the barrier: injection was never the bottleneck; selection is. The fault space explodes combinatorially with dependencies times failure modes, while the experiment budget is bounded on three sides at once — people, risk and time windows. So the real engineering problem is variation and prioritization: start from a complete dependency inventory, spread variation along the failure-mode gradient (dead → slow → wrong data), rank by probability × impact ÷ cost, and never run an experiment whose answer you already know — the product of an experiment is removed uncertainty, not a staged incident.

Glossary

Chaos experiment: injecting a failure that could realistically occur, around a falsifiable hypothesis, to see whether the hypothesis holds.
Steady-state hypothesis: "the system is fine right now" expressed as a business metric (say, checkout success ≥ 99.9%), compared before and after.
Dependency: anything on the request path where "if it breaks, you get slower or fail" — services, middleware, DNS, certificates, config rollout all count.
Implicit dependency: a dependency that is genuinely on the path but was never drawn on the diagram. The biggest source of gaps in this chapter.
Failure mode: the distinct ways one dependency can go wrong: connection refused, timeout, degradation, wrong data.
Degradation: not dead, just slower or only serving part of the traffic. This chapter argues it is harder to handle than an outright outage.
Variation: spreading one injection point along mode, load, region and failure ratio, so you get a family of experiments instead of one.
Blast radius: the worst-case span of users and systems an experiment can touch (Ch3, principle five).
Pool saturation: the caller's connection or thread slots for waiting on a downstream all fill up, so unrelated requests start failing too.

Where it sits

Chapter 6, the third piece in Part II on principles in practice, written from Microsoft's vantage point. It divides labour with the previous two as "how to run one" versus "which one to run": Ch4 has Slack script a drill for a legacy system, Ch5 has Google scale drills to company size, and this one steps back to ask where the budget should go. Ahead, Ch7 (LinkedIn) governs the ethical ceiling on experiments, and Ch12 uses lineage-driven fault injection (LDFI) to promote this chapter's human judgement into an algorithm. In the real world it is aimed at teams whose tooling is already in place but whose experiment list contains exactly one item: kill a random instance.

What this chapter is solving

Start with the scale. For a mid-sized production service, the dependencies on a single request path run to a dozen explicit ones (database, cache, auth, a few downstream RPCs, a queue, object storage) plus a dozen implicit ones (DNS, TLS certificates, service discovery, config and feature-flag rollout, clocks, the metrics backend, the image registry). Call it twenty-five, each with at least 4 failure modes, and single-dependency single-mode candidates already number in the hundreds; allow two dependencies to fail together and it is tens of thousands; layer on load level, region and failure ratio and it keeps climbing.

The budget, meanwhile, is bounded on three sides: people (a decent experiment needs a hypothesis, metrics and an abort plan — Ch4 showed how heavy that is), risk (every production injection can turn into a real incident) and time windows (off-peak, humans in the loop). A team doing this seriously runs experiments well in the dozens per year, not the thousands.

What happens if you skip the problem? Three typical degenerations:

A tool-driven list. The tool kills instances, so the list is kill-instances. Two hundred green runs, while "downstream got slow" — the thing that will actually take you down — was never tried.
An interest-driven list. You pick the technically exciting ones (clock skew, split brain). They are real, but two orders of magnitude less likely on your system, and they consume the same budget.
Running a known answer. You know that call has no timeout, and you inject latency anyway to "prove" it breaks. That is not an experiment; it is a self-scheduled incident.

So the question is: in a fault space you cannot enumerate, what explainable rule picks the next ten experiments?

The core content, properly

The fault space explodes, so the goal is ordering, not coverage

One relation is enough: candidate experiments ≈ dependencies × failure modes × other variation axes. Each factor is merely in the tens; multiplied, you land in the thousands. That settles it — chaos engineering cannot chase coverage the way unit tests do. It is sampling by nature, and all of its quality lives in the sampling rule.

It also explains why injection capability is not the bottleneck: your tooling can already generate far more experiments than you can run. Judgement is the scarce resource.

Three factors in the tens push candidates into the thousands, while the budget is that narrow sliver — so "how to inject" stopped being the question long ago; "in what order to sample" is the question.

Start from the dependency inventory: implicit ones are the biggest gap

Where the list comes from sets your ceiling: start from what the tool can do and the list looks like a feature list; start from dependencies and it finally looks like your system.

The test is one sentence: if it breaks and your request gets slower or fails, it is a dependency. Run that filter and you usually dredge up a batch of never-tested things: DNS, TLS certificate expiry and revocation checks, service discovery, config and feature-flag rollout, NTP, the metrics backend (does a thread block when writes fail?), the image registry (used only when scaling out, so it fails precisely when you most need to scale out).

The most valuable finding hides right here: the dependency labelled "non-critical." "Analytics failing doesn't affect checkout" is a hypothesis; inject once and you often discover the call is synchronous, has no timeout, and shares a thread pool with the main path. Criticality is an experimental result, not a claim in a design doc — so "everyone says it doesn't matter" is a high-priority signal.

Failure modes form a gradient: "slow" and "wrong data" are the real killers

This is the thing to take away. For one dependency, ordered by how hard it is for fault-tolerant code to absorb, there are four rungs:

① Fail fast: connection refused, immediate error. Easiest to absorb — circuit breakers, retries and fallbacks were all designed for it.
② Unavailable without an error: the request hangs until it times out. Assumes you actually set a timeout, and that the value was computed rather than left at a default 60s.
③ Degradation: still alive, p99 just went from 50ms to 3s. The sneakiest rung: no error code, and the dashboard still reads 100% availability.
④ Wrong or partial data: a clean 200 with content that is incorrect or truncated. Zero signal, silently poisoning downstream systems and caches.

Why rung ③ is the most dangerous can be computed: at 200 QPS upstream with a downstream p99 of 50ms, and concurrency ≈ QPS × latency, steady state needs about 10 slots; degrade to 3s and you need 600 slots while the connection pool holds 50 — within seconds the pool is full and requests that never touch that downstream start failing too. A "non-critical" dependency is thereby promoted to a site-wide outage, having never returned a single error.

So the first cut of variation should follow this gradient: every dependency you have already killed still owes you two experiments — inject latency, inject wrong data. Far cheaper than finding another dependency to kill, and far more productive.

Those three lines of arithmetic on the right are the chapter's most useful artefact: a dependency that never errors turns its own degradation into someone else's outage by occupying the caller's waiting slots.

The ranking rule: expected loss over cost, minus the known answers

Priority = probability × impact ÷ cost. All three need inputs you can state out loud:

Probability: the hardest input is incident history — a class of failure that has happened twice has a measured probability, not an estimated one. Next come the dependency's own SLA and availability record, then change frequency: recently modified things break more.
Impact: computed from traffic and money, not from how important the component feels — how much QPS this path carries, whether payments are involved, whether a failed user retries or churns.
Cost: design time, cross-team coordination and blast-radius management. At equal payoff, take the cheap one: run it on a single instance or a small traffic slice first, widen once it passes.

Two filters after the ranking save more budget than the ranking itself:

One: skip the known answers. If you are convinced it will fail (no timeout, no fallback) → file a bug and fix it, do not inject. If you are convinced it will pass and you have verified that → it does not deserve another window either. The product of an experiment is removed uncertainty, so what deserves running is exactly the batch you are confident about but have never verified — every surprise lives there.

Two: one injection collects three things. The same run can verify system behaviour, whether the alert fires, and whether people and runbooks keep up. Attaching monitoring verification costs almost nothing at the margin and is the thing most often forgotten. That also raises the priority of some "I'm confident" experiments — you run them not to watch the system but to watch the pager.

Key trade-offs and choices

The first table lands the gradient: for one dependency, what each of the four injections verifies and why it gets skipped.

Failure mode	What it verifies	Usual defence	Why it goes untested	Priority
① Refused / fail fast	Whether breaker and fallback paths actually work	Circuit breaker, retry, default value	It isn't skipped — nearly everyone starts here	Low (likely already verified)
② Timeout	Whether a timeout exists and was computed	Explicit timeouts + budget propagation	The default hides inside a client library nobody read	Medium-high
③ Degradation (latency)	Whether pool saturation spreads failure to unrelated requests	Bulkheads, load shedding, adaptive concurrency	No error code; the availability dashboard is all green	Highest
④ Wrong / partial data	Validation, reconciliation and cache-poisoning defences	Schema validation, checksums, reconciliation jobs	Costly to inject, and needs a correctness oracle	High (valuable but hard)

The trap is spending the whole budget on rung ①. That is precisely the one rung you have probably already absorbed.

The second table runs the chapter's ruler over a candidate list, to show what "which few first" looks like.

Candidate experiment	Probability from	Impact	Cost	Verdict
3s latency into a "non-critical" dependency	Criticality is a claim, never verified	Possible pool saturation → site-wide	Low, one instance will do	First
Reproduce the last incident's failure class	Measured — it already happened	Known and quantified	Medium, but the clearest hypothesis	Near the top, doubles as regression
Fail a recently changed dependency	Change frequency is the best prior	Scales with the path's traffic	Low	Make it routine, put it in the pipeline
Kill a random instance	High, but the outcome is verified many times over	Usually unnoticed	Very low	Keep as continuous background noise
Clock skew / split brain	Two orders of magnitude lower on most systems	Can be severe	High, hard to orchestrate	Worth it for stateful stores
Inject into a call you are sure will break	—	—	—	Don't: file a bug and fix it

Concretely:

Just starting, with only an instance killer: do not rush to extend the tooling. Spend an afternoon completing the dependency inventory (implicit ones included), then add one latency injection for each of the top three — latency is usually the cheapest and most productive cut.
Hundreds of experiments, all green: all-green is not an achievement, it is evidence of sampling bias. Vary on a new axis — move the same dependencies to rungs ③ and ④, or drop the failure ratio from 100% to 30% (partial failure is often harder to handle than total).
Budget so tight you run a few per year: put everything on "reproduce the last incident's class" — measured probability, known impact, clearest hypothesis, and a regression test as a bonus.

Why it matters / in real systems

Three lines that change review conclusions. One: the quality of chaos engineering is not injection capability, it is the quality of the sampling rule. Two: "everyone says it's non-critical" is a high-priority signal. Three: every dependency you have killed still owes you latency injection and bad-data injection.

Asked in an interview how you do chaos engineering, "we kill random instances weekly" is the tooling layer; "we complete the dependency inventory including DNS, certificates and config rollout, rank by probability × impact, inject latency into the dependencies believed non-critical first, and use the same injection to verify the alert" is another level — it shows you know the bottleneck is selection, not injection.

Evidence from industry

Netflix turned prioritization into an algorithm. The ranking this chapter does by hand is automated in Netflix's ChAP (Chaos Automation Platform) and Monocle: candidate experiments are enumerated automatically from service-dependency and injection-point data, scored and ordered by criticality and safety, filtered to drop the unsafe, the already-running and the recently-run, then executed in priority order — structurally identical to this chapter's claim that there are too many candidates not to rank them. Ali Basiri, Lorin Hochstein, Nora Jones, Haley Tucker, "Automating Chaos Experiments in Production," ICSE-SEIP 2019 (the host domain was blocked by this run's egress proxy; citation given in text only, no quotation)
Meanwhile chaos engineering's most famous tool is the lowest-variation rung. Chaos Monkey's own description: "Chaos Monkey randomly terminates virtual machine instances and containers that run inside of your production environment," and "Exposing engineers to failures more frequently incentivizes them to build resilient services." Note that it defines exactly one mode — termination. The chapter's point is not that this is useless but that it is the start, not the end: the same injection point still owes you latency and wrong data. Netflix · Chaos Monkey official repository README (github.com/Netflix/chaosmonkey, wording verified during this run)
The cautionary case: long outages usually come from "slow," not "dead." In Roblox's roughly 73-hour outage of 28–31 October 2021, per its own public review, the trigger was not a dead node but a newly enabled streaming feature on the service-discovery cluster causing contention under load and severely degrading the cluster's performance — exactly rung ③. Roblox, "Roblox Return to Service 10/28–10/31 2021," January 2022 (domain blocked by this run's egress proxy; citation in text only, no quotation)

Pitfalls and disputes

Treating injection capability as maturity. However many modes you can inject, the wrong list still only verifies what you already knew.
Celebrating all-green. Hundreds of passes more likely mean the sampling leaned toward the side you were already confident about, not that the system is hard.
Treating "non-critical" as low priority. The opposite: it is an unverified hypothesis, and the cost of being wrong is the whole chain.
Estimating probability by feel. Incident history is a free, measured prior, yet in many teams postmortems never flow back into the experiment list.
Only injecting 100% failure. Partial failure (30% of requests erroring, one slow shard) is often harder to handle, because no breaker reaches its threshold.
Dispute: ranking itself prunes the tail. Ordering by expected loss inherently neglects the low-probability high-loss class; guarding against it needs a second list that says "verify regardless of probability" (Ch17's cyber-physical systems are exactly that).
Period limits and what came after. This chapter still has humans ranking. The later direction is to compute where to inject from the system's own data — Ch12's lineage-driven fault injection (LDFI) and Ch16's continuous verification both replace that human step. Separately, many dependencies now sit in SaaS you can neither change nor inject into, which makes ranking matter more than it did at writing time: what you can shake was only ever a subset.

Takeaways

① Core claim: injection is not the bottleneck, selection is. Candidates ≈ dependencies × failure modes × other variation axes; three factors in the tens multiply to 10³–10⁴ while the budget is a few dozen per year — chaos engineering is sampling by nature, and all its quality is in the sampling rule.

② Build the list from dependencies, not from the tool. The test: if it breaks and your request gets slower or fails, it is a dependency — DNS, certificates, service discovery and config rollout all count.

③ Failure modes form four rungs: refused → timeout → degradation → wrong data. Fault-tolerant code is essentially only tested on rung ①; outages happen on ③ and ④.

④ Why rung ③ is lethal, in one calculation: 200 QPS × 50ms needs about 10 slots; degraded to 3s it needs 600 while the pool holds 50 — saturation turns a dependency that never errored into a site-wide outage.

⑤ The first cut of variation: every dependency you have killed still owes you latency injection and bad-data injection — much cheaper than finding a new one to kill.

⑥ Rank by probability × impact ÷ cost: the hardest input for probability is incident history, impact comes from QPS and money. A dependency called "non-critical" is a high-priority signal.

⑦ Two filters: never run a known answer (if you're sure it breaks, go fix it), and let one injection also verify the alert — near-zero marginal cost, most often forgotten.

⑧ All-green is evidence of sampling bias, not of resilience; and a ranking by expected loss needs a separate list for the low-probability, high-loss tail.