Chaos Engineering Ch 4 · Slack's Disasterpiece Theater

Chaos Engineering: System Resiliency in Practice · Ch 4 · Richard Crowley (Slack) · 2020

What is this chapter about?

The previous chapter settled the rules: stop relying on inspection, go cause real failures. Which raises the obvious next question — so what tool do I install?

This chapter's answer is a surprise: the first step isn't a tool. It's writing a script, booking a rehearsal, and getting people into one room. The author is an engineer at Slack, and the things he needed to shake were not shiny cloud-native services. They were the old, critical parts nobody had designed to break in the first place.

An analogy: a stage show with a stunt

Suppose a play calls for flying an actor on a wire. No company does this: opening night, full house, let's hoist someone up and see what happens.

The normal way: first a script — who does what at which minute, and what counts as going wrong. Then a walk-through in an empty house, where you get the embarrassing mistakes out of the way: wrong rope length, bad lighting cue, someone standing in the wrong spot. And on the night itself: the safety line stays clipped, someone stands at the switch, and the whole cast agrees in advance on the word that stops everything.

That rehearsal discipline is this chapter. The author calls it Disasterpiece Theater.

Why the old world was hard

The previous chapter mentioned the monkey that kills machines at random, on the assumption that traffic routes around them. It works well, but it carries a hidden precondition: those machines were built to die. Something takes over automatically; nobody has to go prop anything up.

Run a company for a few years, though, and the parts that matter most rarely look like that. They are more like an old house: pull one wall and the whole thing leans, and straightening it requires a person on a ladder. Throwing a monkey at an old house isn't a drill. It's a collapse. So teams get stuck. Don't touch it, and that layer stays an unverified black box forever. Touch it carelessly, and your first drill becomes a real outage — and the reputation of the whole practice dies on the spot.

The core intuition: four things

Write it down first. What you'll shake, which machines, which commands, what you expect to happen (whose alert fires? will users feel it?), and most importantly — how you stop. Writing and circulating the thing already catches problems: people think of half the surprises right there, with nothing injected.
Walk it through in the empty house. Run it once in an environment with no real users. Not for the conclusion — only to get the embarrassing errors out of the way: wrong command, missing permission, the injection tool itself broken. Otherwise on the real night you die somewhere completely unrelated to what you set out to learn.
On the night, have people in the room. Book a window and gather the people who know these systems to watch together. The reason is neat: one of the things under test is whether the alerts fire at all — so you can't count on alerts to summon anyone.
Anyone can call a stop, and "how long stopping takes" must be measured in advance. People assume stopping is instant. The undo path takes time too, sometimes minutes — so your smallest possible impact is "how long it was broken plus how long stopping takes".

What it buys you

The concrete win: you can finally touch the oldest, most critical parts, and the first touch doesn't turn into an incident. There's also a counterintuitive finding — the thing you hit most often isn't "it didn't hold up", it's "it held up and not one person knew": everything fine, not a single alert. One honest cost: this way of rehearsing is expensive in people. You get a handful per quarter, never one a day.

Remember this

For systems nobody designed to break, the first step in chaos engineering isn't installing a tool — it's writing the script, rehearsing in an empty house, and showing up with both the people and the stop switch in place. That's how you demote a potential outage into a rehearsal with a safety line.

Want the mechanisms, the numbers and the diagrams? → Switch to the deep read

In one sentence

You'd think step one of adopting chaos engineering is picking a fault-injection tool. Slack's answer: step one is a document, a review, a dry run in a non-production environment, and an abort plan whose latency you have actually measured. Because the Chaos Monkey route carries a hidden precondition — the thing being shaken was designed to die — and in any company more than a few years old, the systems that matter most are precisely the ones that weren't. This chapter doesn't hand you a tool. It hands you a process that lets a legacy system be shaken for the first time and produce knowledge instead of an incident.

Glossary

Legacy system: something written years ago that still carries critical traffic. Here specifically: designed on the assumption that infrastructure is reliable, so failure is an exception rather than the norm.
Stateless: an instance holds no data of its own, so another one can serve just as well and it can be killed at any time. The extreme form is crash-only: the only way to stop is to crash.
Failover: moving traffic and roles from a dead primary to a standby. The variety that needs a human is where this chapter's difficulty lives.
Service discovery: the mechanism that tells callers which machines to connect to right now (Consul, DNS, a registry). It decides whether traffic routes around a vanished machine on its own.
Dev / staging: an environment shaped like production but carrying no real users; scale, data distribution and configuration all differ.
Game day: a failure exercise at a pre-agreed time with people present; this chapter's process is a disciplined version of one.
Blast radius: the worst-case reach of one experiment. This chapter adds a dimension to it: duration.
Steady state: the measurable, user-facing output of a healthy system, used to judge an experiment's outcome (principle one of the previous chapter).

Where it sits

Chapter 4 of 21, and the opening of Part II, "Principles in Action" — Ch4 through Ch8 are five companies bending the previous chapter's five principles to their own constraints, and this is the first and most accessible bend: principle three (run in production) is kept; principle four (automate, run continuously) temporarily yields to a human in the loop. Downstream, Ch5 is Google's DiRT (the same posture at company scale), Ch7 is LinkedIn's ethical limits on blast radius, Ch8 is Capital One's gradual path under compliance, and Ch16 returns to full automation. In the real world it speaks to teams with a pile of services written years ago, no unified platform, and a mandate to start verifying resilience — which is to say, most teams.

What problem the chapter solves

The previous chapter's five principles carry an unstated assumption: the system under experiment was designed for failure. Chaos Monkey can terminate instances at random only because those instances are stateless, orchestrated by one platform, and routed around automatically once dead.

But real infrastructure is sedimentary. The dividing line this chapter draws inside Slack is blunt: newer systems are built on "failure is normal" — no local state, restartable at will, traffic routes around them. Older systems are built on "infrastructure is reliable" — local state, failover that needs a human, sometimes a dependency on the identity of one specific machine. Unhelpfully, the older systems tend to be exactly the critical layer: databases, service discovery, message routing, identity. For scale: Slack at the time had daily actives in the millions, and every connected client holds a long-lived WebSocket connection, so a wobble in the connection layer means every online user reconnecting at once.

So the question becomes: how do you experiment on a system that was never designed for failure, without the experiment itself becoming the outage?

What happens if you don't answer it? Either you don't dare — the most critical layer stays an unverified black box and your confidence rests entirely on "it hasn't failed yet", which the previous chapter just disqualified as evidence. Or you barge in — the first exercise becomes a real incident. The technical damage is bounded; the political damage is fatal, and the credibility of chaos engineering at that company goes to zero on the spot. Half of this chapter's process is engineering design; the other half is protecting the practice's political life.

The core content, properly

The dividing line: when a machine vanishes, does traffic route around it, or does a human have to act?

This is the most actionable sentence in the chapter; go ask it of every system you own. "Routes around" means the consequences are bounded and known: instance dies, health check pulls it out, the rest absorb the load. You can compute the worst case, so you can hand the judgment to a machine and let it shake things at random. "A human has to act" means the consequences are not bounded in advance: after injection the system enters a state nobody has seen, and recovery time depends on who's around, how well they know it, and whether the runbook is current. Random automated injection there isn't an experiment. It's a lottery ticket.

This also explains why Chaos Monkey isn't a general answer — it hard-requires a platform that already manages your applications (see the evidence in §6): a system without that platform isn't "not onboarded yet", it fails the precondition architecturally.

The question isn't judging system quality. It's deciding whether the judgment can be handed to a machine — and when it can't, it goes to a process and to people.

The pipeline: document, review, dry run, production exercise, follow-up

This pipeline is the body of the chapter: it looks heavy, but every stage eliminates one specific class of failure.

1. The document (the most important step). The exercise's host writes a short document up front that has to answer: which system, which fault, which commands, which instances; which dashboards, alerts and logs to watch; and the hypothesis — what you expect to happen, including which alerts will fire and whether customers will feel it.

One question in that document is beautifully designed: "how confident are you that fault tolerance in the dev environment predicts fault tolerance in production?" It converts the abstract argument behind principle three into something the host must answer in writing, and the answer changes the design: "very confident" lets the production run be brisk; "not confident at all" demands a smaller radius, a shorter window, and more people watching.

2. Review. The document circulates among the relevant teams. This step is routinely undervalued: many problems get thought of during circulation, with nothing injected — the cheapest experiment on the whole pipeline.

3. Dry run outside production. Run it once in dev. It does not substitute for production (that would be quietly abandoning principle three); its only job is to get the embarrassing errors unrelated to your hypothesis out of the way: wrong command, missing permission, broken tooling, a dependency that won't start. Left in, those kill you somewhere irrelevant and burn a production window.

4. The production exercise. Book a window and gather the experts in one place to watch together. Why not let alerts summon them? Because the alerting is itself under test — you can't guarantee the verification process with the thing being verified. Scheduling it in business hours has a second benefit: the people who know the system best are awake, rather than the on-call being dragged out of bed at 3 a.m.

5. Record and follow up. Write down what actually happened, where it diverged from the hypothesis, and file the fixes. Skip this and the output of the first four steps decays into "everyone knew at the time" and is gone in three months.

The document and the dry run both push failure earlier — and the earliest failure is also the cheapest, happening before anyone types a command.

The abort plan is a first-class citizen, not an appendix

Most teams' abort plan is one line: "roll back if there's a problem." That is not a plan. The chapter wants three specifics:

The exact undo action: which command, which flag, who has the permission to run it.
How long the undo takes to take effect. The most-forgotten and most surprising item: the undo path is usually the less-tested one and can take minutes — service discovery has propagation delay, connection pools rebuild slowly, caches need re-warming. So your real blast radius is reach × (time injected + abort latency), and if you never measure the second term, squeezing the first to 1% means nothing.
Who may call a stop, against what criteria. The answer is "anyone in the room", and the criteria are written down in advance (principle one). Nobody should be arguing on the spot about whether this counts as abnormal.

The three findings this process keeps hitting

Ordered by value, and the most valuable one is the least expected:

The system held up, and nobody knew it held up. Tolerance worked, users felt nothing — and not a single alert fired, or a few irrelevant ones did. It tells you that next time, for real, you'll find out tens of minutes late. In other words: the first thing an exercise verifies isn't whether the system survives, it's whether you'd know.
It survived the fault and died on recovery. The moment you undo: mass reconnection, cold caches, a retry storm. The recovery path gets tested far less than the failure path — during a real incident everyone is firefighting, and few people circle back to verify that the recovery action itself is safe.
The fault won't go in at all. You run the commands from the document and nothing happens, because the real configuration isn't what you believed: that "redundancy" was quietly turned into a single point by some change, or the traffic never took the path you assumed. This one says your mental model is wrong — Ch1's claim cashing out in front of you.

Why a human in the loop is the right starting point, not a backward compromise

Automated injection has a precondition: you already know the consequences of this fault are bounded. The first time you shake an old system you specifically don't — so having people present isn't timidity, it's supplying the missing precondition.

That also gives the direction of travel: the marginal value of humans in the room is highest the first time and lowest the Nth. Once the same experiment has run a few rounds, the consequences are understood and the criteria can be written as code, hand it to a machine to run daily (Ch16). The process is the entrance, not the destination.

And one by-product automation can never produce: education. An exercise puts five to ten engineers in one place, looking at one set of graphs, correcting their own guesses in the same minute — forcibly stitching together the partial mental models of Ch1. A pipeline produces a green light; an exercise produces five updated understandings.

Key trade-offs

The first table applies the dividing line: which approach each of your systems deserves.

System shape	Take one machine away	Automated injection?	The right approach	Cost / failure mode
Stateless + platform-orchestrated	health check pulls it; the rest absorb	Yes	random automated injection, continuously	presupposes the orchestration platform
Stateful + automatic failover (consensus cluster, auto-promote)	it fails over, with seconds of wobble and a data cost	Semi	a few human-in-the-loop rounds first, confirming failover actually happens and takes an acceptable time	failover itself may lose the last sliver of writes
Stateful + manual failover	enters a state nobody has seen	No	Disasterpiece Theater: document, review, dry run, humans present	expensive in people; frequency stays low
A genuine single point	straight to unavailable	No	don't experiment — go build redundancy	shaking it only confirms what you already know

One criterion: when this machine vanishes, does traffic route around it or does a human act? That decides whether the judgment can go to a machine.

The second table is the "where to run it" trade-off, and the point is to be clear about what each tier cannot catch.

Where	Catches	Cannot catch	When to use it
Dry run outside production	wrong commands, missing permissions, broken tooling, absent dependencies	anything that depends on scale: pool exhaustion, retry storms, cache collapse	before every production exercise, always
Production, human in the loop (this chapter)	behaviour at real scale, whether alerts fire, whether recovery is safe	rare fault combinations — frequency stays at a handful per quarter	the first time you shake a legacy system
Production, automated (Ch16)	frequency and coverage, plus regression: does last month's conclusion still hold?	anything needing live human judgment; it only checks criteria already encoded	once the same experiment has run a few rounds

Concretely:

First time shaking a critical legacy system: run the whole pipeline — one variable, one window, business hours, people present. Don't skip the dry run; the step you skip is the one that kills you with a typo inside a production window.
A platform team rolling this out: invest in the abort switch, per-group metrics and an exercise document template before piling up injection mechanisms. The previous chapter said injection was never the bottleneck; this one adds: the host's confidence is the bottleneck, and templates and dry runs are how you manufacture it.

Why it matters / in real systems

Three sentences from this chapter can change a design review outright. One: before asking "which system do we shake first", ask "when it vanishes, does traffic route around it or does a human act" — the answer picks the method, rather than the tool picking it for you. Two: an exercise first verifies not whether the system survives, but whether you'd know. Three: an abort plan has to report a number — how many seconds until the undo takes effect.

Asked in an interview how you verify failure behaviour in production, "we use tool X" is the tool layer. "Our critical systems weren't designed for failure, so we run document, review, dry run, humans in the loop, and the abort latency has to be measured" is a different level — it shows you know what automation's preconditions are.

Evidence from industry

Netflix put the precondition in the README. Chaos Monkey's project description states what it does in one line: it "randomly terminates virtual machine instances and containers that run inside of your production environment." But the hard prerequisite follows immediately: "This version of Chaos Monkey is fully integrated with Spinnaker, the continuous delivery platform that we use at Netflix. You must be managing your apps with Spinnaker to use Chaos Monkey to terminate instances." That is this chapter's dividing line, stated by the tool's own authors: the monkey route isn't "not onboarded yet", it fails the precondition architecturally. Netflix · Chaos Monkey project description (github.com/Netflix/chaosmonkey, text verified during this run)
Google's DiRT is the same posture at company scale. Exercises are planned in writing, reviewed in advance, and required to have a way to be stopped — and what gets shaken includes processes and people, not only technical components. It shows that "humans in the loop plus a written plan" is not a small-company stepping stone but the only honest way to shake what cannot be automated. It is also the subject of the next chapter. Kripa Krishnan · "Weathering the Unexpected" · ACM Queue, 2012 (that domain was blocked by the egress proxy during this run; text-only citation)

Pitfalls and open questions

Verifying "it held up" and not "the alert fired". A silent pass with every alert quiet gets recorded as a success; it is in fact a red light.
A dry run that passes, so you skip the production run. That is quietly abandoning principle three: a dry run catches commands and permissions, and nothing that depends on scale.
Treating the process as a placeholder for "we're not advanced yet" and rushing to automation. Automation's precondition is bounded consequences; jumping before you have it outsources the lottery ticket to a machine.
Conversely, treating the process as the destination. Low frequency is its genuine defect: a handful per quarter, and coverage that doesn't compare. Several boring rounds in a row is a signal to automate, not a reason to celebrate.
Using exercises to evaluate engineers. The moment someone is blamed for what an exercise exposed, nobody will host the next one and nobody will put a pessimistic hypothesis in writing — the process degrades into theatre immediately. That's Ch18's subject, and the unstated precondition for this chapter's process working at all.
The open question: is this even chaos engineering? Read principle four strictly and a human-in-the-loop exercise is closer to a game day. The honest cost is frequency and coverage; the defence is that for these systems automation's precondition does not yet exist, and not doing it is worse than doing it slowly.
Period limits. This is Slack, 2018–2020. Unified orchestration and mature injection frameworks have made the right-hand column shorter, not empty: every company still has its own sediment — databases, identity, billing, settlement. This process keeps its shelf life for exactly as long as that layer does.

Takeaways

1 · The claim: step one of adopting chaos engineering is a process, not a tool — because the systems that matter most were never designed to break, and automated injection against them isn't an experiment, it's an outage.

2 · The dividing line: when a machine vanishes, does traffic route around it or must a human act? Human → consequences unbounded → human in the loop. It also explains why Chaos Monkey isn't general: it hard-requires an orchestration platform.

3 · Five stages: document → review → non-production dry run → human-in-the-loop production exercise → record and follow up; each eliminates a class of failure, and earlier failures are cheaper.

4 · The best question in the document: "how confident are you that dev fault tolerance predicts prod fault tolerance?" It turns "staging isn't production" from an abstract argument into something answered in writing and acted on.

5 · Gather the experts in one place for the production run, because "will the alert fire?" is itself under test — you can't guarantee the verification with the thing being verified. Business hours mean the people who know it best are awake.

6 · The abort plan needs three specifics: the undo action, how long the undo takes, and who may call a stop. Real blast radius = reach × (time injected + abort latency).

7 · Three typical findings, the first most counterintuitive: it held up and nobody knew (alerts silent); survived the fault and died on recovery; the fault wouldn't go in at all (your mental model was wrong).

8 · A human in the loop isn't backwardness — it supplies automation's missing precondition, and produces what automation can't: five to ten mental models updated at once. But frequency stays low, so the destination is still Ch16's continuous verification.

9 · The moment exercises are used to assign blame they degrade into theatre: nobody hosts, nobody writes down a pessimistic hypothesis.