BOOKS DEEP-READ · CHAOS ENGINEERING · CH 2
Chaos Engineering: System Resiliency in Practice · Ch 2 · Casey Rosenthal & Nora Jones · 2020
Chapter 1 said: modern production systems are too complex for anyone to hold in their head, so failure cannot be reasoned out in advance. This chapter asks the more uncomfortable follow-up — if nobody wants an outage, why does the system still walk, step by step, toward the day it has one? The answer is not "somebody got lazy." It is worse than that: the forces pushing it toward danger every single day are all perfectly legitimate.
You are driving across the country. Three invisible cords are tugging at you:
Over a day you settle somewhere comfortable: not burning too much fuel, not exhausting yourself, not crashing. Here is the problem. Pull the first two cords too far and it hurts immediately — you watch the fuel gauge drop, you feel yourself getting tired. The safety cord is different: you can speed and tailgate a hundred times with no consequence, and the hundred-and-first time you crash. It says nothing at all until the instant you cross it, and then it hits you all at once.
A team works the same way. Saving money (switching off standby machines) gets you praised. Doing less fussy work (let's skip the drill this quarter) makes everyone's week easier. And "what you are doing is already dangerous" is told to you by nobody — because it did not blow up the last hundred times. The system does not suddenly become dangerous one day; it moves a little that way every day, and each step looks fine. The cruelest part: when you finally do get feedback from the safety cord, that feedback's name is "the outage."
To stop drifting past it, you have to make the safety cord speak up on ordinary days. The way to do that is to deliberately cause a small failure inside a controlled boundary and watch how the system actually responds. It is the equivalent of practising an emergency stop in an empty lot — learning how far you are from losing control before you learn it on the motorway. That is the whole job of chaos engineering in this model: it is not there to find bugs, it is there to shout on behalf of the safety boundary.
The chapter also counts where complexity actually comes from — four places: how many different situations the system can be in, how many ties there are between its parts, how much sits outside your control, and whether you can undo things after they go wrong. The first three are what the business is buying: one more feature means a pile more situations, one more partner means a pile more ties, and users and markets never did take your orders. That leaves exactly one that can be an engineering goal: how fast you get back to how things were. It is why the industry's big money goes into "one-click undo, try it on a tiny slice of people first, be able to switch it off any time" rather than into "we promise not to make a mistake this time." One honest cost: those abilities are not free either, and they only shrink the damage once you are wrong — they do not stop you being wrong.
A system does not turn dangerous one day; a few entirely legitimate forces nudge it that way daily — and of its three boundaries, only safety never warns you in advance. Chaos engineering is how you give it a voice.
Want the mechanisms, the numbers and the diagrams? → Switch to the deep read
You think an incident starts with "somebody made a mistake." This chapter replaces that starting point: the three forces pushing a system toward failure — save cost, save effort, chase the business — are each entirely legitimate on their own, and of the three boundaries only safety gives you no pain signal at all until you have crossed it. Drift, therefore, is not negligence; it is the inevitable solution to that force diagram. Two models turn Chapter 1's "you cannot reason it out" into "so you must manufacture a safety signal yourself" — which is the reason chaos engineering exists at all.
Chapter 2 of 21, the middle chapter of Part I, and the bridge between Ch1 and Ch3: Ch1 establishes that complex systems cannot be reasoned out, Ch3 delivers the five principles, and this chapter answers the question in between — then why does the system keep getting more dangerous, and where should I spend my effort? The Dynamic Safety Model is not invented here; it comes from Rasmussen's resilience engineering and is carried into a software context. In the real world it is aimed at any organization that has not had an outage in a long time and is therefore steadily cutting redundancy and drills.
Ch1 proved that whole-system behavior cannot be derived. But it leaves a hole: if nobody is actively breaking the system, why does it become more dangerous on its own?
Leave that hole open and you reach an expensive conclusion: a long time without an incident = currently safe. Make the scenario concrete and you can price it. A service runs 18 months with zero incidents, so four things follow quite naturally: standby capacity in the failover region drops from 100% to 30% (seven figures a year saved, and "we have never actually failed over anyway"); the quarterly failure drill is cancelled because it crowds out roadmap work; timeout and retry parameters were set three years ago and nobody has revisited them; the circuit breaker is configured but has never actually tripped.
Not one of those four decisions is wrong. Each has a near-term payoff and each is backed by data — the money really was saved, the features really did ship, 18 months really did pass without an incident.
Then a genuine regional failure arrives. 30% standby capacity has to absorb 100% of traffic; scaling up takes minutes; during those minutes a retry storm saturates what capacity is left; and the circuit breaker trips for the first time, its half-open probing knocking over a dependency that had just recovered. One incident settles all four "correct decisions" at once.
That is what the chapter solves: a force diagram for why systems drift, and where engineering should add the opposing force. Leave it unsolved and your reliability budget is permanently richest one month after an incident, eroded month by month thereafter, and periodically runs into the wall.
The chapter reframes Rasmussen's model into a picture that sticks: an engineer stands in the middle, tied by three rubber bands to Economics, Workload and Safety. They can move around that space and stretch the bands; stray too far from one direction and that band snaps. Each snap looks like this:
The asymmetry is the point: economics and workload are pulling on you well before they snap — the cost report reminds you monthly, on-call fatigue weekly; the pain is continuous and immediate. Safety's band is not. You can walk in its opposite direction for a long time in total silence, right up until it breaks.
Translate that asymmetry into a force diagram and you get the chapter's hardest conclusion: two of the three directions have a steady gradient pushing you outward, and only safety lacks a counter-gradient pulling you back.
Two ever-present forces plus one that shows up briefly after an incident leaves exactly one resultant direction. So drift is not negligence, it is the inevitable solution to this force structure — which disposes of the line you hear in every review, "they should have been more careful": in a system where only the safety direction lacks feedback, "be more careful" is a demand with no fulcrum. It also explains a familiar pattern: the reliability-investment curve is always sawtoothed. A big incident buys budget and attention, then 6–12 months of cost and delivery pressure erode it month by month, then you hit the wall again. The sawtooth is not a culture problem; it is the geometry of a missing counter-gradient.
This gives chaos engineering a very precise job description — it is not there to find bugs, it is there to install a signal on the safety boundary that you can feel on ordinary days.
Plainly: deliberately cause a failure under controlled conditions, and the system's real response is a measurement of how far you are from the safety boundary. It converts safety from "a discrete event that only occurs when you cross the line" into a continuous quantity you can read any time — the counter-gradient that was missing.
Concretely: a planned regional evacuation drill measures that at 30% standby capacity scale-up takes 7 minutes, while a retry storm can saturate the remaining capacity in 40 seconds. Those two numbers are now on the same table as the numbers in the cost report, able to argue with them. Safety loses budget arguments not because nobody cares but because it historically could not produce a figure in the same units as cost and delivery.
The honest cost: this gradient is bought, not found — drills consume people and environments, and run in production they will touch some real users. Chapters 4 through 8 are largely about minimizing exactly that cost.
The chapter's second model answers a different question: if complexity cannot be cut away (Ch1's conclusion), what is holding it up? Four pillars:
n(n-1)/2 — the count of ways to pick two components — i.e. quadratically.They are called economic pillars because the height of the first three is not an engineering choice, it is what the business paid for: one more feature = more state, one more channel = more relationships, one more market = more environment. Their growth is precisely the evidence that the product is succeeding — Ch1's "complexity is a byproduct of success."
Which makes the actionable conclusion sharp: the first three you can only accept; the fourth is your actual lever.
Line up the shopping list by reversibility and it writes itself: one-click rollback, immutable deployment (stand up new machines rather than mutating the old ones), canary releases, feature flags (switch off the one feature instead of reverting a whole version), blue-green cutover, dual-write before switching reads during a data migration. None of them solve "don't be wrong this time"; all of them buy "be back to normal within 2 minutes of being wrong."
Here is the point most readers skip: reversibility only exists once it has been exercised. A rollback script never run, a failover region never actually cut over to, a circuit breaker configured but never tripped — all of them count as reversibility on paper and all of them are unverified assumptions. This is exactly where chaos engineering joins this pillar: a drill measures the fourth pillar's real height.
And a form that is easy to miss: reversibility lives not only at the deployment layer but at the request layer. Under pressure, deliberately shed recommendations, comments and personalization to protect the checkout path — graceful degradation is temporarily narrowing the state space and cutting the relationship count, per request.
What to do about each boundary differs mainly on one axis: is there feedback on ordinary days?
| Boundary | Crossing it looks like | Feedback before you cross? | Where the counter-force comes from | Cost / failure mode |
|---|---|---|---|---|
| Economics | Reliability spend cut in cost review, or the company never lived to use it | Yes — the cost report, monthly | No need to build it; naturally abundant | Over-saving deletes safety margin as if it were waste |
| Workload | Process quietly bypassed; attrition | Yes — on-call fatigue, weekly | No need to build it; naturally abundant | Hidden: the dashboard reads "compliant" while people route around it |
| Safety | The incident | No — you learn at the moment of crossing | Must be manufactured: chaos experiments, error budgets, scheduled drills | Costs people and environments; production experiments will touch real users |
All the information is in column three: of the three boundaries, exactly one requires you to pay for an instrument.
How to treat the four pillars decides where the engineering budget goes:
| Pillar | Who raises it | Can engineering lower it? | What to do |
|---|---|---|---|
| State | Features, flags, configuration, data shapes | Barely — lowering it means deleting features | Converge config sources, eliminate implicit state; accept that it grows |
| Relationships | Splitting services, third parties, shared platforms | Partly; both splitting and merging cost | Map the critical path, mark which dependencies are genuinely indispensable |
| Environment | Users, markets, cloud providers, regulators | No | Assume it will turn hostile and prepare for that |
| Reversibility | Nobody raises it for you | Yes — and it is the only one | Put the budget here: rollback, canary, flags, degradation — each one exercised |
Which to choose when, concretely:
The chapter's most useful output is one sentence you can use in a cost review: "we have had no incidents for 18 months" is not evidence of safety, only evidence that we have not crossed yet. Asked in an interview how you would drive reliability investment, answering "by expressing safety in the same units as cost — an error budget, plus the real recovery time a drill measured" is a level above "I would emphasize that reliability matters."
EVIDENCE FROM INDUSTRY
① Core claim: systems drift continuously toward the failure boundary, and each of the three forces pushing them there (save cost, save effort, chase the business) is legitimate.
② The Dynamic Safety Model (reframed from Rasmussen): an engineer tied by rubber bands to economics, workload and safety; stray too far from any direction and that band snaps.
③ The key asymmetry: economics and workload hurt continuously before they snap (cost report, on-call fatigue). Safety does not — it speaks only at the moment you cross it.
④ So drift is the force structure's inevitable solution, not anyone's negligence; the sawtoothed reliability-investment curve has the same cause.
⑤ Chaos engineering's job here: not finding bugs, but installing the missing counter-gradient on the safety direction so it can produce a figure in the same units as cost.
⑥ The four economic pillars of complexity: state, relationships, environment, reversibility. The business pays up the first three — they are a byproduct of success.
⑦ Reversibility is the only pillar that can be an engineering goal — rollback, immutable deployment, canary, flags, degradation all buy "how fast back to normal."
⑧ Unverified reversibility is no reversibility: a rollback script never run, a failover never exercised, a breaker never tripped are assumptions.
⑨ "N months incident-free" is not evidence of safety, only of not having crossed yet; and the workload boundary's failure mode is the most hidden — not more care, but process quietly bypassed.