BOOKS DEEP-READ · CHAOS ENGINEERING · CH 1
Chaos Engineering: System Resiliency in Practice · Ch 1 · Casey Rosenthal & Nora Jones · 2020
When you open Netflix to pick a film, or load a shopping app's home page, what runs behind that tap is not "a program." It is a few hundred small programs shouting at each other: one recognizes who you are, one decides what to recommend, one serves the cover art, one does the money. Chapter 1 of Chaos Engineering teaches no technique at all. It asks something more uncomfortable first: why does a system whose every part is healthy still fall over?
A mechanical watch has hundreds of parts, but it is merely complicated: lay the parts out on a table, understand each one, and you can work out how the whole watch keeps time. Morning rush hour is not like that. Not one car is broken, and the road still seizes up. The jam does not live inside any car. It lives between the cars. The whole claim of this chapter is that today's production systems have moved from being a watch to being a rush hour.
Reliability used to come from understanding: draw the architecture diagram, write the design doc, hold the review, test each part, then sign off. That method has an assumption nobody writes down — that somebody holds the whole system in their head. That assumption is gone. Every engineer now holds one shard of the picture, and because the system changes daily, their shard is quietly going stale. A review meeting gives you confidence, not reliability, and there is no causal link between the two.
If you cannot reason your way to the answer, go and try it: deliberately cause a failure inside a boundary you control, and watch what the system actually does — instead of guessing what it ought to do. That is the starting point for the entire book. One honest cost: causing failure in a real system means real users get touched, which is why most of the rest of the book is about shrinking the number of people who do.
Failure in a complex system does not live inside the parts; it lives between them. Nobody holds the whole system, so reliability cannot be reasoned into existence — only tried.
Want the mechanisms, the magnitudes and the diagrams? → Switch to the deep read
You assume failure comes from a broken part — find it, fix it, and it will not happen again. This chapter exists to dismantle that instinct: in a complex system, the overwhelming majority of serious failures emerge from the seams between components that are each entirely healthy and each entirely within spec. If the behavior of the whole cannot be derived, then reliability stops being an argument you win and becomes an experiment you run — the single premise underneath every method in the book.
Chapter 1 of 21, opening Part I, "Setting the Stage." Casey Rosenthal built the first Chaos Engineering team at Netflix; Nora Jones was a practitioner of the same generation. This chapter offers no method, only the premise: Chapter 2 goes on to how systems keep drifting toward the failure boundary, and Chapter 3 finally gives the five principles. What it points at in the real world is any production system assembled from dozens to hundreds of services and still being changed every day.
Traditional reliability engineering carries an assumption that never made it into any document: the system is comprehensible. So every method orbits comprehension — draw the diagram, write the design doc, hold the review, push coverage up.
Put real numbers on the table and the assumption collapses. A mid-sized internet backend has a few hundred services and over a thousand deploys a day; nobody can finish drawing the call graph. Worse, the number of seams grows faster than the number of components: by n(n-1)/2 — the count of distinct pairs you can form from n components — 10 components give at most 45 pairs, and 100 components give 4,950. Components grow linearly; seams grow quadratically. And failure lives in the seams.
Nonlinearity then distorts the magnitudes outright. If a request fans out to 100 backends and each backend is slower than one second for just 1% of responses, 63% of user requests hit at least one slow backend: a component-level "1% nuisance" becomes "most people are slow" at the user. By the same arithmetic, 100 serial dependencies each at 99.99% availability multiply out to roughly 99.0% — nearly 87 hours a year unavailable.
What happens if you ignore it: after every incident you fix "the last one," and the next one still arrives as a combination you did not think of. Reviews and diagrams give you confidence, not reliability.
A merely complicated system satisfies one property: the whole equals the sum of the parts. A mechanical watch or a compiler can have an enormous part count, yet once you know what each part does, the behavior of the whole is derivable. The test this chapter gives is usable as-is: can you verify every component in isolation and declare the whole correct on that basis? If yes, it is complicated, and reviews plus tests are the best value for money. If no, it is complex, and you need something else.
Microservices, autoscaling, multi-region deployment, a thousand deploys a day — every one of them pushes a system toward the latter. Complex systems also have an uncomfortable property: nonlinearity. The same perturbation does nothing today and amplifies into a site-wide incident tomorrow, because the system was in a different state each time. So "the last game day was fine" never implies "this one will be" — which is the root of the later principle about running continuously and automatically.
Every engineer's model is locally correct and globally stale. Locally correct is the dangerous part: it makes people certain their judgment is right. An architecture diagram is a hope, not an observation — it describes how the system is supposed to run, not how it is running; and in a system deploying a thousand times a day it begins diverging from fact the moment it is finished.
There is a corollary that gets skipped. In a postmortem, the causal chain always looks obvious. People therefore conclude "he should have seen it coming" — but beforehand that chain was hidden among four digits' worth of seams, and nobody can enumerate those. Mistaking "obvious afterwards" for "foreseeable beforehand" is the most expensive cognitive error in complex systems, and it quietly turns postmortems into blame.
① Business logic mismatched with application logic. At a streaming company, user data was spread across several microservices. One of them handled "record not found" by returning a default value — textbook defensive programming, do not let exceptions escape. The service calling it carried on computing on the basis that "the data I received is the user's real state," which is equally within spec. Each piece of logic is right on its own: one says do not throw, the other says act on the data. Together, a slice of users saw content that did not match their account. No component had a bug; the fault lay in the seam between two correct assumptions.
② A customer-induced retry storm. A backend gets slow → requests time out → clients retry per their configured policy → arriving load doubles → slower → more retries. Retrying is good engineering in itself (transient blips are exactly what retries are for), but laid over "the backend is already saturated" it becomes positive feedback: three retries per failed request is enough to push arriving load to nearly 4× the original. What makes it vicious is that the loop sustains itself — by the time you arrive, whatever first slowed the backend may be long gone, yet the system is still pinned in saturation and only an external force (shedding, rate limiting, a cold start) gets it out.
③ The holiday code freeze. To keep the peak season calm, a company froze all change. The freeze became the source of the incident. The conclusion matters more than the specifics: this system's steady state already included "continuous change" — routine deploys had been restarting processes, clearing accumulated state, and rotating things before they expired, all as side effects. Removing change does not hold the system still; it pushes it into a state nobody has visited and therefore nobody has verified. Put differently: "doing nothing" is also a change, and an untested one.
Lay the three side by side and the pattern you are meant to take away has four parts: every component met its own spec; the failure happened in a seam; it was obvious afterwards and unenumerable beforehand; and the trigger was a point in the state space nobody had ever visited. Together, those four are the proof that reasoning alone is not enough.
In No Silver Bullet (1986) Brooks cut complexity in two: the accidental kind comes from our tools, our baggage and our duplicated implementations, and can be deleted; the essential kind comes from the problem itself. This chapter borrows that knife to say something unpopular: most of the complexity in our systems is essential — personalization, multi-region, A/B experiments, many client platforms, a four-nines availability target. Every one of those is value the business asked for, and deleting the complexity means deleting it.
Hence the most counterintuitive line in the chapter: complexity is a byproduct of success. Your system is this complex because people use it, it makes money, and it grew. The goal therefore cannot be to eliminate complexity (you cannot), but to acquire the ability to act safely inside it. One honest caveat: this is easily quoted as "so stop refactoring." The book is not against cutting accidental complexity — cut it, it is the best value available; it says only that what remains afterwards does not go away, and you need a second method for that part.
If the behavior of the whole is not derivable, where does knowledge about the system come from? The chapter's answer is empiricism: put the system into that condition and watch what it does.
There is a dividing line here worth memorizing: a test verifies an assertion you already know; an experiment produces knowledge you do not. Unit and integration tests all say "I assert that under X we should get Y" — they can only cover the X you thought of. A chaos experiment asks a different kind of question: "under X, what actually happens?" The answer can be a Y you never imagined. That is why chaos engineering is not testing but experimental method. The five principles of Chapter 3 — hypothesize about steady state, vary real-world events, run in production, automate continuously, minimize blast radius — are all the engineering of that one sentence.
There are only so many strategies for facing complexity. They are not mutually exclusive, but each hides an assumption, and used in the wrong place each is money burned:
| Strategy | What it assumes | When it genuinely works | Cost / where it fails |
|---|---|---|---|
| Docs + architecture review | somebody holds the whole system | few components, slow change, stable interfaces | produces confidence, not evidence; stale after one deploy |
| Unit / integration tests | you can think in advance of what to assert | logic inside a component; regression guarding | covers only conditions you imagined; seam interactions are untestable by construction |
| Formal methods (TLA+ etc.) | the critical part abstracts into an exhaustible model | protocols, consistency, concurrency control | costly to model; verifies the model, not the implementation in production |
| Observability | the failure is already happening | cutting time to diagnose (MTTR) | reactive; tells you it broke, never what will break next |
| Chaos experiments | you accept paying a controlled loss up front | many components, fast change, lots of emergent behavior | needs real traffic, so real users are touched; a bad design just manufactures noise |
These are not substitutes but coverage of different failure classes; the chapter only argues that the first four together still leave a gap.
Which to pick where can be made quite concrete:
The most useful thing this chapter hands you is language for pushing back. The review-meeting favorite — "every one of our services is at 99.99%, so we are fine overall" — stops working here: 100 serial dependencies at four nines each multiply out to about 99.0%, nearly 87 hours a year. Asked in an interview how you would make a system reliable, answering "reliability is an emergent property, it cannot be derived, so I would run controlled experiments in production" carries far more weight than reciting middleware names.
EVIDENCE FROM INDUSTRY
① The central claim: serious failures in a complex system come from interactions between components that are each healthy, not from broken parts.
② The test: can you "verify each part separately and declare the whole correct"? Yes means complicated, no means complex.
③ Seams scale as n(n-1)/2: components linear, seams quadratic — 10 components, 45 pairs; 100 components, 4,950.
④ Nonlinearity distorts magnitudes: fan-out 100 at 1% slow each ⇒ 63% of users slow; 100 serial four-nines ⇒ about 99.0% overall.
⑤ The shared pattern of the three encounters: components in spec, failure in the seam, obvious after and unenumerable before, trigger in a never-visited state.
⑥ A retry storm is positive feedback: once inside, removing the original cause does not get you out — shedding, limits or a cold start must cut it.
⑦ A code freeze does not hold the system still; it pushes it into an unverified state — not changing is also a change.
⑧ Most complexity is essential and a byproduct of success; the goal is not to remove it but to be able to act safely inside it.
⑨ Tests verify assertions you know; experiments produce knowledge you do not — that line separates chaos engineering from testing, and starts every method in the book.