BOOK DEEP READ · SRE · CHAPTER 22

Addressing Cascading Failures: failure breeds, so fixing the cause won't bring the system back

Site Reliability Engineering · Ch 22 · Mike Ulrich · Google · 2016

What is this chapter about?

You've seen the headlines: some app "goes down" and stays down for hours. The strange part is that the published cause is always absurdly small — one line of config changed, a few extra machines added, a feature nobody uses switched on. Chapter 22 of Google's SRE book is about exactly that: how one small failure drags an entire system under, layer by layer.

An analogy first

Think about a blackout. On a hot afternoon a power line overloads and trips — but the current it was carrying doesn't vanish, it shifts onto the neighbouring lines. Those were already near capacity, so the extra share makes them trip too, and the current moves on to the next batch. Within minutes a whole city goes dark.

Here's the counterintuitive part: repairing the line that tripped first does not make the city light up again. Every air conditioner, fridge and lift in town is now stopped, and they would all restart at once the moment you close the switch — a surge far larger than normal, which trips the line straight back. The power industry has a whole procedure for this, called a black start: light one small area, let it settle, then light the next.

Why this is hard

Software collapses follow the same script. One server can't cope and dies; its work is automatically shared out among its peers; the peers were already busy, so the extra share makes them slower; being slow makes users hammer refresh, so the request count goes up, not down; the automated health checker notices a few machines aren't answering, declares them broken and kills them — leaving fewer machines, each busier than before.

See it? The result of every step becomes the cause of the next one. A failure isn't being pushed along by something outside; it grows on its own. That is the one thing worth remembering from this chapter.

What actually makes it grow

Three little engines push it along:

Work you can't finish piles up in a line. By the time a request's turn comes, the user has closed the page and left. The machine is running flat out computing answers nobody will collect.
Busy is expensive. When it's busy the machine holds more scratch memory, and clearing that memory takes effort — effort squeezed out of the actual work. Busier means slower, slower means busier.
Restarting has a price. A machine that just started remembers none of its usual answers, so it has to go ask the back end for every single one. For those first few minutes it demands far more from the back end than usual — you think you're firefighting, but you've thrown fuel on at the worst moment.

So what do you do?

In peacetime: stamp every piece of work with "after this moment it no longer matters" and throw away anything past it; never let the queue grow long; separate what matters from what doesn't, and drop the latter first when things get busy. In an incident: don't reach for more machines first (new ones need warming up and may well make things worse). The most effective move is usually the one nobody wants to make — cut the traffic off entirely, let the system catch its breath, then let it back in a small slice at a time. Exactly like restoring power to a city.

And the cost is real: cutting traffic means deliberately creating a stretch where nobody can use the thing at all, and plenty of teams can't bring themselves to press that button — which is how a thirty-minute outage becomes a thirty-hour one.

One thing to remember

A failure by itself isn't the scary part. The scary part is that failure breeds. Once the loop is turning, fixing the original cause achieves nothing — you have to stop it by hand, and then relight the system one district at a time, the way you'd restore power to a city.

Want the mechanisms, the numbers and the diagrams? → Switch to the deep read

In one sentence

A cascading failure isn't "one big outage" — it's a chain of small ones, each making the next more likely. The book's definition is a single line: a failure that grows over time as a result of positive feedback. The corollary is deeply counterintuitive: once the loop is turning, removing the original trigger does not bring the system back, because the load is no longer set by external traffic but by the system's own retries, queues, cold caches and health checks. So the chapter teaches two things: how to cut every segment of the loop at design time (queues, deadlines, dependency direction, startup path), and — when the loop is already turning — which segment each emergency action is actually breaking.

Glossary

Cascading failure — a failure in one place that makes failure elsewhere more likely, spreading and growing as it goes.
Positive feedback — a loop in which the effect reinforces the cause. Here: "failure → more load → more failure".
Resource exhaustion — running out of CPU, memory, threads, file descriptors or similar, so the server can no longer work properly.
Task / instance — one running copy (one process) of a service. A service usually consists of dozens to thousands of tasks.
Cluster manager — the system that schedules tasks onto machines and kills and recreates them when they look unhealthy. Borg at Google; Kubernetes is today's equivalent.
Health check — periodically asking a task "are you OK?". Two flavours: a process health check only asks "is the process alive?", while a service health check asks "can you still do useful work?". Conflating them is a trap this chapter warns about repeatedly.
Garbage collection (GC) — automatic reclamation of unused memory in languages like Java and Go. The reclaiming itself costs CPU.
Thread pool and queue — a server does work with a fixed number of threads; requests it can't get to yet wait in a queue.
File descriptor (FD) — the number the operating system assigns to each open file or network connection. There is a hard limit on how many you get.
Deadline vs timeout — a timeout says "I'll wait this long for you"; a deadline says "after this instant the work no longer matters". The chapter pushes hard for the latter.
Deadline propagation — passing the single absolute instant chosen at the top of the stack down through every layer, instead of each layer inventing its own timeout.
Cold cache — a cache that has just started and holds nothing yet, so nearly every request has to go to the back end. Back-end load spikes accordingly.
Load shedding / graceful degradation — deliberately dropping some requests when you can't cope, or returning a cheaper, lower-quality answer (the subject of the previous chapter).
Drain — deliberately moving traffic off an instance or a datacenter, usually for maintenance. One of the most common triggers of cascading failure.
Goodput — the portion of throughput that is both successful and still wanted by someone. Under overload it diverges sharply from raw throughput.

Where it sits

Chapter 22, by Mike Ulrich, sits in Part III ("Practices"). It is the second half of Chapter 21, "Handling Overload": the previous chapter is about how a single service defends itself (shedding, degradation, quotas, client throttling); this one is about what happens after that defence fails — how failure propagates between services, and how you rescue a system that has already collapsed. It leads into Chapter 23 on managing critical state, whose consensus systems are exactly the kind of "everybody depends on it" hub this chapter warns about. In the real world it maps onto every "X is down" headline: AWS Kinesis 2020, Roblox 2021, and every avalanche set off by a routine release or a routine capacity addition.

What problem the chapter solves

Start with the counterintuitive bit: cascading failures are almost never triggered by something bad. The trigger is usually routine housekeeping — a release, a capacity addition, a datacenter drain, or organic growth crossing an invisible line.

Do the arithmetic. Five tasks, each running at 80% CPU, which looks like 20% headroom. Now one dies (a rolling update, a bad machine — anything). Its load is spread across the remaining four, so each goes to 80% × 5 / 4 = 100% — one death and the survivors are all pinned at the ceiling. Latency climbs, clients time out and retry, request volume climbs again; health checks stop getting timely answers, the cluster manager declares two more "broken" and kills them; the two left standing would have to carry 80% × 5 / 2 = 200%.

And here is the property that matters: even if you repair the task that died first and put it back, it will be killed the instant it comes up into 200% of load. The system will not walk back to the left-hand picture on its own — what's driving it now isn't user traffic, it's a flood of retries, a backlog of queued work and a pile of empty caches.

Fig 1 · The load-redistribution death spiral: 80% utilisation isn't "20% of headroom", it's "we can survive exactly one death".

Two properties earn this its own chapter. It is non-linear — going from 50% to 80% utilisation degrades things smoothly, but past a certain line there is a cliff. And it is self-sustaining — removing the trigger is not enough; you have to break the loop deliberately. So the chapter's job is: keep the loop from ever spinning up, and stop it once it has.

The core content, properly explained

① Positive feedback: the only real protagonist here

The definition is short: a failure that grows over time as a result of positive feedback. The valuable words are "positive feedback", because they hand you a test you can carry into any design review:

Can the result of this failure become the cause of the next one?

If yes, there's a loop. The arithmetic above hides at least three: failure → load redistribution → more failure; slowness → retries → more load → more slowness; process killed → restarts with an empty cache → back-end request volume explodes → back end gets slower. An ordinary outage is pushed along from outside; a cascading failure grows by itself. That's also why it tends to erupt after a late-night routine release rather than during a known traffic peak — you prepared for the peak, you didn't prepare for the release.

② Resource exhaustion: four paths that set each other alight

The proximate cause is usually some resource running out. The chapter lists four main paths, and the insight is that they aren't four parallel paths but a web in which each ignites the others:

Not enough CPU: everything slows down together → more in-flight requests → longer queues → threads tied up → RPCs miss their deadlines → retries → back to step one.
Not enough memory: more in-flight requests naturally cost RAM → containers get OOM-killed; and even when they survive, GC runs more often — and GC costs CPU, so memory pressure translates directly into CPU pressure. Cache hit rates fall at the same time, forcing out yet more back-end calls.
Thread exhaustion: every thread is tied up in a slow request, so new requests error out. Worse, there is no thread left to answer the health check — so a task that is alive and merely busy gets treated as dead and killed by the cluster manager. Your automation just cut your own capacity.
File descriptor exhaustion: connections can't be established, which once again shows up as failing health checks.

Which resource runs out first usually doesn't matter; what matters is that any one of them running out drags the rest down with it. It's why the scene of a cascading failure always reads as a mess: OOMs, timeouts and health-check failures happening at once, with no obvious cause and effect.

Fig 2 · Resource exhaustion isn't four parallel paths but a web that ignites itself; the dashed outer ring is what turns a single-task failure into a cluster-wide one.

③ The queue: not a buffer, an amplifier of latency

Most thread-per-request servers put a queue in front of the thread pool. One intuition to kill first: if arrival rate and processing time are both steady, the queue should be empty. A queue only grows when arrivals exceed capacity — precisely the moment you least want it growing.

Its three costs are all real: it consumes memory (every queued request carries its own context), it pushes latency straight up (queuing time is pure waiting), and — the nastiest — it manufactures zombie requests: the ones at the head that have waited eight seconds belong to users who left long ago, yet you still dutifully compute their answers.

Fig 3 · Same queue, different pick order, opposite outcomes: under overload FIFO spends your CPU on the least valuable requests first.

The chapter's direction is keep queues small and reject early: with a short queue the server starts rejecting sooner, and failing fast is the healthier behaviour. A long queue absorbs bursts, but pays in inflated latency and heaps of zombie work. Some services go further and barely queue at all, preferring to fail so the caller can pick another instance.

For the how, the classic industry answer comes from Facebook: a variant of CoDel (controlled delay) that caps queuing time — if the queue hasn't been empty for the last N milliseconds (evidence of a standing queue), cap time in the queue at a small M milliseconds; if it has been empty recently, allow longer. In plain terms: allow brief bursts of queuing, never allow a queue that stands there permanently. Pair it with adaptive LIFO — FIFO in normal operation, switching to LIFO as soon as the queue starts building. (Sources below.)

④ Deadlines: stop doing work nobody is waiting for

The root of the queue problem is that nobody knows when a request stopped mattering. The chapter's cure is to treat deadlines as first-class, in three moves:

Check the deadline at several stages, especially the moment you pull a request off the queue. If it has expired, drop it and return an error without spending a single CPU cycle. This is the cheapest, fastest-acting fix in the chapter.
Propagate an absolute deadline instead of setting a timeout at each hop. "One second" at every layer looks restrained, but four layers stacked is a four-second budget — and the user left after the first second. The right way is to fix one absolute instant at the top and pass it down, each layer subtracting only a small allowance for network transit. A downstream service that sees "0.35s left" and knows it can't finish simply doesn't start.
Propagate cancellation too. When the caller gives up and doesn't say so, the callee keeps burning resources on an answer nobody wants — during an overload that zombie work can be a substantial share of your capacity.

Fig 4 · A timeout is every layer talking to itself; a deadline is one timetable shared by the whole chain. Only the latter stops the budget being multiplied by depth.

One trap deserves naming here: bimodal latency — the average will lie to you. Do the arithmetic. A frontend with 10 machines × 100 threads has 1000 concurrency slots; at 100 ms per request it comfortably carries 1000 QPS. Now a quarter of the back ends die, and requests routed to them sit until a 10 s timeout. Only 10% of traffic — 100 QPS — lands on that slow path, but each of those holds a thread for ten seconds: 100 × 10 = 1000, exactly enough to occupy every thread. Ten percent of the traffic has eaten one hundred percent of the concurrency. Meanwhile mean latency is 0.9 × 0.1 + 0.1 × 10 ≈ 1.1 s — the dashboard just says "a bit slow". Watch percentiles, not means — an echo of the yardstick from Chapter 1.

⑤ Slow startup and cold caches: healthy in steady state, unable to come back up

A server in its first few minutes is a different animal: connections not yet established, JIT not warm, classes not loaded, and worst of all an empty cache. In steady state a 90% hit rate means 10,000 QPS arriving turns into only 1,000 QPS at the back end — and that is exactly what the back end was capacity-planned for. Empty the cache and the hit rate goes to zero, so the back end faces the full 10,000 QPS: a clean 10×.

Fig 5 · Cold-cache amplification: the higher your steady-state hit rate, the bigger the multiplier on restart — 90% hits means a 10× surge.

That turns the two most instinctive emergency actions into dangerous ones. Restarting throws the cache away. Scaling out brings up instances that are also cold and must fill themselves from the back end — which is precisely the component already gasping. The remedies are overprovisioning, deliberate cache warming, and above all ramping slowly: bring back a small batch, let it warm, then bring back the next. AWS adding only a few hundred servers per hour while recovering Kinesis is this exact principle.

⑥ Always go downward in the stack

The chapter states this almost as a discipline: calls should always go downward — avoid intra-layer communication, and above all avoid cycles. Frontends in different datacenters proxying to each other, or A calling B while B calls back into A, look harmless in normal operation; the moment one side slows down you have a perfect positive-feedback ring.

The most easily overlooked cycle is monitoring that depends on the thing being monitored. If the telemetry, config or service discovery you rely on during an incident runs on the infrastructure that is currently on fire, you go blind exactly when you most need to see. Roblox's 73-hour outage in 2021 is the specimen: the observability stack they needed for triage depended on Consul, and Consul was the thing that had failed.

⑦ Triggers and testing: here's the list

The chapter's list of common triggers is uncomfortable reading, because they are nearly all good things: process updates and new releases, planned changes (drains, turndowns, maintenance), organic growth crossing a line, a shift in request profile (same QPS, more expensive), changes to resource limits, machine failures. So don't expect "not doing bad things" to save you — the design has to be un-collapsible.

On testing, one line matters most: load-test to failure, and then keep going. Testing only up to your target capacity is not testing — what you need to see is the behaviour past the point of failure, and above all whether the service recovers on its own once load drops back to normal (many don't; that's the definition of a cascading failure). Also test your biggest clients, whose retry behaviour is rarely like everyone else's, and test non-critical back ends by killing them and confirming the critical path really is unaffected — rather than assuming it is.

Key trade-offs

Table 1 · Queueing strategies: a short queue isn't conservatism, it's a defence

Strategy	Behaviour under overload	Cost	Where it fits
Barely any queue	Rejects the moment threads are full; caller retries another instance	No cushion at all — normal jitter produces errors	Online services with healthy caller retries and many instances
Small queue + early reject	Absorbs second-scale bursts, then fails fast	Callers must handle rejection properly or errors reach users	The default for most online services
Large queue	Swallows everything while latency climbs	Most dangerous: eats memory, breeds zombie requests, stretches the outage	Offline / batch work, and only with a backlog cap and sidelining
LIFO + queuing timeout (CoDel style)	FIFO normally; switches to LIFO on a standing queue and caps queuing time at milliseconds	Old requests may never be served; parameters need measuring per service	When requests can be reordered and goodput matters more than fairness

Table 2 · Four kinds of resource exhaustion: symptom, what it ignites, defence

Resource	Typical symptom	What it ignites	Defence
CPU	Everything slows together, queues grow	Missed deadlines → retries → more CPU	Shed load, size capacity by CPU (Ch 21), shorten queues
Memory	Containers OOM-killed; GC rate climbs	Becomes CPU pressure via GC; hit rate falls and floods the back end	Cap in-flight requests and queue length, not just heap size
Threads	New requests error; health checks go unanswered	Cluster manager declares it dead — your automation cuts your capacity	Reserve resources for health checks; separate service and process checks
File descriptors	Connections can't be established	Also surfaces as health-check failure, hard to tell from thread exhaustion	Reuse connections; aggregate huge client fleets behind proxies; alert on the limit

Table 3 · The seven field actions: which segment of the loop each one breaks

Action	How fast it works	Risk	When to use it
Add resources	Slow (minutes)	New instances are cold and may add load to the back end	When the bottleneck is genuinely capacity and caches don't matter
Stop health-check induced deaths	Fast	You also lose automatic removal of genuinely broken instances — last resort	When instances are "busy but alive" and being killed wrongly
Restart servers	Fast	Drops caches; if the root cause remains they die again on arrival	GC death spirals, deadlocks, piles of in-flight work with no deadline
Drop traffic (big red button)	Fastest and most complete	Deliberately creates total unavailability; recovery must be ramped slowly	The standard answer once the loop is self-sustaining
Enter degraded modes	Fast	Degraded paths rarely run, so they're often already broken	Read paths with a natural "cheap answer"
Eliminate batch load	Fast	Almost none — usually the best value action available	When deferrable offline work is mixed into the overload
Eliminate bad traffic	Medium	Requires identifying it first; mistakes hit legitimate users	Queries of death, misbehaving clients, attack traffic

Table 4 · Conflating the two health checks lets automation cut your capacity for you

Type	What it asks	Who consumes it	What conflating them does
Process health check	Is the process alive?	Cluster manager (Borg / Kubernetes) — unhealthy means kill and recreate	Treats "busy but alive" as dead, removing capacity exactly when it's scarcest
Service health check	Can it still serve well?	Load balancer — unhealthy means move traffic away	If every instance reports unhealthy at once there is nowhere for traffic to go

Why it matters / in real systems

A decade on, this chapter is still the skeleton of every serious postmortem, because it turns "avalanche" from an adjective into four executable pieces of engineering: cut the loop (deadlines, cancellation propagation, queue caps), stop retries multiplying (backoff, jitter, retry budgets), treat the startup path as first-class (cold caches, warming, slow ramps), and only depend downward (no cycles, monitoring independent of what it monitors). Asked in an interview how you prevent an avalanche, those four are the shape of a good answer — not "add machines and retry harder".

FIELD EVIDENCE · NOT THE BOOK'S CLAIMS — THE COMPANIES' OWN

Facebook published how it handles queues: drawing on bufferbloat research, it uses a variant of CoDel (controlled delay) to bound queuing time — if the queue has not been empty for the last N milliseconds, time spent in the queue is limited to M milliseconds; if it has been emptied recently, longer queuing is allowed — which "prevents a standing queue while still allowing short bursts of queuing". Paired with adaptive LIFO: FIFO normally, switching to LIFO once the queue starts building.Ben Maurer, "Fail at Scale: Reliability in the face of rapid change", ACM Queue 13(8), 2015 ↗
Amazon (AWS Kinesis, 25 Nov 2020): a textbook cascading failure whose trigger was a small capacity addition (started 2:44 am PST, finished 3:47 am). Every Kinesis front-end server connects to every other, each connection taking a dedicated thread, so the new capacity pushed the fleet past the maximum thread count allowed by the operating system configuration; cache construction failed, front ends could no longer route to back-end clusters, and Cognito, CloudWatch, Lambda and AutoScaling degraded with it. Recovery is the real lesson: with a front-end fleet of many thousands of servers, they could only bring them back at a few hundred per hour, reaching normal at 10:23 pm PST.AWS, "Summary of the Amazon Kinesis Event in the Northern Virginia (US-EAST-1) Region", 2020 ↗
Amazon puts the chapter's queue warning even more bluntly in the Builders' Library: overload and failures build backlogs a service cannot recover from in any reasonable amount of time, driven by high enqueue rates, expensive work or slow dependencies — and the answer is prioritisation, sidelining and back-pressure, not a bigger queue.D. Yanacek, "Avoiding insurmountable queue backlogs", AWS Builders' Library ↗
Roblox (from 28 Oct 2021, 73 hours): nearly every item in this chapter, in one incident. A newly enabled Consul streaming feature concentrated contention on a single Go channel under very high read and write load. Disabling it improved cluster health markedly — but after rebuilding the cluster from a snapshot taken hours before the outage, the system looked stable and then degraded again, which is field evidence for "removing the trigger does not restore you". Recovery was drawn out by cache warming, and triage was crippled because the telemetry itself depended on Consul.Roblox, "Roblox Return to Service 10/28-10/31 2021", 2022 ↗
Google Cloud (2 Jun 2019, incident 19009): a configuration change intended for a small number of servers in a single region was incorrectly applied to a much larger number across several neighbouring regions, causing them to stop using more than half of their available network capacity. Packet loss lasted between 3 hours 19 minutes and 4 hours 25 minutes, and took down Google's own G Suite and YouTube along the way. The trigger was routine maintenance automation — squarely in this chapter's "planned changes" bucket.Google Cloud Status Dashboard, Incident 19009, 2019 ↗

Common mistakes and open questions

"Fix the cause and it'll recover on its own." The most expensive misconception here. Once the loop is self-sustaining, the load is coming from retries, backlog and cold caches. Roblox degrading again after the snapshot rebuild is the public footnote.
Adding machines is instinctive and often wrong: new instances are cold, and the first thing they do is hammer the back end to fill themselves — the back end being the component already gasping.
Restarting cuts both ways: excellent against GC death spirals, deadlocks and piles of deadline-less in-flight work; pouring fuel on the fire for a cache-sensitive service. Identify the root cause before you reach for it.
Health checks that kill too eagerly: declaring "busy but alive" dead means automation removes capacity at the worst possible moment. The book permits turning health checks off temporarily, but calls it explicitly a last resort — you also lose the automatic removal of genuinely broken instances.
The big red button's cost is organisational, not technical: "cut everything, let it settle, ramp slowly" is often the fastest technical path, but nobody wants to take a service from 30% availability down to zero. Write that authority down in peacetime; don't expect it to be granted mid-incident.
Watching only mean latency: in a bimodal distribution, 10% of requests can eat 100% of threads while the mean moves from 0.1s to 1.1s. Watch p99 and concurrency-slot occupancy.
Testing only up to target capacity is not testing: push past failure and keep pushing, and specifically verify that the service climbs back once load returns to normal.
Dated in places: this was written in 2016 in Google's homogeneous RPC world. Since then service meshes have pushed deadline propagation, retry budgets and circuit breaking into the sidecar, making all of it far cheaper to adopt. But a new problem arrived too — autoscaling has itself become a source of positive feedback: latency rises, so you scale out; cold instances hammer the back end; the back end slows further; so you scale out again. The framework holds completely, but today's trigger list needs one more entry: your own automation.

Takeaways

① The definition is one line: a failure that grows over time as a result of positive feedback. The test is "can the result of this failure become the cause of the next one?"

② The counterintuitive corollary: removing the trigger is not enough to recover. Once the loop sustains itself, what drives the system is retries, backlog and cold caches, not user traffic.

③ 80% utilisation is not "20% of headroom": five tasks at 80% each go to 100% when one dies, and 200% when two more do. Past the line it's a cliff, not a slope.

④ The four exhaustion paths (CPU / memory / threads / file descriptors) ignite one another: memory pressure becomes CPU pressure via GC, and thread exhaustion becomes being killed by your own automation via the health check.

⑤ A queue is not a buffer, it's a latency amplifier — in steady state it should be empty. Keep it small and reject early; under overload FIFO serves the least valuable zombie requests first.

⑥ Deadlines are the cheapest cure: check for expiry when dequeuing; propagate an absolute deadline rather than a timeout per hop (four hops at 1s each is a 4s budget); propagate cancellation too.

⑦ The mean lies: 10% of requests taking 10s can occupy all 1000 threads while the average sits at about 1.1s. Use percentiles.

⑧ Cold caches are the hidden price of restarts and scale-outs: a 90% hit rate means an empty cache hits the back end 10×. Overprovision, warm caches, ramp slowly.

⑨ Depend downward only, never in cycles — and never let monitoring or service discovery depend on the thing that's burning, or you go blind when it matters most.

⑩ The triggers are nearly all "good things" (releases, capacity additions, drains, growth), so test past failure and verify self-recovery; of the seven field actions, dropping traffic and ramping back slowly is the standard answer once the loop sustains itself.