Handling Overload: better to refuse some work than collapse under all of it

In one sentence

Overload was never about whether you can take it — it is about what happens in the moment you can't. This chapter's claim is mildly counterintuitive: a healthy service must be able to actively refuse some requests, because accepting everything indiscriminately does not end in "everyone is a bit slower," it ends with goodput collapsing to near zero. Around that claim it gives four separately implementable moves: measure capacity in resources rather than QPS; let the server degrade and shed; classify requests by criticality in advance; and make clients throttle themselves and put three gates on retries.

Glossary

Where it sits

This chapter comes from Part III ("Practices") of the SRE book, written by Alejandro Forero Cuervo. It follows directly on the two load-balancing chapters — those explain how to spread load evenly so you avoid overload; this one covers what to do when you are overloaded anyway — and leads into Chapter 22, "Addressing Cascading Failures." This chapter is a single service defending itself; the next one is what the whole system looks like after that defense fails. In the real world it maps onto every flash sale, every viral event, every avalanche triggered by one slow dependency.

What problem is it solving?

Start with the counterintuitive part: a service's genuinely dangerous window is not "slow" — it is the thirty seconds after slow.

Watch what happens with no defenses. A service rated for roughly 10k QPS gets pushed to 15k. Requests queue; as the queue grows, response time goes from 50 ms to seconds. In-flight requests pile up in memory, GC pressure rises and eats CPU in turn. Clients set a 1-second timeout, don't get an answer, and retry — so real demand of 15k arrives as 30k, then 45k. And while those retries land, the queue is still faithfully processing requests that timed out long ago and that nobody is waiting for. At that point CPU is pinned, every cycle is spent on results no one wants, and goodput is near zero.

That is not "not fast enough." It is structural collapse: load has started feeding itself. So the problem the chapter takes on is — when load exceeds capacity, how do you make output flatten at the capacity line instead of falling off a cliff? It splits the answer in two: the server must be able to do less on purpose, and the client must be willing to send less on purpose. You need both, because a rejected request that immediately comes straight back still costs the server CPU to reject.

The core, properly explained

Throw out QPS: measure capacity in resources

The usual way to state capacity is "this service handles X QPS." The chapter says flatly that this is a bad ruler. Request costs can differ by one or two orders of magnitude — a primary-key read of 3 rows and an aggregation that scans 500 shards both count as "1 query" but consume nothing alike. Worse, the traffic mix drifts as features launch and the customer base changes, so the ceiling you measured yesterday no longer holds today.

Google's approach is to define capacity directly in resource consumption, and in the overwhelming majority of cases that means watching CPU. Two reasons. First, in a garbage-collected runtime, memory pressure naturally translates into CPU consumption (tight memory → more frequent GC → more CPU burned), so watching CPU watches memory for free. Second, the remaining resources can be provisioned generously enough that they are very unlikely to run out before CPU does. CPU also has one crucial property — it is a compressible resource. Running short only makes requests slower; it does not kill the process outright the way exhausting memory does. That is precisely the window a shedding mechanism needs in order to react.

The concrete signal is executor load average: an exponentially decayed average of the number of active threads in the process, compared against the number of available CPUs, giving a continuous reading of "how full am I right now?"

Graceful degradation and shedding: two ways of doing less

The server has two roads under overload, and they are qualitatively different. Graceful degradation makes the answer cheaper: search only a small in-memory slice of the index instead of the full on-disk one, swap ranking for a simpler algorithm, fall back from personalized recommendations to a static popularity list. The request is still served, just one notch worse. Load shedding declines to serve at all: return an error (usually 503 over HTTP) and keep the resources for the requests you can still serve properly.

On degradation the book offers a notably restrained warning, worth memorizing on its own: the degraded path is almost never executed, so it is very likely already broken. On the day you finally switch to it, you discover that code drifted away from the main path months ago, or does not run at all. The counter is: keep the number of modes small, keep the logic simple, and actually exercise it on a schedule.

Per-customer quotas and client-side adaptive throttling

A shared service's capacity has to be divided among many customers, so you start with per-customer quotas — allocated as a resource rate (CPU-seconds per second, say) rather than as QPS. The quotas usually deliberately sum to more than total capacity, betting that customers won't all max out at once; when they do, the ones over quota get rejected first.

But there is a problem you cannot design around: rejecting isn't free. One furiously retrying customer can consume the server's entire CPU budget generating rejection responses. So the chapter's prettiest move lands on the client: adaptive throttling. Each client task keeps two numbers over a sliding 2-minute window: requests (how many the application layer issued) and accepts (how many the backend actually took). Normally the two track each other closely; once the backend starts rejecting, accepts falls behind. The client then drops a fraction of requests locally, probabilistically, without ever putting them on the wire:

In plain words: when what you send exceeds what gets accepted by too much, refuse that excess proportion yourself first. K is typically 2 — "I may send twice what the backend accepts." Smaller K is more aggressive and self-limits earlier; larger K is more permissive. The + 1 in the denominator merely avoids dividing by zero.

Two design details are the real substance. First, the probability never reaches 1: the client always leaks a trickle of requests through, which is its only way of detecting whether the backend has recovered. When accepts climbs back, the throttle releases itself — nobody has to clear it by hand. Second, locally rejected requests still count toward requests, so the client cannot fool itself into believing the pressure has gone away.

Criticality: decide what to cut first, in advance

Shedding means dropping some requests — but which? The chapter's answer: don't improvise in the middle of an overload; classify requests up front. Google's RPC system has four levels, highest to lowest:

• CRITICAL_PLUS — failure causes serious, user-visible impact.
• CRITICAL — the default for requests issued by production jobs; user-visible impact, but less severe. Capacity planning must cover the sum of these top two levels.
• SHEDDABLE_PLUS — partial unavailability is acceptable; the default for batch jobs, which can retry minutes or hours later.
• SHEDDABLE — frequent partial unavailability and occasional full unavailability are both acceptable.

Two mechanisms make it genuinely usable. First, criticality propagates down the call chain automatically: a service that receives a CRITICAL request issues its own downstream requests as CRITICAL by default — no layer has to guess. Second, quotas are accounted per criticality: a customer's CRITICAL quota and its SHEDDABLE quota are separate books, so low-value traffic cannot eat the high-value allowance. When shedding, a task always cuts from the lowest criticality upward.

Retries: the easiest way to finish yourself off

The most dangerous positive feedback loop under overload comes from retries. The chapter offers one piece of arithmetic worth internalizing: three attempts per layer, stacked three layers deep, means the bottom layer receives 3³ = 27× the traffic — and that bottom layer was already failing because it couldn't cope. You think retries are raising your success rate; in fact you are pouring fuel on a machine that is already on fire.

Three gates push it back down:

1. A per-request attempt limit. At most 3 attempts for the same request; once spent, let the failure propagate up rather than grinding forever in this layer.
2. A per-client retry budget. The client continuously tracks the ratio of retries to requests and only retries while that ratio is below 10%. This is the harshest and the most useful of the three — it caps request amplification for the whole system at roughly 1.1×, so however badly the downstream is doing, the flood cannot rise.
3. Let "don't retry" travel upward. When a backend rejects because it is overloaded, it returns an explicit "overloaded; don't retry" marker rather than a generic error. An upstream that sees it stops retrying and passes the failure on. This is exactly what cancels the 27× multiplication — otherwise every layer independently, and helpfully, tries again.

There is one more rule of judgement: whether the overload is local or widespread decides whether retrying elsewhere is worth it. If only a few tasks are overloaded, immediately retrying against a healthy task is right. If a whole datacenter is broadly overloaded, retrying just relocates the problem — the correct move is to let the error bubble all the way up to the caller and let it decide between degrading and giving up.

Don't forget that connections are load too

One last thing that is easy to miss: merely maintaining connections costs money. Health checks, keepalives, connection setup and teardown can outweigh the actual request handling once the client count gets large. The classic case is a batch job that spins up thousands of workers, each connecting to the same backend — request volume is modest, but connection overhead flattens the backend first. The fix is to funnel such clients through a small set of proxy tasks, dropping the backend-facing connection count from "number of workers" to "number of proxies."

Key trade-offs

Table 1 · Which signal tells you "I am overloaded"

Table 2 · The server's four possible reactions to overload

Table 3 · What each of the three retry gates actually stops

Why it matters / in real systems

A decade on, this chapter is still the common basis for rate-limiting, circuit-breaking and degradation designs, because it decomposes overload handling into four separately implementable moves: what to measure, how the server does less, how requests get ranked, and how the client sends less. The adaptive concurrency filter you see in Envoy/Istio, the breakers and fallbacks in Hystrix/Sentinel, the retry token buckets in every vendor SDK — all of them are re-implementations of this material. The good answer to "what do you do when a service is overloaded?" in an interview is these four layers, not "add machines."

Common mistakes and open arguments

• Treating shedding as free. Rejecting a request still means parsing, authenticating and logging it. If the cost of one rejection approaches the cost of serving, shedding won't save you — the answer is to push rejection outward (load balancer, edge proxy). The earlier you refuse, the cheaper it is.
• CPU is not a universal signal. For IO-bound services bottlenecked on a downstream database, CPU sits idle while the system is long past overloaded. Netflix's 2024 switch to latency and concurrency exists precisely for this.
• Client throttling assumes cooperative clients. Adaptive throttling is a product of Google's homogeneous internal world, where every service shares one RPC framework. Facing clients you don't control — a public API, an old app version on someone's phone — you must enforce server-side and cannot rely on good behavior.
• An unshared retry budget is no budget. Each client honors its own 10%, but every layer of the call chain is "another client," so amplification still multiplies. That is exactly why the third gate exists; don't implement only the first two.
• Too many degradation modes. Every extra mode is another code path that almost never runs. The book is unambiguous: keep them few, keep them simple, and rehearse them.
• "Autoscaling solves overload." Scaling is minutes; spikes are seconds. New instances also need warm-up, config pulls and cache fills, which can add downstream pressure at the worst possible moment. Scaling and shedding are complements, not substitutes.
• Its period shows. Written in 2016, the chapter describes Google's homogeneous internal RPC world. Since then service meshes have pushed these capabilities down into the sidecar, and adaptive concurrency control (deriving limits from latency, borrowing from TCP congestion control) has become the mainstream complement. The framework has aged well; the two implementation details — CPU as the signal, and an in-house RPC framework to propagate criticality — need substitutes in a heterogeneous environment.

Takeaways