LinkedIn: Being Mindful of Members

In one sentence

You think "minimize the blast radius" means making the experiment smaller — kill fewer machines, route less traffic. This chapter changes the unit: blast radius is measured in people, not machines. LinkedIn has hundreds of millions of members, and one of those sessions may be an interview invitation; so the real defect in "kill a machine" is not that it is risky but that you have no say in who gets hurt. The engineering answer is to push injection granularity down to a single request: the failure instruction propagates down the call graph with the request (LinkedOut), turning the affected set from "whoever happened to be routed there" into an explicit input of the experiment — then trade information up a ladder (your own session → employees → a targeted cohort → cluster level), with every rung wired to an abort that runs faster than the harm.

Glossary

Where it sits

Chapter 7, the fourth entry in Part II ("the principles in practice"), told from LinkedIn. The previous three cover how to run one, how big, and which one first: Ch4 Slack scripting experiments for legacy systems, Ch5 Google scaling DiRT to company size, Ch6 Microsoft solving prioritization. This one covers on whom — turning Ch3's fifth principle, "minimize blast radius," from a slogan into a mechanism. Ahead, Ch8 (Capital One) is the same problem under heavy regulation, and Ch16 (continuous verification) automates this chapter's ramping and verdicts. In the real world it speaks to any team whose tooling can already inject failure but who dare not press the button in production.

The problem this chapter attacks

Start with the constraint. Around the time this book was written, LinkedIn's publicly stated membership was in the 600–700 million range, and what makes the product unusual is how unevenly weighted a single session is: in the same minute, somebody is idly scrolling the feed and somebody else is pressing "Submit application" or answering a recruiter. A hiccup for the first costs nothing; one failure for the second may genuinely change something.

Now look at the default move. Killing 1 instance out of 1,000 affects roughly 0.1% of requests. Counted in machines, that is "small." Counted in people, two more questions are owed: who are those 0.1%, and what were they doing? Killing an instance does not answer either — the answer is decided by whatever the load balancer was doing that second.

Worse is the amplification sticky routing performs. If routing hashes on member, "1% of requests" is really "100% of the requests of 1% of members." One person failing a hundred times and a hundred people failing once each are the same number on an availability dashboard, and one is tolerable while the other is an incident.

What happens if you don't solve it? Two degenerations: you never dare — the tool and the permissions exist but nobody presses the button, so chaos stays in a staging environment that has neither real load nor real dependencies; or you force it and get shut down — one experiment becomes a real incident and chaos is unmentionable for six months. The restraint in this chapter exists precisely to keep the long-term license to experiment.

So the question is: how do you turn "who gets affected" from a bet into a parameter?

The core content, properly

1. Blast radius is measured in people, not machines

The one thing to take away is a change of units: stop asking "how many hosts, what percentage," and ask how many people, which people, what were they doing, for how long, and is it reversible.

The last three get dropped most often. For how long decides whether a person hits one retry or five minutes of blank screen. Reversible or not splits "a module is missing from the page" from "the application submission was lost, the message half-sent" — and no percentage is small enough to justify experimenting on the second kind. That is also why shaking a read path and shaking a write path are not the same class of risk here.

Once the units change, experiment design points somewhere else: the goal is not to shrink the scale but to make the affected set something you can write down. Killing instances cannot do that — its affected set is only knowable at runtime.

2. The mechanism: the failure instruction travels with the request

LinkedIn's framework is LinkedOut, and one sentence places it: the injection point is a request filter in the Rest.li stack, and whether this hop should fail is decided by context handed down with the request (the Invocation Context).

It injects three modes: error (throw an exception, mimicking an unavailable resource), delay (add latency before forwarding) and timeout (expire after a set period). Triggering comes in two forms: target a population through LiX, or put the instruction in a cookie — which makes the granularity as fine as "this one page load of mine."

Why this is not easy: a page fans out to dozens of downstreams, and the failure must land only on "this member's this request on its hop into Service X" — not leaking to other members, not leaking to Service X's other callers. That requires the framework to carry the context intact on every hop. Flip it around and you have the implicit cost: paths that never joined the propagation scheme cannot be injected (third-party SDKs, offline and asynchronous jobs, consumers behind a message queue).

What you buy is an order-of-magnitude shift in blast radius: from "every in-flight request of every member on that host for the length of the restart window" down to "one person's one click."

3. The ramp ladder: shake yourself, then a small crowd

Once injection is precise to one request, the cheapest subject is the engineer. LinkedOut ships a browser extension for exactly that: an engineer flips a switch in their own session, makes one downstream on the page "break," and looks at whether the page still works. Blast radius: one consenting person.

One rung up is automation: walk every downstream of a page, break each in turn, and check for graceful degradation. These experiments can run fully automatically and often because the only account affected is the one that launched them — their marginal harm is zero, so they can be used as tests.

The whole point of the ladder is to squeeze the most defects out of the cheapest rung: missing timeouts, missing fallbacks, a whole page going blank when one module dies — all of that surfaces on rung one, with no member involved. What actually justifies a ramp is only the class of problem that needs real traffic to be visible.

And the honest ceiling: some failures only exist at scale — connection-pool saturation, retry storms, mass cache expiry, a downstream hammered by every caller at once. Single-request injection cannot see them; those need a ramp or a cluster-level experiment.

4. Automatic abort: the guardrail must outrun the harm

Once real members are in scope, "are we hurting anyone?" cannot hang on a human watching a screen. Three points:

• Pick metrics members can feel: message send success, page render success, application submit success — not CPU and memory. "CPU is fine while users are failing" is exactly the blind spot this chapter fears.
• The verdict must be visible within the rung's own time scale. If harm takes ten minutes to show in a metric, the rung must run a full ten minutes before you can judge — and harm accumulates throughout. The conclusion is not "tolerate ten minutes," it is find a faster metric, or shrink this rung's population.
• The abort must fire by itself, in seconds. The injection switch has to be killable without a deploy or a restart; a human approval workflow is not an abort mechanism.

A criterion you can take straight to a review: an experiment with no automatic abort condition should not run on real people. That is not a moral posture, it is cost arithmetic — the latency of "human notices plus human responds," multiplied by the ramped population, is the harm you prepay for the experiment.

Key trade-offs and choices

The first table makes the ladder concrete: what each rung buys, and what it costs.

Most defects die on rung 1 — that is the ladder's real return, and the precondition for daring to do any of this in production.

The second table cuts the same question differently: what each harm-limiting technique costs you in fidelity.

Concretely:

• Just starting, afraid to press the button: don't start by killing machines. Get injection down to request level, then give engineers a "shake me" entry point — zero harm, and it catches most of the missing timeouts and missing fallbacks.
• Request-level ability in place, ready to ramp: build the automatic abort first, then ramp — never the other order. Pick the percentage not by what "looks small" but by how fast the harm becomes visible in a metric.
• Systems with sticky routing: target by member rather than hitting requests at random, and cap how long one member can be hit — otherwise "1% of requests" becomes "1% of people, fully broken."
• Money and irreversible writes: shake read paths and retryable writes only; for irreversible writes, take Ch8's Capital One route — rehearse outside production, gather the evidence, then talk about production.

Why it matters / in real systems

Three sentences that can change a design review. One: blast radius is measured in people, not machines. Two: "only 1% of traffic" means "1% of people, continuously" under sticky routing. Three: an experiment with no automatic abort condition has no business running on real people.

Asked in an interview how you do chaos engineering in production: "we kill random machines at night" is one level; "injection is request-level, experiments default to the initiator's own session, ramps go through a targeted cohort wired to an automatic abort on member-visible metrics" is another — the second shows you know the hard part is who gets affected, not whether you can inject.

Pitfalls and disputes

• Counting blast radius in machines or traffic percent. The same 0.1% landing on feed scrollers and landing on people submitting applications are two different events.
• Assuming "random" means "fair." Random injection plus sticky routing means a small group absorbs all of the harm.
• Ramping first, guardrails later. Automatic abort is not an optimization; it is the precondition for the ramp.
• Treating "shake yourself" as the whole program. It is cheap and productive, but structurally blind to pool saturation and retry storms, which only appear at scale.
• Treating ethics as a PR problem. This chapter treats it as a design constraint with concrete criteria: the affected set must be enumerable, the harm reversible, and every experiment explainable to a non-engineer.
• Dispute: request-level injection has limited fidelity. It injects "this one call failed," not "the service is actually down"; a real outage hits every caller at once and triggers retry storms and pool saturation. So it is the everyday, high-frequency rung and does not replace cluster-level experiments — the two divide labour rather than substitute for each other.
• Period limits and what came after. LinkedOut rests on in-house Rest.li, LiX and IC, which is a high bar to build; today the same mechanism is largely something you install — service meshes such as Istio natively support HTTP-level abort and delay fault injection, OpenTelemetry baggage provides cross-service context propagation, and feature-flag platforms provide targeted ramps. What has not changed is the unit of measure: ask who gets affected first.

Takeaways