BOOKS DEEP READ · SRE · CH 19
Site Reliability Engineering · Ch 19 · Google · 2016
You open a global site on your phone — mail, video, a cross-border shop — and the request lands on one particular machine. Why that one? Chapter 19 of Google's SRE book is about exactly that: how traffic gets handed out before a request ever reaches a datacenter. It is not "put a box in front of the building." It is a relay, with three legs.
Think of a global restaurant chain. Getting fed takes three steps: ① you ask for directions — "where's the nearest branch?"; ② you drive there, and the forks in the road decide which branch you actually walk into; ③ inside, a host seats you at a table. Each step is a different mechanism, and each one knows different things and reacts at a different speed — which is why none of them can replace the others.
You think you're asking head office. You're actually asking the fellow at the end of your street — your phone doesn't query the site directly, it delegates to a middleman. So head office knows where the middleman is standing, not where you are. He may be a hundred miles away, and you get sent to a far-off branch.
Worse, he writes the answer in a notebook and reads it back next time. So if head office wants to close a branch temporarily, it has to wait for every notebook to go stale — this leg is slow, too slow to use in an emergency.
One: have the middleman mention roughly where you live. When he asks, he adds "I'm asking on behalf of someone over on the east side," and head office can point better. The cost: his notebook now needs a separate page per neighbourhood, so it gets fat.
Two: make one street address valid in many cities at once. Everybody worldwide uses the same address and the road network delivers you to the closest branch. Nice: no directions needed, and you can pull a branch out simply by taking its sign down. Not nice: "closest" is measured by the road map, not by how long the drive takes — and if the roads get re-signed halfway, you may be dumped at the same-named branch in another city, and the order you just placed is gone.
Three: seat people by calculation, not by bookkeeping. There's more than one host at the door. If they keep a ledger of "who sits where," handing over to the next host breaks it. Instead, compute a fixed table from your name — whoever greets you computes the same table. No reconciliation between hosts, and losing one host changes nothing. There's a companion move too: don't route the food back past the host's desk. The kitchen carries it straight to your table, because the food going out is far heavier than the order coming in.
Coarse decisions go to the directions step (which region); fine decisions go to the door (which machine). The honest cost: computing seats instead of recording them saves the ledger, but when a branch really does open or close, a small share of people get recomputed — and the meal they were in the middle of is interrupted.
Handing out traffic is a three-leg relay: the directions leg (DNS) can only point coarsely and slowly at a region, the road network drops you into some datacenter, and the machine is actually chosen at the door — where the trick is to compute rather than remember, and to keep the return trip away from yourself.
Want the mechanisms, notation and diagrams? → Switch to the deep read
You think frontend load balancing is "a box in front of the datacenter." It is really a three-layer relay in which no single layer can carry the job alone: DNS can only do coarse, sluggish geographic steering; routing (anycast) can only deliver to whatever is "near" in topology terms; and the thing that actually pins a connection to one backend is the network load balancer at the VIP layer — whose core craft is replacing the connection table with share-nothing consistent hashing, and keeping return traffic off itself.
8.8.8.8). This is what the authoritative server sees — not the user.This is the first half of the load-balancing pair in Part III, Practices: Ch19 covers everything before traffic enters a datacenter (DNS, anycast, VIPs), Ch20 covers what happens once it is inside (subsetting, weighted round robin, lame duck). Next comes Ch21, Handling Overload — because however well you hand traffic out, there is always a moment when it cannot be handed out. The real-world counterpart is a whole class of systems: CDNs and global frontends, cloud providers' global load balancers, and that row of L4 boxes in front of your own racks.
The chapter opens by arguing that power isn't the answer: more datacenters and bigger machines do not add up to users being served well. Three reasons.
70–90 ms, transpacific about 100–160 ms, and a full TLS 1.2 handshake costs 2 round trips. Pick the wrong entry point and you have donated one or two hundred milliseconds before the page even starts loading.What happens if you don't solve it? Two symptoms: hot and cold — one site queueing badly enough to spike p99 while another idles at 30% CPU; and the more painful one, you can't drain — a site needs maintenance or has already failed, and there is no knob that moves traffic away within minutes, because the decision is scattered across DNS caches all over the world.
The plainest form: the authoritative server returns different addresses depending on where the query came from, or one name carries several A records for round robin. Cheap, universal, no data-plane cost. But the chapter lays out its three hard walls very clearly.
First, you don't see the user, you see the resolver. The authoritative server almost never talks to the user; a recursive resolver sits in between. You know the resolver's address, not how many users are behind it nor how far away they are. The patch is the EDNS0 client-subnet extension (later standardised as RFC 7871, 2016): the resolver carries a prefix of the user's IP (say a /24) in the query so the authoritative server can decide per user rather than per resolver. The costs are real, and there are three: it only works with resolvers that implement it; one answer becomes one answer per subnet, which fragments the resolver's cache; and it hands over part of the user's address — an acknowledged privacy trade.
Second, you don't control caching. TTL is a suggestion. Resolvers cache, some raise TTLs they consider too short, and clients and application layers cache on top. So when you want a site out, what you are waiting on is the longest-lived copy, not the TTL you configured. Dropping to tens of seconds improves reaction time; the price is an extra resolution round trip every time it expires (straight onto the user's time to first byte) and proportionally more queries.
Third, you don't control how the client picks. Given several records, a client typically takes the first, but may sort them itself or try them in parallel; and the reply has a size budget (the classic 512-byte limit for UDP, which takes EDNS0 to lift), so only a handful of addresses fit.
Hence the chapter's restrained conclusion: use DNS for coarse geographic steering and do not expect it to balance anything — it cannot see live load, and it cannot take back an answer it has already given out.
Announce the same VIP from several sites over BGP and internet routing delivers each user's packets to one of them. The upside is immediate: no per-region DNS answers, no hostage to caching, and when a site fails you simply withdraw the announcement — routing converges elsewhere in seconds to minutes. That is the most useful emergency switch at this layer.
The trade-off: "near" in BGP's eyes means near in path attributes (hop count, policy preference), not near in latency, so you can land on a site that is the long way round. The harder one is connection stability: if routing changes while a TCP connection is alive, later packets arrive at a different site which holds no state for it, and the user gets a reset. In practice two things soften this — making the load balancers inside a site stateless, identical and unanimous (next section), and putting a global proxy at the frontend so any site can forward a request to the backend it belongs to, at which point "changed site" no longer has to mean "connection void."
A VIP belongs to no interface; a set of network load balancers announce it jointly, upstream routers spray packets across any of them by ECMP, and each one decides which backend this packet belongs to. There is exactly one difficulty, and it is hard: TCP is stateful, so every packet of a connection must go to the same backend or the connection dies instantly. Two ways:
Google's Maglev stacks both: consistent hashing as the foundation (stateless, recomputable at any time) with the connection table only as a layer on top — to protect already-established connections when the backend set really does change.
Why must it be consistent hashing; why won't plain hash mod N do? Do the arithmetic. Go from 10 backends to 11 and nearly every result changes: only about 1/11 ≈ 9% of flows happen to land where they were, so roughly 91% of connections are broken. Consistent hashing inverts that number: about 1/11 of flows have to move and the rest stay put. The cost is that it only guarantees that everyone computes the same thing, not that load comes out even — skew has to be flattened with virtual nodes (or a pre-generated lookup table, Maglev-style).
Having chosen a backend you still have to deliver the packet. Two roads, with very different costs.
NAT: rewrite the destination address. The cost is that the return trip must come back through the load balancer (otherwise the client sees the wrong source address), so the balancer carries both directions. And the downstream direction dwarfs the upstream — a request is a few hundred bytes, a response tens of KB to a few MB, commonly a 10x difference or more — so the balancer becomes the bandwidth bottleneck, and it must hold state per flow.
Encapsulation plus DSR: wrap the original packet whole inside a new IP packet (GRE or IPIP) addressed to the backend; the backend decapsulates, still sees the original destination (the VIP is configured on its loopback), and sends the response straight back to the client, never touching the balancer. The balancer now handles only the ingress direction, so its capacity is measured in inbound packet rate rather than total bandwidth — far cheaper to scale out.
Spell out the costs: the outer header eats MTU (from 20 bytes for IPIP, 24 for GRE), and once path MTU discovery is broken by some middlebox the symptom is "small requests fine, large requests hang" — among the nastiest failures to diagnose. The backend must have the VIP configured and must not answer ARP for it. And the balancer never sees responses, so observability has to be built elsewhere.
Table 1 · The three steering layers: what each can do, and what it costs
| Mechanism | Granularity | Time to drain | Sees real load? | Main cost |
|---|---|---|---|---|
| DNS | Site / region | Bounded by the longest cache: minutes to hours | No | Caching out of your hands; you see resolvers, not users; client choice out of your hands |
| Anycast + BGP | Site | Seconds to minutes after withdrawing | No (routing only) | "Near" is not low-latency; a mid-flow reroute can reset connections |
| VIP + consistent hashing | One backend / one connection | Immediate, on changing the backend set | Yes (health checks, weights) | L4 only; a few connections still suffer when the backend set changes |
Table 2 · Pinning a flow to a backend: connection table vs consistent hashing
| Connection tracking table | Consistent hashing | |
|---|---|---|
| State | One entry per flow; millions of flows is GB-scale | Stateless, recomputable any time |
| LB restart / ECMP re-spray | Table no longer matches, connections die | Unaffected |
| Backends 10 to 11 | Existing flows stay valid (this is its value) | About 1/11 of flows remap |
| Even load? | Naturally even | Needs virtual nodes / a lookup table |
| Role | An optimisation above consistent hashing | The foundation |
Table 3 · Delivering the packet: NAT vs encapsulation plus DSR
| NAT | Encapsulation + DSR | |
|---|---|---|
| Return path | Must transit the LB | Backend answers the client directly |
| Bandwidth on the LB | Ingress + egress (egress often 10x) | Ingress only |
| State | Required per flow | Can be fully stateless |
| MTU | Unchanged | Outer header eats 20–24 bytes; mind PMTUD |
| Observability | LB sees responses and timings | LB sees no responses; build it elsewhere |
| Fits | Small scale, or when you need L7 | Large-scale L4 frontends (Maglev and Katran both) |
In one line: the three layers do not replace each other, they stack — each step down is finer-grained and faster to react, but sees less. If you may keep only one, keep the VIP layer: it is the only one that actually knows whether a backend is alive.
The value of this chapter is that it splits the single phrase "load balancing" into three separate questions: which region the user is sent to (DNS), which site the packets enter (routing), which machine the connection is pinned to (the VIP layer). Different components answer each one, and conflating them is how people end up concluding that lowering a TTL is load balancing. The three questions most often pressed in interviews are exactly these three mechanisms: why DNS cannot balance load, why consistent hashing beats modulo, and why DSR changes the balancer's capacity unit from bandwidth to packet rate.
1. Frontend load balancing is not a box, it is a three-leg relay: DNS picks the region, anycast + BGP picks the site, the VIP layer picks the machine. The opening argument is that power isn't the answer — capacity isn't in one place, distance converts into latency (a cross-ocean round trip of 70–160 ms), and requests aren't equal.
2. DNS's three hard walls: you see the recursive resolver, not the user (patched by EDNS0 client subnet, at the cost of cache fragmentation and privacy); caching and TTL are out of your control; and so is how the client picks. It is distribution, not balancing.
3. Anycast's gift is draining by withdrawing an announcement (seconds to minutes); its cost is that BGP's "near" isn't low latency, and a mid-flow reroute breaks connections.
4. The VIP layer's one difficulty: every packet of a connection must land on the same backend. A connection table is state (GB-scale at millions of flows, and mismatched after a restart or an ECMP re-spray); consistent hashing is stateless and recomputable.
5. The numbers: going from 10 to 11 backends, modulo hashing breaks ~91% of connections while consistent hashing moves ~9% — but the latter only guarantees agreement, so evenness still needs virtual nodes. Maglev = consistent hashing as foundation, connection table as optimisation.
6. Two ways to deliver: NAT makes the balancer carry both directions (egress often 10x ingress); encapsulation + DSR lets the backend answer the client and the balancer carry ingress only — paid for in MTU, ARP configuration and blindness to responses.
7. In the field: Google Cloud's global LB is a single anycast VIP + Maglev + consistent hashing, and Meta's Katran independently chose the same combination.