Scale From Zero To Millions Of Users

In one line

You think this chapter teaches you how to make a system big. What it actually teaches is an order. Load balancers, caches, CDNs, shards — those parts are documented everywhere. What this chapter contributes is arranging them into a build sequence with falling returns and rising irreversibility, where each step removes exactly one bottleneck. Get the order wrong and you pay permanent complexity for a problem you do not have yet.

Glossary

Where it sits

This is the first of the three chapters in Volume 1's Part I on method and fundamentals, and the only chapter in the book that designs no product. Ch 2 teaches you to get QPS, storage and bandwidth right to an order of magnitude on a whiteboard; Ch 3 teaches the four-step interview framework; Ch 1 first supplies the shared skeleton both of those operate on. The twelve case studies that follow — rate limiter, URL shortener, news feed, YouTube, Google Drive — are almost all additions to one layer of that skeleton.

The problem it solves

Ask an engineer "how many users can your system take" and the usual answer is a menu of names: we run Redis, Kafka, sharded databases. Names cannot answer why now, and why this one. What this chapter answers is a different question: which thing gives out next, and whether the money and complexity it costs are worth it.

What goes wrong otherwise? Too early: five thousand users and you already run microservices, sharded databases and multi-region, so eighty percent of the team's time goes into infrastructure rather than product — and sharding is essentially not refundable. Too late: you only think about read replicas once your one database is pegged at 90% CPU, and now every change has to be made under live production traffic, at maximum risk.

An anchor with real numbers: a content community with 1M daily actives, each opening the app 10 times a day and pulling 20 items per open. That is roughly 10M reads/day, about 116 QPS averaged out, and 350–600 QPS at peak if peak runs 3–5× the mean; writes are around one percent of reads. At that level one decent server plus one database genuinely still copes — what breaks first is not "volume" but one particular kind of request (the home-feed query joining three tables) and having no substitute. Which is exactly why the first move is usually a load balancer and a replica, not a shard.

The core, explained

First, see how many hops a request actually takes

The starting shape: one server running the web app, the database and the cache together. A user types a domain → DNS returns the IP → the browser connects directly → the server returns HTML or JSON. Look at this path first, because every later step is essentially inserting one more hop into it — a load balancer, a cache, a CDN, a queue. Miss the path and all you can do is memorise part names.

The first real separation is moving the database off the web server: the two want fundamentally different resources — web wants CPU and concurrent connections, databases want memory and disk I/O — so together they fight, and neither can scale on its own. On database choice the book's rule is plain: default to a relational database (RDBMS), and only consider NoSQL when you need very low latency, the data is unstructured, no complex queries are required, or the volume is enormous — never pick it because it sounds more modern.

Vertical vs horizontal: the turning point is "no substitute," not "not fast enough"

Vertical scaling means a beefier single box. Its upside is badly underrated: it is simple — the application barely changes and you face none of the distributed-systems tax. It has two problems: a hard ceiling, and no failover, no redundancy. The second is the decisive one. Plenty of teams believe they went horizontal because they were not fast enough; what actually forced them was not daring to run on one box.

Horizontal scaling is a pile of ordinary machines with a load balancer in front. The LB owns the public IP and is all the client knows; the web servers behind it have private IPs only and talk over the internal network, never exposed directly. Two payoffs: when one dies, the LB sends traffic to the rest; when traffic grows, you add machines to the pool.

Replication: the cheapest lever when reads dominate

Adding web machines is easy because they store nothing. Databases are not so lucky. The first move is replication: one primary takes every write, while several replicas hold copies and serve reads only.

It pays off because nearly every application reads far more than it writes — the community above runs about 100:1. Spread reads over N replicas and read capacity grows close to linearly, while the application changes in exactly one way: reads to replicas, writes to the primary. You also get durability and availability for free (a dead replica means reads go elsewhere; a dead primary means promoting a replica).

The honest cost: replication is usually asynchronous, so replicas lag. If the request right after a user posts happens to land on a replica that has not caught up, they will not see their own post. The usual remedy is to pin that user's reads to the primary for a short window after a write.

Cache and CDN: move data closer to whoever wants it

The cache tier is a separate in-memory store (Redis, Memcached) between web and database. The common pattern is cache-aside: check the cache, return on a hit; only on a miss do you query the database, write the result back, and return it.

One number explains why it works: reading from memory is microseconds, while a network round trip plus a database disk lookup is typically milliseconds — three orders of magnitude. At a 90% hit rate the read load reaching the database drops to one tenth: the community peaking at 600 QPS leaves only 60 QPS hitting the database.

The book's five considerations each correspond to a class of production incident. When to use it — only data that is read often and changed rarely is worth caching, and a cache is not durable storage; a restart empties it. Expiry (TTL) — too short and you may as well not have one, too long and the data goes stale. Consistency — cache and database are not updated in one transaction, so a window of disagreement is unavoidable. Single point of failure — with one cache server, losing it sends every read to the database in the same instant and usually kills it, so spread across several and leave headroom. Eviction — when it is full, who goes? LRU (least recently used) is the default, with LFU and FIFO as alternatives.

A CDN is the same idea on a different axis: push static content out to edge nodes worldwide so users fetch it nearby. Caching saves backend computation; a CDN saves intercontinental round trips — a US-to-Asia round trip is typically 150–200 ms, and an edge node cuts that to low tens of milliseconds. Its pitfalls are just as concrete: you pay per byte served, so cold content on a CDN is pure waste; TTLs must match how often content changes; and updates only propagate via an invalidation API or a versioned URL (?v=2).

A stateless web tier: the real switch behind "just add machines"

Saying "adding web machines is easy" quietly assumed something: a web server holding session data cannot be swapped out freely. The stateful version keeps user A's session in the memory of server 1, so every one of A's requests must be routed back to server 1 — a sticky session. The consequences chain: the LB's scheduling is nailed down, server 1 dying logs A out, and every scale event drops a batch of users. Autoscaling is simply off the table.

The fix takes one sentence but carries enormous weight: move sessions into a shared persistent store and let the web servers remember nothing. Now any box serves any request, and machines go from being assets with identities to disposable, freely replaceable resources. Every elastic capability that follows — autoscaling, multi-region, canary and blue-green releases — depends on this step. If you take one thing from this chapter, take this.

Multiple data centres and message queues

Multiple data centres: geoDNS (DNS that resolves one domain to different IPs by the user's location) sends users to the nearest site, cutting latency and letting you drain traffic away when a whole site fails. The book names three difficulties in passing, and each is hard: traffic redirection (geoDNS changes are subject to DNS caching, so they do not take effect the moment you flip the switch), data synchronisation (keeping sites consistent — the hardest of the three), and test and deployment.

Message queues: producers post tasks, workers consume them asynchronously, and the value is decoupling. The book's example is photo processing — the user uploads, the web server drops "crop, sharpen, blur" into the queue and returns success immediately, and workers grind through it behind the scenes. The real gain is that the two sides scale independently: add workers when the queue backs up, remove them when it empties, with no effect on the web tier.

Sharding: the last step, and the only one you cannot really undo

Only when the database outgrows a single machine does sharding arrive: split one big table horizontally by a sharding key — the book's example is literally user_id % 4. The sharding key decides whether load is even, which makes it the most consequential and least revisable decision here. Three new problems come with it:

• Resharding: growing from 4 shards to 8 means moving data while still serving traffic. With plain modulo, adding a machine reshuffles nearly everything; consistent hashing (the subject of Vol 1 Ch 5) exists to hold the movement down to 1/N.
• The celebrity problem (hot keys): a handful of huge accounts landing on the same shard melts it while the others idle; in the extreme you give a single hot user their own shard.
• Joins get hard: across databases, what was one SQL join becomes several queries stitched together in the application — or you denormalize and duplicate the fields that get read together into one table.

So the conclusion of this section is not "how to shard" but "finish everything cheaper first, then consider sharding."

Logging, metrics and automation

Past a certain size, invisible means unfixable. The book gives three levels of metrics: host level (CPU, memory, disk I/O), aggregated level (database tier performance, cache hit rate) and business level (daily actives, retention, revenue), plus centralised logging and automation. It is the step that turns everything above from "it runs" into "it keeps running."

Trade-offs and choices

Table 1 · The scaling ladder: which bottleneck each step removes, what it costs, whether you can walk it back

Table 2 · Cache / CDN / replica: all three keep an extra copy — what differs is what they store and what they save

Why it matters / in real systems

This chapter became the standard opening of system design interviews because it hands you a route you can recite. Asked "what if traffic grows ten times," the bad answer is a menu of names; the good one is: "look at which tier gives out first — if web CPU is saturated, add machines; if database reads are, add a cache and replicas; only when one machine can no longer hold the data do we discuss sharding." Being able to say, at each step, "here is where I am stuck, so here is what I add" wins most of the interview and most of the design review. And the route has a public record on both sides:

Pitfalls and disputes

• Treating the order as a checklist: the most common misreading is that all seven steps must happen. In the real world the overwhelming majority of products never reach sharding — Stack Overflow is the living proof. The route tells you what to do next, not what to do in total.
• Sharding too early: it is the one nearly irreversible step, and after it joins, transactions, global ordering and global uniqueness all get harder. The test is strict: confirm that vertical scaling, read replicas and caching are all fully exhausted first.
• Underrating statelessness: many teams treat it as "move the session somewhere else." It is in fact the sole precondition for autoscaling, multi-region and canary releases. Sticky sessions are debt, and it gets dearer the longer you carry it.
• "We added a cache, so it is fast now": caching lowers the average but stretches the tail — a miss now pays for two hops. Worse, when the cache tier dies the database absorbs ten times the read load in one instant. Always ask: if the cache vanished, would the database survive?
• The closing "split into microservices" deserves scepticism: the book offers it as a parting suggestion, whereas a decade of industry experience suggests microservices mainly solve an organisational scaling problem (letting teams ship independently) rather than a performance one; used as a performance tool they usually buy more network hops and harder debugging.
• Terminology and period limits: the book says master / slave, while the industry has largely moved to primary / replica — use the current terms in documents and interviews. Written in 2020, its examples are of the IaaS era; serverless and managed databases with automatic sharding have since cut the cost of several steps — but "each step removes exactly one bottleneck, ordered by return" has not aged.

Takeaways