BOOKS DEEP-READ · SYSTEM DESIGN INTERVIEW VOL 1 · CH 10
System Design Interview Vol 1 · Ch 10 · Alex Xu · 2020
The line that pops onto your lock screen — "your parcel is out for delivery", "someone replied to you" — is the visible end of a whole notification system. Chapter 10 asks you to build one. It sounds like a small job. It is in fact the book's cleanest lesson in a hard truth: you are accountable for something whose most important step is not yours to control.
Think of the system as a mail depot. You write the letters, but the last mile must be handed to three outside couriers: one for iPhones (Apple's), one for Android phones (Google's), one for text messages (the carriers). The catch is that each of them follows only its own house rules.
Apple's courier says: if nobody is home, I keep only the most recent letter and throw the rest away. Google's says: I will hold at most 100 letters per door, and if a 101st arrives I discard the lot; anything undelivered for four weeks I discard too. Neither gives you a signed receipt — only "I have taken the letter".
The obvious approach: as soon as the order goes through, your code knocks on the courier's door and waits for an answer before moving on. Three problems, each worse than the last. The courier takes an extra second and your checkout endpoint takes an extra second too. The one machine doing the errands dies and every pending notification evaporates. Sale day arrives, volume goes up fiftyfold, and there is nowhere to add hands.
The fix is a conveyor belt. The business side just drops "what to send, to whom" onto the belt and walks away — a matter of milliseconds. At the far end stands a row of workers who pick items off and do the running around. However slow the courier is, that slowness never reaches your main flow.
The crucial detail is that there is one belt per courier: one for Apple, one for Google, one for SMS, one for email. If the SMS courier jams today, only that one belt backs up while the other three keep moving; if SMS falls behind, you add workers to that belt alone. One blocked lane must not block the other three — that is the most valuable sentence in the chapter.
A belt alone is not enough. The switch: before sending, check whether this person wants this kind of notification at all. The serial number: stamp every letter with a unique number and have the worker check "have I sent this one already?" before running — because failed errands get retried, and retries always produce duplicates. The cap: at most N letters per person per day. The third is the one people skip, and its cost is the worst of the three: annoy someone into switching notifications off and you can no longer reach them at all — not even with "payment successful" — and that door usually stays shut.
The honest cost: once there is a belt, notifications are no longer instant — usually a few hundred milliseconds to a few seconds, and minutes when a lane backs up. And you can only promise "delivered at least once"; keeping duplicates out is on you, via the serial number.
A notification system is hard not because sending is hard, but because the last mile belongs to someone else: third-party channels guarantee neither delivery nor exactly-one delivery. All you truly control is your own side — a belt that separates business logic from delivery, a separate lane per channel, a serial number to absorb duplicates, and not annoying users into switching you off.
Want the architecture, the delivery semantics and the diagrams? → Switch to the deep read
This chapter looks like a lesson in pushing a message to a phone. It is really a lesson in building reliability on top of a channel you do not control: the last hop belongs to APNs, FCM or an SMS gateway, none of which promise delivery or non-duplication. So every engineering lever moves back onto your side of the line — a message queue to decouple production from delivery, one queue per channel to contain failure, an idempotency key to absorb the duplicates that retries inevitably create, and preference checks plus rate caps to defend the one irreversible failure: a user who switches notifications off for good.
This is the seventh case study in Part II of Volume 1. It continues the "large-scale read/write and scheduling" thread of Ch 8 (URL shortener) and Ch 9 (crawler), and sets up Ch 11 (news feed) — both are fan-out problems, differing in where the fan-out lands: here on third-party channels you do not control, in Ch 11 on inboxes you own. In the real world it maps to the notification platform that grows inside almost every consumer product: LinkedIn's ATC, Netflix's Pushy, every company's in-house messaging hub.
Start with the scale. The book's estimate is roughly 5M iOS pushes, 10M Android pushes, 1M SMS and 5M emails per day — about 21 million a day, which averages out to a mere 243 per second. That sounds like nothing.
And the average is the least useful number here. Notification traffic is spiky by nature: a campaign push to 5 million users that must land within five minutes is 16,700 per second — nearly 70× the average; a production alert must reach every on-call within 30 seconds. You design for the peak and pay for the average.
Skip the design and it breaks in three concrete ways. It drags down the main flow: call APNs synchronously after checkout, and with a provider that normally takes 200ms and occasionally 5s, your checkout p99 gets dragged along. It loses silently: a notification that lives only in memory dies with the process restart, and that "payment successful" never existed — and nobody will notice, which makes lost notifications the quietest failure mode there is. It drives users away: sloppy retries send the same push five times, the user switches notifications off, and that is the one irreversible failure in the chapter.
Step one is collecting contact details: when a user installs the app or signs in, the client exchanges a token with APNs/FCM and hands it to the API server, which stores it. The model is one-to-many — a user table (user_id, email, phone) and a device table (device_id, user_id, device_token, last login).
Hidden here is a point most people skip: push addresses devices, not people. A user with a phone, a tablet and an old handset means one notification fans out into three provider calls. So device count, not user count, is the real load parameter — sizing by users underestimates by two or three times.
The other half is recycling. Tokens die on uninstall and device change; APNs answers 410 Unregistered, FCM answers UNREGISTERED. You must use that receipt to delete dead tokens, or two things rot at once: wasted call quota, and a delivery rate diluted into meaninglessness — and delivery rate is your only external signal that a channel has broken.
The most direct design — business services → one notification server → providers → devices — dies in three places: a single point of failure (that server goes, all notifications stop), no room to scale (template rendering, media attachments and every provider SDK crammed onto one box), and tight coupling (a slow provider slows the upstream business).
There is a counter-intuitive point worth doing the arithmetic on: what kills the naive version is usually not QPS but the provider's latency variance. With 200 concurrent threads on one box and a provider averaging 200ms, theoretical throughput is 1,000/sec. Let the SMS gateway wobble to 5s and the same 200 threads deliver 40/sec — a 25× collapse while the CPU sits idle. Every thread is parked on a wait. That is the structural disease of synchronous direct calls.
Three cuts. One: move the database and cache out of the notification server so it becomes stateless — only a stateless server scales horizontally and auto-scales. Two: insert a message queue between the server and the providers; the server returns the moment the write lands, and main-flow latency drops from "wait for a provider" to a 1–5ms enqueue. Three: give every channel its own queue with its own pool of workers.
Why one queue per channel rather than one big queue with a type field? Two reasons, both hard:
The queue also hands you an observability handle: the book singles out queue backlog as the metric to watch. The operational rule is backlog ÷ drain rate = time to clear — once that exceeds your freshness target (say 60 seconds), scale out or shed load, rather than waiting for users to complain.
Don't lose — persist, then retry. Before enqueueing, write a notification log row recording event_id, recipient, channel and state; a worker that fails retries with exponential backoff (1s / 2s / 4s…) and lands in a dead letter queue with an alert after the cap. The honest cost: one notification goes from a single network call to "write, call, update state", roughly 2–3× write amplification. What you buy is the ability to answer "did this one actually go out?" — without the log you have no starting point for the investigation.
Don't duplicate — via idempotency, because exactly-once is unreachable. On a timeout you cannot distinguish "the provider never got it" from "the provider got it and the response was lost", so retrying necessarily produces duplicates. The only fix is to take the responsibility onto your side: every notification carries a globally unique event_id, and the worker does one atomic "have I seen this?" check before sending (Redis SET key NX EX, or a unique index). The TTL must cover the entire retry window — a 24-hour retry window needs a 24-hour dedup key, and anything shorter is the same as no dedup at all.
Don't annoy — via preferences and rate caps. The preferences table is trivial (user_id, channel, opt_in); what matters is that the check moves ahead of the enqueue. Otherwise the queue fills with messages that should never be sent, and a user who flips the switch mid-flight still gets whatever is already queued. The book's reason for rate caps is blunt: annoy a user and they turn off system-level notifications, and that door rarely reopens. The pragmatic form is "coarse filter before enqueue, recheck before send", the latter catching preference changes from the last few seconds.
Templates: one "your order has shipped" needs three languages × three channels = 9 variants; a template separates the fixed skeleton from the variable parameters so one edit propagates everywhere. Auth: calls to the notification API carry an appKey / appSecret, or anyone who reaches the internal endpoint can broadcast to your entire user base through your channels. Tracking: delivery, open and click rates go to analytics — and not only as a product requirement. A provider's 200 means only that it accepted the request; a sudden drop in delivery rate is your one external signal that a channel has broken.
The real choices in this chapter are not on the architecture diagram. They are two: which channel to use, and which side of the line owns the duplicates.
Table 1 · Choosing a channel: the certainty / cost / latency triangle
| Channel | Delivery certainty | Typical latency | Cost | Fits / costs |
|---|---|---|---|---|
| Push (APNs / FCM) | Weakest: no delivery guarantee, user can switch it off, hard cap on offline backlog | seconds | marginal cost ≈ 0 | High-frequency, low-value, droppable content. Cost: one permission toggle voids the whole channel |
| SMS | Higher (carrier-side), but still no "user read it" receipt | seconds to tens of seconds | Most expensive, priced per message | OTPs, account security, critical alerts. Cost: spend grows linearly with volume |
| Medium: bounces give real receipts, but spam folders exist | seconds to minutes | very low | Invoices, receipts, long content. Cost: too slow for anything urgent | |
| In-app inbox | Highest — the only channel entirely under your control | next time the user opens the app | one database write | Backstop and history. Cost: entirely passive; if they don't visit, nothing was sent |
In practice this becomes tiering plus fallback: promotions go to push and the in-app inbox only; critical transactions (payment, security) go "push first, fall back to SMS if unclicked after N minutes"; and everything also lands in the in-app inbox, the only channel that guarantees the user can find it when they go looking.
Table 2 · Delivery semantics: where the responsibility sits decides how much code you write
| Semantics | How it's built | What goes wrong | Where it fits |
|---|---|---|---|
| At-most-once | Send once, give up on failure, no retry | Drops messages, and drops them silently | Pure promotion, content nobody misses |
| At-least-once (this chapter's choice) | Persist + exponential backoff + DLQ | Duplicates | The default for almost everything |
| At-least-once + idempotency | The above, plus event_id dedup (Redis SET NX / unique index) | One extra lookup; a dedup TTL shorter than the retry window is the same as no dedup | The engineering answer — behaviourally equivalent to exactly-once |
| Exactly-once | Unreachable end to end: the last hop is a third party that doesn't promise it either | Chasing it makes the design more complex without making it more correct | In an interview, "you can't; use idempotency instead" is the right answer |
One more question that comes up: should the business call a notification API synchronously, or write to the queue directly? Writing directly is fastest, but validation then moves to the worker — by the time a bad parameter surfaces the message is already queued. Most teams call a deliberately thin notification API instead: validate and enqueue only, return in 1–5ms, keeping bad data out without ever waiting on a provider.
This skeleton is not a teaching model — it is the thing companies actually call a notification platform. Asked about notifications in an interview, saying these four things clearly gets you most of the way: one queue per channel (failure containment), event_id idempotency (duplicates are your responsibility), preference checks ahead of the enqueue, and backlog monitoring (the only useful health signal). Real systems grow one more layer on top — the intelligence that decides whether a notification should be sent at all, which is where the serious effort goes.
apns-expiration, APNs stores the notification and tries to deliver it at least once, repeating as needed until that date; set it to 0 and APNs attempts delivery only once and does not store it — evidence that retry and discard policy is set by the provider, not by you. Apple Developer Documentation, "Sending notification requests to APNs" ↗100 non-collapsible messages are stored for an offline device, and beyond that limit all stored messages are discarded; the default storage window is 4 weeks — confirming that push channels do not promise delivery. Firebase Cloud Messaging docs, "Non-collapsible and collapsible messages" ↗200 means APNs or FCM accepted the request. Whether the device was offline, the message collapsed or discarded, or permissions revoked, you never learn.1. The thesis: the difficulty is not sending, it is that the last mile is not yours — third-party channels guarantee neither delivery nor non-duplication, so every lever moves back onto your side.
2. The recipient granularity is a device, not a user: several devices per person means one notification becomes N calls, so size capacity by devices; dead tokens must be recycled from receipts like 410.
3. The naive synchronous design dies of three things — single point, no room to scale, tight coupling — and what kills it is the provider's latency variance, not QPS (200ms wobbling to 5s collapses single-box throughput 25×).
4. Three cuts to the skeleton: make the server stateless, insert a queue, and give each channel its own queue — the last buys failure containment and independent scaling, so a dead SMS gateway never touches push.
5. The metric to watch is queue backlog: backlog ÷ drain rate exceeding your freshness target means scale out or shed load.
6. The reliability trio: don't lose (persist + backoff + DLQ, at 2–3× write amplification), don't duplicate (event_id idempotency, with a TTL covering the whole retry window), don't annoy (preferences ahead of the enqueue, plus rate caps).
7. The semantics can only be at-least-once plus idempotency; end-to-end exactly-once is unreachable, and saying so proves more understanding than drawing the diagram correctly.
8. Channel choice is a certainty / cost / latency triangle: push is cheapest and least certain, SMS the most expensive and most immediate, and the in-app inbox is the only backstop you fully control. The most expensive failure is the irreversible one — a user who switches notifications off can no longer be reached even with an OTP, which makes rate caps a reliability control rather than product fastidiousness.