Books Deep-Read · ML System Design Interview · Chapter 6
Machine Learning System Design Interview · Ch 6 · Ali Aminian & Alex Xu · 2023
You open a video site and the home page hands you about twenty videos. You probably imagine a machine that knows you well, picking your twenty favourites out of everything it has. This chapter says the real problem is different: the hard part isn't picking well, it's that there is nowhere near enough time to pick at all.
Imagine a city with a billion restaurants. You are hungry, standing at your door, and you need to be told where to eat before you can blink.
"Try all billion, then rank them" was never on the table — not because the tasting is inaccurate, but because there simply isn't time. So the real method splits into two passes: first something very fast and a bit crude narrows a billion down to a few hundred plausible ones, then something slow and fussy ranks those few hundred properly. Almost every design in this chapter grows out of that arithmetic.
Before you can split the work, you have to answer a simpler question: what makes the system think you'd like something? There were two old answers, each with its own way of dying.
So real systems run both: the second one does the discovering, the first one covers what the second can't reach.
The trick is doing the work ahead of time. The system precomputes, for every video, a short string of coordinates — roughly, where it sits on a huge map of interests. It computes one for you too: which part of that map you usually hang around in. When you arrive, it does one thing: find the points on the map near you — the way you'd find nearby restaurants on a map instead of measuring the distance to every restaurant in town.
That pass only tries not to miss anything, and leaves about a thousand. Only then does the slow, fussy model rank them properly. Finally there's a review pass: this one isn't licensed in your country, this one is stale, these two are the same clip, all twenty are from the same creator — all removed here.
The thing worth remembering is that each layer chases something different. The first chases "don't drop anything" — what it drops, no amount of later precision can recover. The second chases "get the order right". The third handles the rules you can't teach a model: licensing, freshness, no duplicates, don't be monotonous. Collapsing all three into one layer is the classic beginner's mistake.
The system can only learn from things it has already shown you — anything it never showed you, it will never find out whether you'd have liked, so it keeps recommending an ever-tighter version of who it thought you were yesterday.
A recommender's shape isn't set by "how do we get smarter", it's set by an arithmetic of time: a billion candidates against a budget shorter than a blink forces the three passes — coarse filter, fine ranking, final review. And its deepest flaw comes from that same shape: it can only learn from what it has already shown.
Want the mechanisms, the notation and the diagrams? → Switch to the deep read
On the surface this chapter builds a video recommender; underneath it makes two calls. First, recommendation has to be staged — not for accuracy but for arithmetic: a billion-item corpus against a few tens of milliseconds means "run the model over every video" was never possible, so a dirt-cheap model cuts a billion down to a thousand (chasing recall), and an expensive one ranks that thousand (chasing precision). Second, the model has to be able to eat user features — methods that only know the interaction matrix (matrix factorization) work beautifully on established users and go completely mute on new users and new videos, which are exactly the platform's largest daily increment. The difficulty isn't the architecture; it's cold start and exposure bias: the system can only learn from what it has already shown.
This is the fifth of the ten case studies in Part II, and the book's first ranking problem. It follows Ch5 on harmful content detection — there several signals converge into a verdict, here into an ordering — and leads into Ch7 on event recommendation, same skeleton with far heavier feature engineering. In the wild: the YouTube home page, Netflix's row-based home, the TikTok feed.
The book's setting is a YouTube-scale platform: a corpus in the billions, hundreds of millions of daily users, and a home page that needs a few dozen videos per user. The business objective is engagement (not click-through rate — the difference bites later). A home page response budget is typically on the order of 100ms, of which recommendation often gets only tens of milliseconds.
The whole shape of the chapter lives in one sum: even at 1 microsecond per video, scoring a billion of them takes 1,000 seconds — and that's for one request from one user. So "score the whole corpus, take the top 20" isn't slow; it doesn't exist.
The second trap is just as hard: the training data is produced by the system itself. Users can only click what they were shown, so the logs contain only what was shown, and a video that was never surfaced never gets a chance to prove itself. Not enough time, and self-generated data — those are the two things this chapter is really up against.
Content-based filtering pairs the video's own attributes with your history and offers "things like what you watched". It has one ability nothing else gives you: a freshly uploaded, zero-view video can be surfaced the same day as long as it has a title and tags; niche taste is fine too, because it never requires that someone else be like you. The cost is hand-built features, and worse, it can't surface a new interest — it only winds tighter around the circle you're already in. That's the filter bubble.
Collaborative filtering is the mirror image: it reads only the "who watched what" table. It needs no domain knowledge, and it can find interests you couldn't have articulated — cross-category associations are visible only to it. The cost is cold start: the new user's row is empty, the new video's column is empty, and a large slice of the corpus is cold at any moment.
So the book picks a hybrid, and industry usually wires it in series rather than in parallel — collaborative filtering produces candidates and content-based fills in what it can't reach (new videos, cold users, the tail), rather than two rankings being averaged. One judgement to take away: collaborative filtering does the discovering, content-based does the covering.
Matrix factorization is the classic realisation of collaborative filtering. Picture an enormous table: rows are users, columns are videos, cells hold feedback, and 99.99% of the cells are empty. Factorization splits it into two skinny tables — a row of numbers per user, a row per video — and the dot product of the two is the predicted score. Training adjusts those two tables so the dot products match the real feedback on the observed cells.
What goes into a cell is itself a choice. Explicit feedback (likes, ratings) is accurate but brutally sparse — most people finish a video and tap nothing. Implicit feedback (clicks, watch time) is dense but noisy — a click isn't fondness, and finishing may just mean it was background noise. In practice both are used, weighted differently.
There's also a training trap the book calls out: compute the loss only over observed cells and the model learns to recommend everything — unseen cells were never penalised, so inflating every dot product costs nothing. Unobserved cells must be folded in as zeros (weighted alternating least squares, WALS, is exactly a weighting between the two terms).
But factorization has one flaw it can't design away: it only knows IDs. A new user has no row in the table, so the system knows nothing about them — even when you plainly know they're 25, in Tokyo, and searched for "camping gear" five minutes ago.
The two-tower model replaces "look the ID up in a table" with "let a neural network eat a pile of features and emit a vector". The user tower takes demographics, context (time of day, device) and history (the average of recently watched video vectors); the video tower takes title and tag text vectors, length, language. Each tower emits a vector, the dot product is the relevance score, and training is a binary classification — watched items are positives, randomly sampled ones are negatives. Cold start opens up as a result: a new user has no history, but the demographic and context paths are still there, so the tower still produces a vector; likewise for a new video.
What actually makes two towers deployable, though, is a purely engineering property: the two towers' inference decouples. The video tower runs offline over the whole corpus and the vectors go into an ANN index; a live request runs the user tower once and does one vector lookup. The costs: more training data, an index that will "occasionally miss", and features from the two sides that meet only at the final dot product, which limits how much crossing they can express — precisely what the ranking layer exists to repair.
Turn that arithmetic into structure and you get the three-layer funnel: candidate generation → ranking → re-ranking. Each layer chases something different, and that is the single most useful picture in the chapter.
Retrieval: several sources in parallel, each doing one job — a relevance source (two-tower plus ANN), popular and trending sources, new uploads from channels you follow. Each yields hundreds to thousands; merged and deduplicated you're left with about 10³. This layer over-collects rather than under-collects: what it drops is gone no matter how good the ranker is, while what it over-collects the ranker will clean up. So you evaluate retrieval with recall, not precision — using precision here is the most common case of the wrong yardstick.
Ranking: with only a thousand candidates left you can finally afford a heavy model, and finally afford what retrieval couldn't — cross features between user and video, real-time counters. The objective is one of the chapter's key calls: optimise engagement (watch time, completion), not click-through rate. Optimise CTR alone and the system reliably promotes clickbait — the user clicks, bounces after three seconds, the metric goes up and the experience collapses.
Re-ranking: three families of filters, all of them constraints you can't write into a model. Compliance — region licensing, misinformation, harmful content (last chapter's machinery plugs in right here); freshness — demote the stale; experience — dedupe near-duplicates, enforce diversity, cap how many items one creator gets on a screen. It gets treated as decoration, but it carries the part of product intent that a loss function can't hold.
Features come in three families: user (demographics, context such as time of day and device, behaviour history), video (ID, length, language, title and tag text vectors, upload time, cumulative views) and interaction counters. How does history become a feature a model can eat? Most commonly by averaging the vectors of the last N watched videos — crude but strong, at the cost of throwing the order away.
Evaluation runs on two sets of books. Offline: precision@k, mAP, and diversity (the mean pairwise similarity of the recommended list — lower is more diverse); the book notes pointedly that diversity alone is meaningless, since recommending random videos maximises it. Online: click-through rate, completed videos, total watch time, explicit feedback.
"Offline up, online flat" is the norm rather than the exception here: offline metrics are computed on logs produced by the old policy, and what the new model wants to surface is exactly the part the old policy never showed and the logs never labelled. The only credible judge is an online A/B test.
Table 1 · Three recommendation paradigms
| Content-based | Collaborative | Hybrid (the book's pick) | |
|---|---|---|---|
| New video | Servable day one (metadata suffices) | Can't serve it (nobody watched) | Covered by the content-based path |
| New user | Needs a little history | No answer at all | Popular / trending + demographics |
| New interests | Weak: circles the existing bubble | Strong: only it sees cross-category links | Supplied by the collaborative path |
| Cost | Hand-built features; filter bubble | Cold start; sparse tail | Two systems to build and maintain |
Table 2 · Retrieval model: matrix factorization vs two towers
| Matrix factorization | Two-tower (the book's pick) | |
|---|---|---|
| Input | Interaction matrix only, ID lookup | User + video features, encoded to vectors |
| Cold start | New user / video: goes mute | Profile and context remain; still emits a vector |
| Training | Fast; WALS or SGD both work | Slow, and needs notably more data |
| Serving | One dot product, very light | One user tower + one ANN lookup |
| Weakness | Loss over observed cells only → recommends everything | Approximate index misses; crossing only at the last step |
Table 3 · What each funnel layer is actually chasing (the most commonly conflated table)
| Retrieval | Ranking | Re-ranking | |
|---|---|---|---|
| Input size | 1B → 10³ | 10³ → 10² | 10² → a few dozen |
| Chases | Recall / coverage | Precision (engagement estimate) | Compliance and experience constraints |
| Usable features | Only precomputable, two-side-decoupled ones | Cross features and live counters both fine | Rules + business metadata |
| Latency budget | ~10 ms | ~30 ms | single-digit ms |
| Failure mode | A miss is unrecoverable | Bad order, content still present | Policy-violating or monotonous screen |
One operational judgement isn't in any table: cold start is mostly a product problem, not a model problem. New users are carried by popularity plus demographics on the first screen and converge from their first few clicks; new videos enter through metadata and get a slice of exploration traffic to accumulate interactions.
"Multi-source retrieval + ranking + re-ranking" is now the universal shape of every feed product: the YouTube home page, Netflix's row-based home, the TikTok feed, Spotify's Discover Weekly, e-commerce "picked for you" — and even RAG's "coarse vector retrieval, then cross-encoder rerank". Same funnel, different payload. Three interview follow-ups recur: why can't one model do it end to end (do the 1,000-second sum out loud), what goes on a new user's first screen, and what to do when offline improves but online doesn't.
① The shape follows from arithmetic: 1B candidates × 1 µs each ≈ 1,000 seconds, so it must be staged — retrieval, ranking, re-ranking.
② Content-based serves new videos and the tail but is trapped in the filter bubble; collaborative finds new interests but has no answer to cold start. Collaborative discovers, content-based covers.
③ Matrix factorization splits a sparse feedback table into two skinny ones with the dot product as the score; a loss over observed cells only makes it "recommend everything", so fold the unobserved in (WALS).
④ Its fatal flaw is that it only knows IDs; two towers replace the lookup with encoders, letting profile and context into the model and opening up cold start.
⑤ Two towers ship because the towers decouple: video vectors are precomputed into an ANN index and a request runs one user tower plus one lookup. Costs: more data, index misses, weak feature crossing.
⑥ Each layer chases its own thing: retrieval chases recall (a miss is unrecoverable), ranking chases precision (target engagement, not CTR), re-ranking holds the constraints no loss function can.
⑦ Implicit feedback is dense and noisy, explicit is accurate and sparse; in practice mix them and weight them separately.
⑧ The deepest trap is exposure bias: the training data is the system's own output, so offline metrics can only circle inside the old policy's field of view. Exploration traffic, position-bias correction and online A/B are the three standard fittings this chapter leaves you with.