BOOK DEEP READ · ML System Design Interview · Ch 4
Machine Learning System Design Interview · Ch 4 · Ali Aminian & Alex Xu · 2023
You type "cat jumps on fridge" into YouTube and within a second it pulls a dozen clips out of a library counted in the billions. This chapter is about how that gets built: how one line of text becomes a list of videos sorted by "how much this looks like what you asked for."
You probably assume the trick is that the system "understands video." In truth, more than half of the credit goes to reading words — titles, descriptions, uploader tags, and machine-generated captions. The path that actually looks at the picture exists to cover the gaps: the things nobody bothered to write down.
Video has no words in it to search. The old answer was to let uploaders type their own titles and tags, and two problems show up immediately. First, people type junk — clickbait titles, piles of trending keywords. Second, a lot of what's in the frame nobody would ever write down: you want "red sports car drifting in the rain," and the person who shot it called the file "weekend vlog."
So put humans on it? YouTube alone takes in hundreds of hours of fresh video every minute. Several hundred people watching nonstop would barely keep pace with one day's uploads — and "which tags should I even write" has no finishable list to begin with.
In one line: search down two paths at once, then merge and sort.
Their strengths point in opposite directions. The word path is precise about proper nouns, model numbers and people's names, but it only finds what somebody wrote down. The picture path digs out content nobody described, but it also makes the "looks like it, so it must be it" mistake — handing you a different ginger cat jumping onto a couch. Serious systems never pick one; they run both and let each cover for the other.
Converting every video in the library into coordinates ahead of time and storing them is a real bill. And the day the model gets upgraded, all billion of them have to be recomputed and the index rebuilt — that is weeks-to-months of engineering.
Video search doesn't rest on "understanding video." It rests on two paths that are blind in different places, searching at the same time — one matching words, one measuring nearness — followed by a funnel that narrows their candidates down to the final dozen.
Want the mechanisms, the notation and the diagrams? → Switch to the deep read
On the surface this chapter builds "a video search." Underneath it is a cross-modal retrieval problem: the user gives you text, the corpus holds pixels. The book's answer is not to train a bigger model that "understands video," but to split the system into two independent retrieval paths — an ordinary inverted index over text metadata, and representation learning plus approximate nearest neighbor over the visual content — then collapse both candidate sets into one rankable relevance score through a multi-stage funnel. The difficulty was never the model; it is the metric definitions and the split between what runs offline and what runs online.
This is the third of the ten case studies in Part II. It follows Ch 2 on visual search — where both query and item were images, whereas here the query is text and the item is video, so you cross a modality and gain a time dimension — and sets up Ch 5 on harmful content detection, which pushes "how do you fuse several signals" further. It maps onto any scenario where text has to find non-text: in-platform video search, stock footage libraries, enterprise knowledge bases, the retrieval layer of multimodal RAG.
Search engines work on the premise that the thing being searched has words attached to it. Web pages have body text, products have titles and specs. A video has neither — just a stream of pixels and an audio track.
The naive fix is to let uploaders fill in metadata and then run the web-search playbook. That works, but it misses two large things:
Human labeling doesn't rescue it either: YouTube's own long-cited upload rate is hundreds of hours of new video per minute. Even if one person watched at 1× and tagged instantly, you would need several hundred of them just to keep pace with a day's growth — and "which tags" has no closed set anyway.
So the real question is: how do you make "what's in the frame" searchable too, without throwing away the lexical retrieval that already works well?
The book first translates the problem into an ML task: input a query string, output a list of videos ordered by relevance — a ranking problem, not a classification problem. There is no answer key, only relative order. The decision that really sets the shape comes next: two retrieval paths in parallel.
Why not fuse them into one? Because their failure modes are complementary: lexical retrieval misses content nobody wrote down, while vector retrieval makes the "close enough, so call it relevant" error — and is especially weak on proper nouns, model numbers and names that demand exact string matching. Splitting them also pays an engineering dividend: you can scale and roll out each independently, and when one breaks the other is still serving.
A 10-minute clip at 30 fps holds 18000 frames; run a billion videos through a network frame by frame and any budget burns down. So step one is always frame sampling: a fixed interval (say 1 frame per second, leaving 600 for those ten minutes) or keyframes at shot boundaries, then uniform decoding, resizing and pixel normalization. Encoders then come in two flavors:
The book leans somewhere very practical: start frame-level, because encoding a billion-item corpus is a bill paid in real money. Upgrade specific categories to video-level once you have confirmed temporal semantics is the bottleneck. This is the chapter's most characteristic judgment — cost before accuracy.
Each encoder emits a vector — but why should "a vector of words" be comparable to "a vector of frames" at all? Because training forces the alignment.
The training data is (query text, video) pairs, mostly harvested from search logs: a user searched this phrase, clicked this video and watched it through, so that becomes a positive. Negatives need almost no labeling — the other videos in the same batch serve as negatives, since the odds any of them happens to match this query are tiny.
The objective is a contrastive loss: push the similarity sim(q, v) of the matched pair (usually just the dot product of the two vectors) as high as possible, and the similarity between this query and every other video in the batch as low as possible. In plain words: turn "which footage goes with this sentence" into a multiple-choice question over a batch, ask it over and over, and both encoders are forced to move matching things next to each other.
That structure is the two-tower: the towers never look at each other, and meet exactly once, at a dot product. It looks like a handicap — the model gets no chance at fine-grained interaction between words and pixels — but it buys the property this chapter cares about most: the video side can be computed entirely offline.
An online query walks four stages (Fig 1): query understanding does spelling correction and intent detection and encodes the query into a vector; candidate generation runs both paths in parallel; ranking puts a heavy model over the few hundred survivors, with query-video feature interaction plus user profile, video quality and freshness; re-ranking carries the business rules — policy filtering, same-channel deduplication, regional restrictions, diversity.
Why must retrieval use ANN rather than exact search? A billion dot products cannot finish inside a millisecond budget; ANN compresses that with quantization and hierarchical indexes, at the price of occasionally missing the genuinely nearest few. Retrieval can absorb that error — its job is only "don't miss too much," and precision is covered by the two stages after it.
This is where two towers pay off: video vectors are batch-computed offline and loaded into the index, and online you encode exactly one thing — the query. Encoding the whole corpus once may occupy a GPU cluster for days, but amortized per query it rounds to zero. Conversely, swapping the model means recomputing the whole corpus and rebuilding the index — the largest hidden cost of this architecture, the thing the book says least about, and the point most worth raising unprompted in an interview.
Offline you use precision@k, recall@k, MRR and mAP; online you watch click-through rate, completion rate and total watch time. What each measures and when to use it is in Table 3 below. The line worth keeping is this: offline gains that don't show up online are the normal case. The reason isn't mysterious — the offline set is sampled from historical logs, so it only contains videos that were shown back then, and the good videos your new model wants to surface never had a chance to appear in it at all.
Table 1 · The two retrieval paths: what they cover, where they fail
| Text path · inverted index | Visual path · vectors + ANN | |
|---|---|---|
| Signal | title, description, tags, ASR captions | content of the sampled frames |
| Needs ML? | No — mature lexical scoring suffices | Yes: two encoders plus corpus-wide encoding |
| Best at | proper nouns, model numbers, names, exact phrases | content nobody wrote down |
| Typical failure | clickbait, cross-language queries, on-screen but untitled content | "looks alike, call it relevant"; out-of-distribution categories land absurdly close |
| Update cost | metadata edits take effect immediately | a model swap forces full recompute + index rebuild |
| Lean on it when | the query names a concrete entity; navigational queries | descriptive queries; long-tail semantic queries |
Table 2 · Choosing a video encoder
| frame-level | video-level | |
|---|---|---|
| How | sample frames → ViT per frame → pool | whole clip into a 3D conv / temporal Transformer |
| Captures | objects, scenes, visual style | also motion and inter-frame change |
| Cost | low; reuses off-the-shelf image-text weights | an order of magnitude more compute and memory |
| Fatal weakness | time is gone; forward ≈ reverse | the bill for encoding a billion items doesn't add up |
| Pick it when | default starting point | after confirming motion semantics is the bottleneck, for specific categories |
Table 3 · Picking metrics (four offline, three online)
| Metric | What it measures | When to use it / what it costs |
|---|---|---|
precision@k | how many of the top k are relevant | first-screen experience; meaningless unless k is stated |
recall@k | share of relevant items recovered in the top k | for the retrieval stage; the full relevant set is unknowable, so only a small labeled set approximates it |
MRR | reciprocal rank of the first correct result | navigational queries with one right answer; distorts when many answers are right |
mAP | precision aggregated across positions | overall ranking quality under binary relevance; not for graded relevance |
| Click-through rate | share of shown results that get opened | most sensitive, but skewed by clickbait titles and thumbnails |
| Completion rate | share of opens watched through | cancels out clickbait; naturally low for long videos |
| Total watch time | watch time attributable to search | closest to the business, but the longest attribution chain and the highest variance |
Two towers, contrastive training and ANN are today the default skeleton whenever text has to find non-text in industry: text-to-image search in e-commerce, stock footage retrieval, the retrieval layer of multimodal RAG — the shape is identical to this chapter's. Interview follow-ups cluster in two places: why not a single-tower interaction model (answer: the item side could not be precomputed, and a billion items cannot pass through a model online), and how do you debug offline gains that don't land online (answer: suspect exposure bias in the eval set first, then check whether online and offline features agree).
precision@k without saying how large k is, who judged relevance and how the query set was sampled makes the number incomparable.1. In one line: video search is cross-modal retrieval — text query, pixel corpus. The answer is not "train a bigger model that understands video" but two complementary retrieval paths plus a multi-stage funnel.
2. The root problem: the thing being searched carries no words. Uploader metadata misses everything nobody described and is polluted by clickbait; human labeling collapses against hundreds of hours uploaded per minute.
3. The two paths: text goes through an inverted index over title, description, tags and ASR captions and uses no ML at all; visual goes through representation learning plus ANN. Complementary failure modes, so run both rather than choosing.
4. Video encoding starts with frame sampling (10 minutes at 30 fps is 18000 frames; 1 fps leaves 600). Frame-level is cheap but loses time, video-level captures motion at ten times the cost — start frame-level.
5. Contrastive learning pins text and pixels into one space: positives from search-log pairs, negatives from the rest of the batch, pushing the matched sim(q, v) up and everything else down.
6. The real value of two towers is engineering: video vectors are computed offline into the index and only the query is encoded online; the cost is that a model swap forces a full recompute and index rebuild.
7. The four-stage funnel: query understanding → two candidate paths (10³) → ranking (10²) → re-ranking → results (10). Retrieval can absorb ANN's approximation error because the later stages cover precision.
8. Evaluation: the four offline metrics each measure something different, online you watch CTR, completion rate and total watch time; offline gains that don't land online are normal, and exposure bias in the eval set is the first suspect.