Books Deep-Read · ML System Design Interview · Ch 1
Machine Learning System Design Interview · Ch 1 · Ali Aminian & Alex Xu · 2023
When you scroll a short-video app, it somehow keeps queuing up the next clip you want. Your inbox quietly keeps spam out. Behind both sits a system that learns: nobody wrote the rules one by one — it worked them out from a mountain of examples. This chapter doesn't teach you to build one particular system of that kind. It hands you an order of operations: what to think about first, and where people fall over.
Most people assume the hard part is "picking a clever model." Inside a real company, the model is usually a small piece of the whole thing. Wrapped around it: where the data comes from, how it gets cleaned and stored, how it reaches the model in time, and how anyone notices when the thing quietly gets dumber. Engineers spend most of their hours on that ring, not on the model.
Ordinary software is "you set the rules": spend 200, save 30, written into the code, and when it's wrong you can trace it line by line. A learning system is "it finds the pattern in the examples itself": you hand it a million past records and it distills a way of judging that nobody can read back. Which leads to something genuinely awkward — when it goes bad, nothing throws an error. No server is down, the API still returns, the recommendations just get worse and worse. The usual debugging instincts stop working here.
The chapter splits building such a system into seven steps, each answering a very concrete question:
The crucial bit: these seven steps aren't a line, they're a loop. What step seven catches sends you back to step three to redo the data and retrain.
It's slow: just pinning down the requirement and standing up the data pipeline often takes several times longer than training the model — and almost none of that work is visible from outside.
Building a system that learns is never mainly about picking the model. It's about translating a fuzzy business wish into a scorable question, getting the data where it needs to be, and then proving on real people that it actually helps — and after launch it degrades on its own, so someone has to keep watching.
Want the mechanisms, notation and diagrams? → Switch to the deep read
The framing chapter for the whole book: designing a system with a model in it splits into seven steps — clarify requirements, frame it as an ML task, prepare data, develop the model, evaluate, deploy and serve, monitor. The core insight is that the model is a small piece of the system: what decides success sits on either side of it — upstream, "how does a business objective become an ML objective"; downstream, "how do you prove online that it helps, and how do you notice when it rots."
This is the only chapter of Part I in Aminian & Xu's Machine Learning System Design Interview (2023), and the template every one of the ten case chapters (Ch 2–11) follows. It inherits the four-step interview method of Xu's earlier two volumes but changes the terrain: those books deal with deterministic systems (what you write is what you read), this one with statistical systems (the output is a probability; correctness only holds in aggregate). In the wild it maps to recommendation, search ranking, ads, risk scoring and content moderation.
ML design questions are even more open-ended than pure system design questions, and there is one dominant failure mode: opening with the model — "I'd use a Transformer two-tower retriever." That may be technically fine while skipping everything that actually decides the outcome: what the business objective is, how positives and negatives are defined, where the features come from, whether an offline win moves anything online, and how you'd know three months later that it still works.
The deeper pain: ML systems fail silently. A broken service throws 5xx, times out, pages someone. A drifting ranker does none of that — latency normal, error rate zero, recommendation quality sliding for weeks until it shows up in a business review. Skip this and you build a system that runs beautifully while quietly losing money. The seven-step framework exists to nail down everything that lives outside the model.
In "design a video recommendation system," nearly every word is undefined. Ask about: the business objective (watch time or new-user retention? the optima differ), which features the system supports (can users say "not interested"?), data (which sources, is anything labelled, can behaviour serve as implicit labels), constraints and scale (compute budget, how many users, how much content, requests per day), and performance requirements (inference latency ceiling, freshness).
The output isn't a design, it's a list that pins the boundaries down. One order of magnitude changes everything: a candidate pool of 10 thousand versus 1 billion decides whether you need approximate nearest-neighbour retrieval; a 500 ms versus 30 ms budget decides how big the model can be.
The most valuable step in the chapter and the one most often skipped. Three parts.
(a) Define the ML objective. A business objective is human language; an ML objective must be a quantity a model can optimise. The book puts the two side by side: a video platform's "increase user engagement" → "maximise time users spend watching"; harmful content detection's "make the platform safer" → "accurately predict whether a piece of content is harmful"; friend recommendation's "help users grow their network faster" → "maximise the number of connections formed." The sloppier the translation, the further the system drifts — translate "engagement" bluntly into "maximise click-through" and the model learns to promote clickbait.
(b) Specify input and output. Same requirement, different definitions, two different systems. The book's cleanest example: harmful content detection can be one multi-label model (one piece of content in, probabilities for violence / nudity / hate speech out) or one binary model per category. The former shares representations and saves compute, but adding a category means retraining everything; the latter lets each category iterate and tune its own threshold, at the cost of multiplying training and serving by the number of categories.
(c) Choose the ML category. Supervised / unsupervised / reinforcement; within supervised, classification versus regression. The most commonly misjudged family is ranking: recommendation, search and feeds look like classification, but what you actually need is a sortable score, so they are usually framed as learning to rank or representation learning — encode users and items as embeddings and use vector similarity as the score, which is what lets a billion-item candidate pool be pre-filtered by approximate nearest-neighbour search.
Two halves. Data engineering: sources (user-generated, system-generated, third-party), storage choices, ETL pipelines, and data types — structured (numerical, categorical) versus unstructured (text, images, audio, video). Feature engineering: turning raw data into something a model can consume, with four routine moves —
[0,1]), standardisation (zero mean, unit variance), log transform (for long-tailed features like "follower count" spanning several orders of magnitude).The hard part isn't the algorithm, it's train/serve consistency: a feature computed in SQL offline and by different code online only needs to disagree slightly (misaligned time window, different missing-value fill) for online quality to drop inexplicably — training-serving skew.
Model selection weighs more than accuracy: training data volume, number of features, training speed, hyperparameter count, compute needs, interpretability. The book's path is plain — start from a simple baseline: get the whole chain working with heuristics or logistic regression, then swap in more complex models, each compared against the baseline. The reason: a dumb model that has a working data pipeline, evaluation and release path beats a better model that only exists in a notebook.
Model training deals with constructing the dataset (defining features and labels, sampling, train/validation/test splits, handling class imbalance — in CTR settings positives are often a fraction of a percent, addressed by resampling or loss weighting), choosing a loss function, training from scratch versus fine-tuning a pretrained model, and at scale distributed training (data parallelism: one model copy per device, each on a different batch; model parallelism: the model itself split across devices).
Offline evaluation scores held-out history, with metrics by task type: classification uses precision / recall, F1, ROC-AUC and PR-AUC; regression uses MSE / MAE / RMSE; ranking uses precision@k, MRR, nDCG (@k meaning "only the top k results" — nobody scrolls to the 50th); generation uses BLEU / ROUGE / FID.
Online evaluation measures something else entirely: business metrics — click-through, watch time, conversion, retention, revenue. There is no stable correspondence between the two, which is the counter-intuitive lesson of the chapter: offline AUC climbs two points, you ship it, and the business metric doesn't budge or goes backwards. Usually because the offline data carries the incumbent system's exposure bias (you only have feedback on what was already recommended; anything never shown has no label), or because the ML objective was never aligned with the business objective.
Deployment settles three things: where it runs (cloud or on-device), how to shrink it (distillation: a small model learns the large one's outputs; pruning: drop low-contribution parameters; quantisation: 32-bit floats down to 8-bit integers), and how to roll it out (shadow, A/B, canary, interleaving, bandits — see the next section), plus whether the prediction pipeline is batch or online.
Monitoring answers one question: why does a model degrade in production? The leading cause is data distribution shift — behaviour changed, the content ecosystem changed, an upstream source quietly redefined a field. So you watch two families: operational metrics (latency, throughput, utilisation, uptime) and ML-specific metrics (drift in input feature and output prediction distributions, accuracy over time where ground truth flows back, and which model version is actually live).
Table 1 · Cloud deployment vs on-device deployment
| Cloud | On-device (phone / edge) | |
|---|---|---|
| Latency | One network round trip; 100ms+ cross-region | No round trip, tens of milliseconds |
| Model size | Essentially unbounded; GPUs available | Pinned by memory and battery; usually needs distillation / pruning / quantisation |
| Privacy | User data must be uploaded | Data never leaves the device; far less compliance pressure |
| Updates | Change the server once, everyone gets it | Ship a release, users may not upgrade: version fragmentation |
| Cost | Inference compute scales linearly with requests | Compute paid for by the user's device; near-zero marginal cost |
| Fits | Large models, frequent iteration | Real-time interaction, offline use, privacy-sensitive (keyboard autocomplete, on-device face grouping) |
Table 2 · Batch prediction vs online prediction
| Batch | Online | |
|---|---|---|
| How | A scheduled job precomputes everything into a key-value store | Fetch features and run inference when the request arrives |
| Request latency | One lookup, milliseconds | Feature fetch plus inference sit on the critical path |
| Freshness | Results may be hours old; what the user just did isn't reflected | Can use the "right now" context (just clicked, current location) |
| Compute | Computed for every user; most results are never seen — pure waste | You only pay for real requests, but must absorb peak QPS |
| Fits | Stable candidate sets, slow-moving behaviour (daily digest emails, offline profiles) | Strong context dependence (search ranking, feeds, ad auctions) |
Table 3 · How to test a new model in production
| Method | How it works | What you get / what it costs |
|---|---|---|
| Shadow | Both models take the same traffic; the new model's output never reaches users, it's only logged and compared | Zero user risk; costs double inference, and yields no business metrics (nobody reacts to its output) |
| A/B test | Randomly split users between old and new, compare business metrics | The only method giving a causal answer; needs enough samples for significance — a small change may need one to two weeks |
| Canary | Send 1% of traffic to the new model, widen if nothing breaks | Contained blast radius, fast rollback; too noisy at small traffic to conclude anything |
| Interleaving | Mix both rankings into one list and see which source the user clicks | For ranking, an order of magnitude fewer samples than A/B (between-user variance is cancelled); only applies to ranking |
| Bandit | Shift traffic share between versions based on live performance | Earns while it tests, wasting less traffic on bad versions; statistical inference is harder |
All three tables run on one theme: every choice trades "more accurate" for "slower, costlier, or harder to change." What an interview actually tests is not how many options you can name, but whether you can justify one against a stated scale and latency budget.
The framework isn't only for interviews. It maps onto what the industry calls MLOps: feature platforms, experimentation platforms, model registries and versioning, drift monitoring. Pull apart the internal ML platforms at Google, Meta, Netflix or Airbnb and you get roughly these seven steps, engineered.
In an interview it gives you pacing: of 45 minutes, spend 5 clarifying, 5 translating the objective and pinning input/output, 20 on data, features and model choice, and the last 10 on evaluation and post-launch monitoring. Skipping step 2 and going straight to the model is the most common way to lose these interviews.
1. The book's framing chapter: ML system design = seven steps — clarify requirements, frame as an ML task, prepare data, develop the model, evaluate, deploy and serve, monitor.
2. Core claim: model code is a small piece of the system; data, features, serving, experimentation and monitoring are where the work is.
3. Step 2 matters most: translate the business objective into an optimisable ML objective and pin input/output down — same requirement, different definitions, two different systems; ranking problems are usually framed as learning to rank or representation learning.
4. Feature engineering has four moves: missing values, scaling, discretisation, categorical encoding (learned embeddings for high cardinality); the engineering risk is train/serve consistency.
5. Build a baseline before anything complex; training must handle class imbalance, loss choice, fine-tuning and distributed training.
6. Evaluation has two layers that routinely disagree: offline gains don't imply online gains — exposure bias and objective mismatch are the usual culprits.
7. Rollout methods each cost something: shadow is risk-free but yields no business metrics, A/B gives causality but slowly, canary contains blast radius, interleaving saves an order of magnitude of samples for ranking.
8. ML systems degrade silently: drift is the leading cause, so monitoring must cover both operational metrics and input/output distributions — the seven steps are a loop, not a line.