Books Deep-Read · ML System Design Interview · Chapter 3
Machine Learning System Design Interview · Ch 3 · Ali Aminian & Alex Xu · 2023
A camera car drives past your front door and photographs the whole street on the way. In those photos there are pedestrians, license plates parked at the curb, and possibly you. The photos are going online for the whole world to browse — so before they do, every face and every plate has to be smudged out. This chapter is about how a machine finds the parts that need smudging, and then smudges them.
You would assume the people building this want it to be as accurate as possible — smudge what needs smudging, nothing more. The opposite is true. They deliberately tune the machine to be paranoid: it would rather smear things that are not faces at all than let one real face through. They know it will smudge the wrong things, and they tune it that way anyway.
The first problem is volume. Count the archive in billions of photos. Even if one person could glance at a photo in five seconds, that is five billion seconds — roughly 158 people working full-time for a year — and the camera cars are out again tomorrow.
The second problem is sneakier. A machine that says "there is a face in this photo" is useless here. What you have to smudge is that particular patch of pixels, so you need to know where the face is and how big it is. "Noticing something is there" and "pointing at exactly where" are two completely different difficulties.
Four moves, each more important than the last.
Lean paranoid and the imagery fills up with inexplicably erased shop signs, billboard portraits and street art — and blurring is destructive, so once a patch is smeared it essentially never comes back.
When one kind of mistake costs orders of magnitude more than the other, "be as accurate as possible" is the wrong goal. The right move is to skew knowingly toward the cheap mistake — and then spend a whole extra stage cleaning up the mess that skew creates.
Want the mechanisms, the notation and the diagrams? → Switch to the deep read
On the surface this chapter teaches object detection. What it actually teaches is how to pick an operating point when the two error costs are wildly unequal: a missed face is an irreversible privacy incident, an over-blurred shop sign is an ugly photo. So the right answer is not the balance point between precision and recall — it is a detector deliberately parked at high recall, low precision, followed by a stage whose only job is killing false positives, plus a human report loop to catch what still slips.
The second of the ten case studies in Part II. The input is still an image, but the task moves from Ch2's "encode to a vector and retrieve" to "localise and classify" — and this is the only question in the book whose business objective is compliance rather than experience. It sets up Ch5 (harmful content detection), which runs the same low-base-rate / threshold / human-review triad. In the real world it maps onto the redaction pipelines behind Google Street View and Apple Look Around, and onto anonymising autonomous-driving road capture under GDPR.
Camera cars photograph public roads in dozens of countries, and those photos inevitably contain passers-by and license plates nobody ever asked about. Privacy law (GDPR and local regimes) requires de-identification before publication, and Street View itself has drawn privacy disputes and regulatory scrutiny in several countries.
The difficulty is not "can we detect faces". It is scale and consequence:
5×10⁹ seconds, about 158 person-years, while the cars keep adding more every day. Pure manual review is not expensive — it is nonexistent.So the real question is not "how high can we push mAP" but "how high can we push recall at an over-blur rate the business can live with". Treat it as a routine accuracy-optimisation problem and the design is wrong in round one.
Image classification only answers "does this image contain a face". But the thing you have to blur is that patch of pixels, and without coordinates there is nothing to act on — so the task must be object detection, emitting (box, class, confidence) with two classes: face and license plate.
One corollary that gets skipped: this is multi-class detection, and the two classes are not equally hard. Faces vary in pose, occlusion and scale (a distant face is a dozen pixels tall) and their shape is fluid; plates are rigid and carry a text prior, which helps, but they glare, skew under perspective, and fakes on posters and toys fool the model. Two classes deserve two independent thresholds; sharing one treats two problems as one (the open-source evidence in §6 is built exactly this way).
Two-stage (Faster R-CNN and relatives): the backbone compresses the image into a feature map → a region proposal network uses anchors to filter out a few hundred to a few thousand "something may be here" boxes → the second stage classifies only those candidates and refines their coordinates. Propose first, identify second.
One-stage (SSD, YOLO): a single forward pass emits, at every location on the feature map, "is there a target + which class + how far is the box off". No proposal stage.
The trade-off comes down to where the compute goes. Two-stage splits "where might something be" from "what is it", so the heavy compute lands on a small candidate set: usually higher accuracy, noticeably so on small targets, at the price of a longer pipeline and more time per image. One-stage drops the proposal stage and tends to run several times faster on comparable hardware, historically at the cost of weaker performance on small, dense targets (focal loss and successors narrowed that gap considerably).
The book lays out both, and its conclusion travels with its premise: Street View redaction is offline batch processing with no online latency budget, so optimise for accuracy — recall in particular — and take the two-stage design. Memorise the conclusion together with the premise: move the redaction onto the capture vehicle in real time and the answer immediately flips to one-stage. Saying only "two-stage, because it's more accurate" in an interview throws away the point.
Detection is supervised on two things at once — how well the box is drawn (a regression against ground-truth coordinates) and whether the class is right (cross entropy). They combine into a multi-task loss, L = L_cls + λ·L_box, where λ is the dial: turn it up and the model cares more about tight boxes, down and it cares more about not confusing classes.
This problem has a convenient property: a box that is slightly too big is nearly harmless (nobody complains about a generous smudge) while a miss is fatal — which is why production pipelines commonly dilate each detected box by a fixed margin before blurring, spending slack to absorb small localisation error.
On the data side, annotation means drawing boxes one at a time and is expensive, so the book leans on data augmentation: horizontal flips, random crops, rotation, brightness and saturation jitter, grayscale conversion. Offline augmentation materialises the variants to disk — fast training, more storage; online augmentation transforms during training — no storage, slower steps. One domain judgement: street scenes have a fixed gravity direction, so vertical flips produce samples that do not exist in reality and are wasted augmentation.
A detector fires on the same face from neighbouring locations and different anchors, so one street image easily produces dozens or hundreds of overlapping boxes. NMS is almost common sense: sort by confidence, keep the highest, discard every box whose IoU with it exceeds a threshold (0.5 is typical), then repeat on what remains.
Why the chapter has to cover it: without NMS your precision and recall are simply wrong. Twelve boxes on one face get counted as twelve detections, and over-blurred area is miscounted too — and every decision in this chapter rests on those two numbers.
The confidence threshold is a single dial: lower it and more boxes are reported, so recall rises and precision falls (more over-blur); raise it and the reverse. The default answer in almost every ML problem is "take the point where F1 peaks", and the correct answer here is explicitly not that point.
The reason fits on one line: call the cost of missing a face C_fn and of over-blurring a sign C_fp. Here C_fn / C_fp is orders of magnitude — an irreversible privacy incident with regulatory exposure on one side, image quality on the other — so the expected-loss-minimising operating point sits on the high-recall side.
But high recall is not free. Push the threshold down and each extra point of recall costs steeply more over-blur: erased shopfront signage, billboard portraits, toy license plates. The fix is not to compromise along one curve — it is to add a stage. Leave the detector parked at high recall and low precision, then run a dedicated false-positive killer behind it, using domain information the detector never sees to veto impossible boxes: a box whose absolute size makes no sense at that depth, a box entirely inside the sky region, a detection that is wildly inconsistent across adjacent frames of the same spot. The point of two stages is that each error type gets its own owner, instead of one threshold carrying both jobs.
Offline metrics rest on IoU deciding what counts as a hit, then AP / mAP on top. But mAP folds both error types into a single score, and a privacy decision needs them separated: at an acceptable false-positive level, what is recall (the recall value at a specified precision on the PR curve)? Try explaining "how many faces we missed" to legal with a composite score and you cannot.
The easier miss: report it stratified. An overall recall of 95% comfortably hides "frontal 99%, profile 70%, distant small faces 40%" — and misses concentrate in exactly those last two. Slicing by region, lighting and demographic group is the one move that stops you fooling yourself.
Online, the metrics go operational: blur requests per million images, the share of reports confirmed as genuine misses, and the sampled review rate for over-blurs.
Table 1 · One-stage or two-stage — decided by latency budget, not accuracy taste
| Skeleton | Mechanism in one line | What you get / what it costs | When to pick it |
|---|---|---|---|
| Two-stage Faster R-CNN | A proposal network filters to 10²–10³ boxes, then each is classified and refined | Higher accuracy, steadier on small targets; more time per image, longer pipeline, heavier to deploy | This problem: offline batch, no online latency budget, recall first |
| One-stage SSD / YOLO | One forward pass emits class and box offset at every location | Often several times faster on the same hardware, fits on-device; historically weaker on small dense targets | Redaction at capture time, on-vehicle, live video masking |
Table 2 · Which operating point — the only real design decision here
| Operating point | Consequence | Premise it needs |
|---|---|---|
| High precision threshold up | Clean imagery, almost no over-blur; but real faces get through — and one is an irreversible privacy incident | Only holds when a false alarm costs at least as much as a miss, which is false here |
| Peak F1 the balance point | Neither error type is common; looks the most "professional" | The default answer, but it assumes the two errors are equivalent — and their inequality is the whole question |
| High recall threshold down + post-filter | Misses pushed to a minimum; over-blur rises sharply and a second stage kills most of it | The book's choice: C_fn / C_fp spans orders of magnitude, and over-blur only costs image quality |
Table 3 · How to report metrics (a privacy system cannot ship one composite score)
| Metric | What it looks at | Fit for this problem |
|---|---|---|
| AP / mAP | Average precision across classes and IoU strictness levels | The academic standard, handy for comparing models; folds misses and over-blur into one number, so it cannot carry a privacy decision |
| Recall at a fixed precision | At an over-blur level the business accepts, how much did we miss | The headline metric here — it answers "how many faces are still exposed" |
| Stratified recall | Frontal / profile / small faces, by region and demographic group | The only reporting style that prevents self-deception; an overall 95% often hides a 40% slice |
| Report rate in production | Blur requests per million images, and their confirmation rate | The real-world proxy for misses, and a compliance record you need anyway |
All three tables point at one sentence: almost none of the choices here are about "which is stronger" — they are about how the costs are priced. Price the costs and the selections follow as corollaries; leave them unpriced and no selection can be defended.
The shape on the ground is a batch pipeline, not an online service: newly captured imagery is sharded geographically into a queue → a GPU / TPU fleet runs detection → NMS and post-filtering → a blur service dilates each box and applies a Gaussian blur → results land in object storage and go live. Two constraints have to be designed in from day one. First, blurring is destructive and irreversible, so retention period and access control on the originals must be pinned to the privacy policy. Second, the pipeline must be able to re-run history under a new model version, or else improvements only ever touch new photos and the misses already in the archive stay there forever.
The other half is the human loop: anyone can report "that is my face / my house number, please blur it", a reviewer confirms and blurs it — and those samples happen to be precisely the hard cases the model lacks, so feeding them back is active learning by construction. Its value is not only remediation: what compliance requires is that a usable redress channel exists, not that the model is 100% correct. Treating the human backstop as cover for a weak model misreads the chapter.
Swap the class list and the same skeleton is reused: anonymising road-capture and dashcam data, de-identifying medical imaging, masking whiteboards in meeting recordings, live-stream redaction. Whenever the rule is "some category must be covered, and one miss is an incident", this chapter is the answer.
1 · The core claim: when two error types differ in cost by orders of magnitude, "as accurate as possible" is the wrong objective — the right operating point for Street View redaction deliberately favours high recall and low precision.
2 · It must be object detection, not classification: you blur a patch of pixels, and without coordinates there is nothing to do. Output (box, class, confidence) over two classes, face and plate, each with its own threshold.
3 · Two-stage (proposal network then per-box classify and refine) vs one-stage (one forward pass); two-stage wins here because offline batch has no latency budget — change the premise and the answer changes.
4 · The training signal is a multi-task loss, L = L_cls + λ·L_box: localisation plus classification. A slightly large box is harmless and a miss is fatal, so pipelines dilate boxes before blurring.
5 · NMS is not a detail: leave the overlapping boxes on one target unsuppressed and recall, precision and blurred area are all wrong.
6 · The answer is not a compromise on one curve but a two-stage structure: the detector sits at high recall, the post-filter kills false positives with domain information. Google's Street View paper describes exactly this structure, with the fully automatic system blurring more than 89% of faces and 94–96% of plates.
7 · Report metrics separately and stratified: mAP for comparing models, recall at a fixed precision for decisions. An overall 95% routinely hides profile and distant faces at 40–70% — and unequal misses are unequal privacy protection.
8 · The architecture is a batch pipeline plus a human report loop: blurring is irreversible, so retention on originals is policy-bound and the pipeline must re-run history under new models. The human backstop is not cover for a weak model — a usable redress channel is what compliance asks for in the first place.