ML System Design Interview Ch 5 · Harmful Content Detection

Machine Learning System Design Interview · Ch 5 · Ali Aminian & Alex Xu · 2023

What is this chapter about?

You scroll past hundreds of posts a day and almost never run into gore, slurs or scams. That isn't because nobody posts them. It's because every piece of content passes through an invisible checkpoint before it reaches you. This chapter is about how that checkpoint is built.

Start with something odd

There's a kind of content where every piece, taken apart, is spotless. A photo of a skunk — harmless. A line that reads "love the way you smell today" — harmless. Put them together and send them at a person, and it's an insult.

The malice isn't in the picture or in the words. It's in the join. So a checkpoint built as "let the picture inspector look, then the text inspector look, and wave it through if both say fine" is structurally blind to this. Half the design in this chapter exists because of that fact.

Why the old world struggled

Three old approaches, all of them short.

Wait for user reports. Reports always arrive after the fact — by the time someone spends the fifteen seconds to file one, thousands have already seen the post. Harm is counted in views, not in posts.
Keep a banned-words list. Split the letters, swap in a homophone, or just write the sentence inside the image, and the list is dead. This is a chase where the other side actively dodges.
Review everything by hand. A platform with hundreds of millions of daily users takes in new content by the billion each day. Review capacity is a fixed number and hiring cannot close that gap — and staring at this material for a living harms the reviewers themselves.

The core mechanism

Three moves.

Combine first, judge second. Turn the picture and the words each into a string of "impressions", stitch them into one complete impression, and judge that — instead of judging separately and voting at the end. Only then does the skunk-plus-caption malice get a chance to be seen.
One well-rounded reviewer, not ten specialists. Violence, nudity, self-harm, hate — rather than training one model per category, train one shared foundation with a dozen small per-category judgements on top. The payoff is concrete: self-harm, a category with very few examples, gets to borrow the foundation the common categories paid for.
The output isn't yes or no, it's a degree of confidence. Then act by band: very confident, take it down; unsure, cut its reach and queue it for a person; looks fine, let it through and send a random sample to people anyway. What the reviewers decide becomes new teaching material for the next model.

How to think about it

The thing that matters isn't how clever the model is. It's where those two dividing lines get drawn. Draw them loose and people get hurt; draw them tight and people speaking normally get their posts pulled for nothing. There is no "correct" spot — and different categories belong in different spots: missing one self-harm post and wrongly deleting one rudely-worded one are not the same order of cost.

What it costs

Handing the least certain band to people ties a fixed-capacity tail to the system: loosen the threshold and the queue explodes, so the content that most needs a human ends up waiting hours.

Remember one thing

The hard part of harmful content detection isn't "recognising bad things". It's two other things: information has to be merged early, because malice often lives only in the combination; and the model scores, it does not rule — you leave the least certain band to humans and accept that their capacity is the ceiling of the whole system.

Want the mechanisms, the notation and the diagrams? → switch to the deep read

In one sentence

On the surface this chapter builds a harmful-content classifier. At its core it makes two calls. First, fusion must happen early: harm often sits neither in the image nor in the text but in their combination, and any structure that judges each modality separately and then votes is blind to it by construction. Second, the output must be a score, not a verdict: on a distribution where positives are a few in a thousand, every single threshold forces a choice between missing harm and deleting legitimate speech, so real systems cut the score into bands and give the least certain band to people. The difficulty was never the architecture — it's thresholds under a low base rate, and human review capacity.

Glossary

Multimodal: one item carrying several forms at once — text, image, video. A social post is multimodal by nature.
Early / late fusion: the first concatenates per-modality features and then judges once; the second lets each modality judge for itself and merges the conclusions.
Embedding / vector: a piece of text or an image squeezed into a string of numbers (say 512); the more alike the content, the closer the strings.
Multi-label classification: one item can belong to several categories at once (both violent and hateful), so each category gets its own 0–1 score instead of one exclusive pick.
Multi-task learning: several "task heads" sitting on one shared backbone; a single forward pass answers every task, and the tasks share the learned representation.
Base rate: the share of positives in the whole population. For harmful content it is often a few in a thousand or less, which distorts many familiar metrics.
Precision / recall: the first is "of the items flagged harmful, how many really were" (wrongful deletion); the second is "of all harmful items, how many were caught" (misses).
ROC-AUC vs PR-AUC: the areas under two evaluation curves. The first is inflated when the base rate is tiny; the second stares straight at precision.
Human review queue: where items the model is unsure about land, to be judged one by one. Its throughput is fixed and is the hard constraint on the whole system.
Appeal: the channel by which an author asks for a deletion to be reviewed. The success rate is the direct measure of what wrongful deletion costs.

Where it sits

This is the fourth of the ten case studies in Part II. It follows Ch4, video search — there multimodality served retrieval (pinning text and pixels into one space so likeness becomes distance); here it serves adjudication (funnelling several signals into one decision). It leads into Ch6, video recommendation. In the real world it maps to every platform's Trust & Safety stack, comment moderation, and the content-safety layer every LLM product now bolts on.

What the chapter is solving

The book's setting is a social platform with hundreds of millions of daily users taking in new content by the billion per day, where a post can carry text, images, video and links at once. The target is harmful content — violence, nudity, self-harm, hate speech and the like. Note that the book carves off a separate problem: bad actors (bots, scams, harassment, engagement farming) are a behavioural problem, fought with account graphs and action sequences, and are out of scope here.

Why does this deserve its own chapter? Because the earlier case studies can all be pushed forward with a better model, and this one cannot. Each old path dies its own death: user reports arrive after the fact — cost is counted in reach, not in posts, and a violating video that takes off can reach hundreds of thousands before anyone reports it, while reports themselves are both late and sparse; banned-word lists cannot survive an adversary — posters actively evade with split characters, homophones, variant spellings, text baked into images, and every list update changes the evasion; pure human review does not arithmetically work — at a billion new items a day, even at two seconds each and forty thousand items per person per day, you would need tens of thousands of reviewers online simultaneously just to keep pace.

So the real question is: how do you produce an actionable judgement before content is widely seen, while placing the two errors it will inevitably make — misses and wrongful deletions — where each can be absorbed?

The core, properly

First call: early fusion, because the malice lives in the combination

A post has text and an image. How do the two meet? Late fusion has each judged once and merges the two scores; early fusion encodes each modality into a vector, concatenates them into one full feature, and hands it to a single classifier. Late fusion is engineering-seductive — independent training, off-the-shelf models, separate scaling. The book still picks early fusion, for two reasons, the second harder than the first:

One, late fusion cannot see cross-modal malice. Content whose image is fine and whose text is fine and which is harmful only jointly — the industry calls these benign confounders — returns a low score on both paths, and a low score after merging. This is not a weak model; it is a structural blind spot: each path has already been squeezed into a scalar before the two can meet.

Two, labels are not per-modality. Annotation yields "this post violated these policies", never a per-modality label. Late fusion requires an independently trainable classifier per modality, which requires you to manufacture a whole extra per-modality annotation set — an expense that simply does not pencil out at a billion items a day.

Early fusion's costs are real too: the interaction space is large, so it needs meaningfully more training data to learn at all; you forfeit the dividend of off-the-shelf per-modality classifiers; and when a modality is missing the concatenated feature still has to be filled with defaults.

Fig 1 · A benign confounder: image fine, text fine, combination harmful. Late fusion merges only after each path is a scalar, so this malice is lost by construction; early fusion concatenates before compression and keeps the interaction.

Second call: multi-task learning, not one model per category

What you're blocking isn't one kind of content but a set of policies — violence, nudity, self-harm, hate speech — and one item can violate several at once. Four shapes, eliminated in order:

A single binary classifier (harmful / not). Cheapest, but it gives one score and cannot say which policy. That's a defect, not a UX quibble: with no category you cannot set per-category thresholds, tell the author why, or run an appeals process.
One binary classifier per category. It names the category, at the price of N trainings, deployments and monitors — and worse, rare categories fight alone: self-harm may have on the order of one percent of the positives violence has, and cannot be fed on its own.
A single multi-label classifier: one model, one output per category. Cheap, but one shared representation must serve wildly different tasks.
Multi-task learning: a shared backbone learns a general representation, with one task head per category on top. The book's choice.

Multi-task wins in three places, the third most useful in practice: data efficiency — rare categories borrow the representation the common ones paid for, exactly what "one model per category" cannot provide; serving cost — one forward pass yields every category's score; each head can set its own threshold — cost structures differ by orders of magnitude across categories, and missing a self-harm post should not share a line with deleting a rudely-worded one. The price: tasks can drag on each other (negative transfer), and changing one policy means retraining the whole backbone.

Fig 2 · Multi-task learning: the shared backbone lets rare categories borrow what common ones paid for, while the heads keep per-category thresholds. The price is negative transfer, and one policy change means retraining the backbone.

Features: the other half of the signal

The book splits features three ways, and interviewees routinely drop the last two whole:

Content features: text through a BERT-family encoder, and the book specifically calls for a multilingual variant — the platform is global, and a model trained only on English collapses on low-resource languages; images through a pretrained visual encoder, video sampled into frames first.
Author features: prior violations, account age, follower count. An account registered three days ago with two takedowns already has nothing in common, as a prior, with a ten-year-old one.
Context features: views, shares, report count, comment text. Report count is the strongest single feature and the most dangerous one — it can be brigaded, and it is late by nature. It can be a feature; it must never be a trigger.

Evaluation: why you have to watch the PR curve

Every evaluation difficulty here comes from one fact: the base rate is tiny. Run the arithmetic. Suppose a million posts contain a thousand genuine violations (base rate 0.1%). A classifier with 99% recall and a 1% false-positive rate — which sounds formidable — recovers 990 true ones while wrongly hitting about 9,990 ordinary posts. Precision is under 10%: nine of every ten deletions are wrongful.

And its ROC-AUC will look gorgeous: the ROC's x-axis is the false-positive rate, so hitting 1% of 990,000 negatives moves it by 0.01. Under a low base rate ROC-AUC systematically flatters the model; the curve to read is PR, whose y-axis is precision itself.

Online, the yardsticks change. The three worth memorising: prevalence (harmful content's share of impressions, not of posts — counting views is what matches the real distribution of harm), proactive rate (the share caught before any user report), and valid appeals (the share of removals successfully appealed, the only directly observable measure of what wrongful deletion costs).

Key trade-offs

Table 1 · Where fusion happens

	Late fusion	Early fusion
Structure	one classifier per modality → merge scores	encode each → concatenate → one classifier
Convenience	High: train/scale separately, reuse off-the-shelf	Low: trained as a whole, one change moves everything
Labels needed	per-modality labels (which don't exist)	post-level labels (which is what you have)
Fatal flaw	Blind to cross-modal malice, silently	Large interaction space, needs far more data
Fits	categories damning in one modality alone	the chapter's default

Table 2 · Four model shapes

Shape	What it yields	Cost / why it loses
Single binary	one harm score	can't name the policy → no per-category thresholds, no explanation, no appeals
Binary per category	per-category scores	N trainings / rollouts / monitors; rare categories starve
Single multi-label	all scores in one pass	cheap, but one shared representation spread over very different tasks
Multi-task	shared representation + one head each	the book's pick; price is negative transfer and backbone retrains

Table 3 · From score to action (band → handling → who pays)

Band	Handling	Where the cost lands
High	auto-remove + tell the author which policy	Wrongful deletion: generates appeals, hits legitimate speech
Middle	demote reach + enqueue for human review	Review capacity: loosen the threshold and the queue explodes; delay is harm
Low	allow + a small random sample to review	Misses: sampling is the only way to learn what this band let through

One more operational rule beyond the three bands: the review queue should be ordered by score × projected reach, not by score. A 0.72 video currently taking off deserves a human before a 0.88 post nobody is seeing — cost is counted in reach.

Fig 3 · Bands and handling: the model only scores; the action comes from the thresholds. Each band hands one kind of error to whoever can absorb it, and review decisions flow back as training data.

Why it matters / in real systems

"Multimodal early fusion + multi-task + score bands + review feedback" is the shape of every platform's Trust & Safety stack today, and it was lifted wholesale into the content-safety layer of LLM products — one pass on the input, another on the output, likewise multi-category scoring rather than a blanket refusal. Interviews probe three spots: why not late fusion (the structural blind spot, plus labels that only exist at post level), why an ROC-AUC of 0.99 doesn't ship (base rate one in a thousand, precision possibly under 10%), and how the review queue is ordered (by score × reach).

Evidence from the field · not the book's claims, these teams' own

Meta AI (FAIR) · MMF describes itself as "a modular framework for vision and language multimodal research from Facebook AI Research", and says it "also acts as starter codebase for challenges around vision and language datasets (The Hateful Memes, TextVQA, TextCaps and VQA challenges)" — Facebook ran a public challenge and shipped a framework specifically for the cross-modal adjudication problem, which is itself an endorsement of this chapter's first call. facebookresearch/mmf · README ↗
OpenAI · Moderation publishes an evaluation dataset that cuts content into eight categories — sexual, hate, violence, harassment, self-harm, sexual/minors, hate/threatening, violence/graphic — noting that "the category label is a binary flag" — the industrial form of exactly the reason this chapter eliminates the single binary classifier: you must score per category before per-category thresholds or explanations are even possible. openai/moderation-api-release · README ↗ (paper: A Holistic Approach to Undesired Content Detection, 2022)
Google Jigsaw · Perspective API calls itself "a free API that helps you host better conversations online", whose models "score the perceived impact a comment might have on a conversation" — note the output is defined as a score, and explicitly of perceived impact, not a ruling that the comment is harmful; the action and the threshold are left to the platform. Exactly this chapter's "the model scores, it does not rule". conversationai/perspectiveapi · README ↗
YouTube has long published two quantities isomorphic to this chapter's online metrics in its Community Guidelines Enforcement Report: the violative view rate and the share first detected by automated systems; the Q4 2022 report states that of the 5.6 million-plus videos removed that quarter, over 94% were first detected automatically. YouTube Community Guidelines Enforcement Report · Q4 2022 (the egress proxy blocked that page this run, so the wording was not re-verified and no link is given)

Pitfalls and disputes

Reporting ROC-AUC. At a base rate of one in a thousand, a model with ROC-AUC 0.99 can still be wrong on nine of every ten deletions. Once the base rate is low, report PR-AUC and precision at a stated recall.
Treating human review as a backstop rather than a bottleneck. Review capacity is a fixed number, and it back-solves how wide the middle band may be: compute how many items a day people can clear, then derive the threshold — not the reverse.
Forgetting this is adversarial. A recommender's distribution drifts slowly; harmful content's distribution is pushed by an opponent, and probing starts the day you ship. Continuous retraining, adversarial mining and review feedback are routine operations here.
Assuming labels are objective. What counts as hate speech has no cross-cultural, cross-jurisdiction consensus, and inter-annotator agreement is low to begin with. Label quality caps model accuracy, and bias in the labels gets amplified onto particular groups — sliced evaluation (by language, by region) is mandatory, not a bonus.
Dated in places. The book is from 2023; since then multimodal foundation models and "let a large model be the moderator" have become the new route, and the hand-assembled encoder stack here looks old. The skeleton, though, has only firmed up: bands, human backstop, feedback, ordering by reach — all determined by the cost structure rather than model capability, and required whatever model you swap in.

Takeaways

① Two things at the core: information must be fused early, and the output must be a score rather than a verdict.

② The problem: reports are after the fact, word lists cannot survive an adversary, and pure human review does not arithmetically work at a billion items a day — while cost is counted in reach.

③ Early vs late fusion: benign confounders (image fine, text fine, combination harmful) are a structural blind spot for late fusion; and labels exist only at post level, so per-modality classifiers cannot even be trained. The price is needing far more data.

④ Multi-task > per-category models > a single binary: the shared backbone lets rare categories borrow common ones' data, one pass yields all scores, and each head still sets its own threshold.

⑤ Features aren't only content: author violation history and account age, plus views/shares/reports and comments, are strong signals; report count is a feature, never a trigger.

⑥ Be able to do the low-base-rate arithmetic on the spot: base rate 0.1%, recall 99%, FPR 1% → precision under 10%. Hence PR, not ROC.

⑦ Three online metrics: prevalence (by impressions), proactive rate (how far ahead of reports you run), valid appeals (the cost of wrongful deletion).

⑧ Three bands: high → auto-remove (cost: wrongful deletion), middle → demote and enqueue (cost: review capacity), low → allow and sample (cost: misses); order the queue by score × projected reach, and let review capacity back-solve the thresholds.