Book Deep Read · Generative AI System Design Interview · Ch 3
Generative AI System Design Interview · Ch 3 · Ali Aminian, Hao Sheng & Alex Xu · 2024
You point your phone at a street sign in a language you can't read, and a few hundred milliseconds later a fluent sentence appears. This chapter is about building that machine — one that has to serve a hundred-plus languages at once, most of which it has barely seen a decent example of.
The old way was like someone working from a phrasebook: chop the sentence into pieces, look each piece up, glue the results back together. You can understand the output, but you can tell instantly that no human wrote it — none of the pieces knows what the whole sentence is about.
The new way is like a real interpreter. They listen to the entire sentence first, form a meaning in their head, then speak the translation one word at a time. And as they say each word, their eyes flick back over the original to check which part they're on right now. That one habit is why long sentences stop getting lost halfway through.
There are around seven thousand languages, but nearly all the bilingual material on the internet piles up in a dozen big ones — English and French have as much as you could ever want, while some languages can't scrape together ten thousand clean matched sentences. And then there's sheer count: two languages need one interpreter, but a hundred languages, every pair, is close to ten thousand directions. You cannot keep an interpreter for each.
First, teach one person all the languages. Instead of one machine per language pair, build one machine and stick a little tag on the front of the sentence saying "put this into Chinese." The payoff is that languages help each other: the feel for grammar it picked up from Spanish gets lent to a neighbouring language it has barely studied. You even get something odd — pairs it was never taught at all, which it can still translate, because it learned each end separately.
Second, make up homework for the languages that have none. If a language has articles but no matched translations, run the existing machine backwards to turn them into mediocre English, and use that fake pairing as practice material. The exercises are synthetic; the skill it builds is real.
Third, decide who gets to practise. If you allocate practice time in proportion to what's actually on the internet, the machine spends ninety percent of it on English and French, and the small languages never get a turn. So you deliberately raise the small languages' quota — and the big ones get slightly worse. That is the central trade in this chapter: the plate is only so big, and anyone who gets more means someone gets less.
How do you know whether it translated well? The usual method is to compare its output against one "correct answer" and count how many words overlap. But a sentence has many correct translations — phrase it differently, the words stop matching, the score drops, and a human reads it as equally good. So the number is only good for spotting trends. To actually judge quality, you still have to ask a person.
The engineering difficulty in translation isn't "how do I turn one language into another." It's how to serve everyone when the resources are wildly unequal, and how to tell whether you're improving at a task that has no single right answer.
Want the mechanisms, the notation and the diagrams? → Switch to the deep read
You think machine translation is hard because of linguistics; it is actually hard because of resource allocation and measurement. The model shape — encoder–decoder with attention — largely settled after 2017 and is no longer the argument. What decides whether the system works is how to serve a hundred-plus languages from one set of parameters without starving the small ones, and how to know you are improving on a task with no single correct answer. This is the book's first problem whose output is a full sentence that must be judged sentence by sentence.
un-, -ing) so a fixed-size vocabulary can spell any word, including unseen ones.0–100, higher is better.This is the second of the ten case studies in Part II. It follows Ch2 Smart Compose — also sequence generation, but Ch2 had a hundred-millisecond budget and was forced into a small model, while translation can spend a few hundred milliseconds and afford a large one; the two chapters together measure how far a latency budget can pull a design apart. It leads into Ch4 ChatGPT: from generation that has a reference answer to generation that has none. In the real world it maps onto Google Translate, DeepL, subtitling and document localization, and the substrate of any cross-lingual search or support system.
Statistical MT chopped sentences into chunks, looked them up in a phrase table, and glued the pieces back. It worked, but no chunk knew what the whole sentence was about, so every long-range relationship was lost — agreement, word-order reordering, pronoun reference. A gluing system structurally cannot reach them.
The harder problem is scale and skew. First, pairs grow quadratically: N languages translating both ways is N×(N−1) directions, which at N=100 is 9900. Training, shipping and monitoring a separate model per direction is not an option. Second, the corpus distribution is brutally long-tailed: English–French or English–Spanish pull in hundreds of millions of sentence pairs, while many languages have fewer than a hundred thousand clean pairs available anywhere — four orders of magnitude and up.
What happens if you don't solve it? You ship a product where twenty languages are genuinely useful and the other hundred read like a machine-translation joke. The real ambition of the chapter is this: make the hundred borrow from the twenty.
Translation is modelled as: given the source sentence, generate the target sentence token by token, each step picking from the whole vocabulary. Two engineering consequences. First, latency scales with output length — emitting N tokens means N forward passes, so "translate a long article" and "translate a short message" are two different SLOs: interactive translation typically pins p95 in the low hundreds of milliseconds, batch document translation is budgeted on throughput. Second, there is more than one correct translation — which is where the entire evaluation problem later in this chapter comes from.
What requirements clarification should actually ask: how many languages, text or speech, interactive or batch, must formatting and glossaries survive. "How many languages" decides the architecture; the rest only decides parameters.
A word-level vocabulary collapses immediately under multilinguality: the union of a hundred languages' words is millions of entries, and the embedding matrix scales with the vocabulary. Subword segmentation (BPE / SentencePiece) splits words into common fragments, so a shared vocabulary of a few tens of thousands can spell any word in any language, including names and coinages never seen in training.
A shared vocabulary has an overlooked bonus: it creates genuine parameter sharing between languages — Latin-script languages reuse the same fragments heavily, so a representation learned from Spanish is directly reusable for Catalan. The cost is a fight over vocabulary budget: without sampling balance, high-resource languages take almost all of it, small languages get shredded into fragments, the same sentence can cost three or four times as many tokens as in English, and cost and quality degrade together.
The method is startlingly plain: don't change the architecture, just prepend a target-language token — <2zh> meaning "into Chinese" — and train one set of parameters on all directions mixed together.
It buys three things. Operationally, 9900 models collapse into one: a single thing to train, ship, monitor and roll back. In quality, you get cross-lingual transfer: low-resource languages borrow syntactic and semantic structure learned from high-resource ones. In capability, you get zero-shot: a Portuguese→Korean pair never seen in training still translates, because each end's mapping to English pulled the intermediate representations into a shared space.
The cost is called capacity dilution (also the curse of multilinguality): with parameters fixed, more languages means less capacity each, and high-resource directions typically come out slightly worse than their dedicated bilingual models. So real systems are usually hybrid — head directions get dedicated adapters, the tail all rides the multilingual model.
Transfer borrows some of it, but a shortage still has to be filled. Three main lines:
Back-translation has the best return. Target-language monolingual text is abundant; run a reverse model over it to produce "machine-translated source + human-written target" pairs. The direction is the whole point: the noisy side is the input, the clean human side is the output, so the generation distribution the model learns stays clean.
Corpus mining: automatically pair sentences from web pages, subtitles and multilingual official documents, filtered by multilingual sentence-embedding similarity. Enormous yield, enormous noise — dirty parallel data is routinely underrated: a mispaired sentence doesn't teach the model a wrong word, it teaches the model that it may ignore the source and improvise.
Data balancing: sampling in proportion to real corpus size hands almost all the compute to English–French–German–Spanish. The practice is to reweight by corpus size to the power 1/T (larger T is flatter), deliberately raising the small languages' quota. This is a straight zero-sum dial: low-resource scores rise, high-resource scores fall. There is no free lunch, only a choice about where you concede.
BLEU counts the n-grams shared between the output and a reference translation, times a penalty for being too short. It is cheap, fast and regressible, which is why it has been the default ruler for twenty years. But its assumption is that a good translation looks like the reference — and translation is one-to-many. Paraphrase, reorder, restructure: a human reads it as equally good, BLEU docks points.
So the standard practice is three layers together: BLEU / chrF as the fast ruler for daily regression (chrF works on characters, steadier for morphologically rich languages like Finnish); COMET / BLEURT, learned metrics trained directly on human ratings and correlating far better with people, as the pre-launch gate; MQM human evaluation as the final arbiter — expensive and slow, run only on critical releases.
And a class of failures that automatic metrics structurally cannot see needs its own checklist: hallucination on low-resource directions (output unrelated to the source yet perfectly fluent — wrong, fluently, the signature failure mode of generative systems), default gender and occupational stereotyping, and terminology or brand names getting "translated" away.
Table 1 · Three ways to hold a hundred languages
| One bilingual model per direction | Pivot through English | A single multilingual model | |
|---|---|---|---|
| Model count | about N×(N−1), ~9900 at N=100 | about 2N, ~200 at N=100 | 1 |
| Quality | highest on head directions | two hops: errors compound, register lost | head dips slightly, tail rises a lot |
| Latency | one hop | two hops, roughly double | one hop |
| Adding a language | train about 2N new models | train 2 | add to the mix, retrain once |
| Cost | operationally impossible; tail neglected | non-English pairs collapse | capacity dilution; one bad run hurts every language |
| Fits | a handful of pairs, quality maxed | a stopgap / fallback for rare pairs | the default, with adapters on head directions |
Table 2 · Four routes to data for low-resource languages
| Back-translation | Web corpus mining | Cross-lingual transfer | Raising sampling temperature | |
|---|---|---|---|---|
| Needs | target monolingual text + a reverse model | web scale + multilingual sentence embeddings | a closely related high-resource language | nothing — just change the mix |
| Payoff | steady and stackable; the industry default | the largest volume | cheapest when the families are close | immediate |
| Cost | needs a usable reverse model first; synthetic style is uniform | noisy pairs teach the model to ignore the source | near useless across unrelated families | high-resource directions get worse |
| Do first? | first priority | parallel with back-translation, filter hard | comes free with the architecture choice | last, tuned to product priorities |
Table 3 · Three rulers, three things measured
| BLEU / chrF | COMET / BLEURT | MQM human evaluation | |
|---|---|---|---|
| Measures | n-gram / character overlap with a reference | a learned estimate of the human score | each error by type and severity |
| Cost | near zero, seconds | a model run, minutes | expensive and slow, person-days |
| Human correlation | weak; paraphrase is punished | markedly higher | the reference itself |
| Fatal weakness | change tokenization and the score moves; not comparable across papers | biased itself, and can be optimized against | only affordable as a sample; never covers the tail |
| Used for | daily regression, fast screening | the pre-launch gate | critical releases and disputes |
The skeleton here is much bigger than translation: any system serving wildly unequal subpopulations from one model looks like this — multilingual speech recognition, recommendation and risk scoring across country sites, intent classification across dialects. Share parameters to get transfer, add a token to condition, tune the sampling mix for fairness, then get stuck behind an imperfect automatic metric. Same four steps.
The most valuable line in an interview: talk about multilinguality as a capacity-allocation problem, not a linguistics problem. Saying "I'll let English–French drop a BLEU point to gain three points on twenty tail directions, because our growth is in emerging markets" beats reciting any model name.
600M to 54.5B. "The tail needs a purpose-built evaluation set" is itself the conclusion: a language with no ruler is a language nobody maintains.facebookresearch/fairseq · NLLB branch ↗28.9 BLEU, 29.9 with GNMT-style attention. First-hand support for "attention is not a garnish, it repairs a structural defect".tensorflow/nmt README ↗① Translation is conditional generation: read the whole sentence, emit word by word; latency scales with output length and there is no single right answer.
② Attention repairs a concrete structural defect — squeezing a whole sentence into one fixed vector necessarily loses information.
③ A shared subword vocabulary is the entry ticket: tens of thousands of entries spell every language, and create cross-lingual parameter sharing for free.
④ One model per direction is an N×(N−1) dead end; one model plus a target-language token collapses it into a single artifact.
⑤ That single model buys cross-lingual transfer and zero-shot, and costs capacity dilution: head directions usually trail their dedicated bilingual models.
⑥ Four routes for the tail: back-translation (first), corpus mining (huge and noisy), transfer from close relatives, sampling temperature.
⑦ Sampling temperature is a zero-sum dial: whatever you lift at the tail, you lose at the head — a product-priority decision, not an optimization.
⑧ BLEU's assumption that a good translation looks like the reference simply fails on a one-to-many task; change the tokenization and the score moves.
⑨ Evaluate in three layers: BLEU / chrF to screen → COMET / BLEURT to gate → MQM to arbitrate.
⑩ List the failures metrics cannot see: fluent hallucination on tail directions, gender stereotyping, terminology and brand names translated away.