Book Deep Read · Generative AI System Design Interview · Ch 5
Generative AI System Design Interview · Ch 5 · Ali Aminian, Hao Sheng & Alex Xu · 2024
You glance at a photo on your phone and instantly know what is going on. But for someone who cannot see the screen, and for a system that has to file away hundreds of millions of images, the picture first has to become a sentence. This chapter is about getting a machine to look once and say something human — not "dog, grass, frisbee", but "a dog catching a frisbee on the grass".
Machines are good at labels: there is a person here, a dog, a frisbee — and they are usually right. But read those three words to a friend who cannot see the photo and they still have no idea what happened. Did the person throw the frisbee to the dog, or did the dog bring it back? Labels list the things and drop every relation between them. And the relations are usually the point of the picture.
So stitch the labels into a sentence? What comes out reads like an inventory: "person dog frisbee grass". Worse, ten people shown the same photo will write ten different sentences that are all correct — one describes the action, one the weather, one how delighted the dog looks. There is no single right answer, which makes teaching the machine and checking its work hard at the same time.
Two people work in relay. The first one looks at the picture. He writes nothing down in words; he chops the image into a few hundred small squares and keeps one scribbled card per square — this one is furry, that one is green grass. The second one does the talking. He never sees the picture, only the stack of cards, and he speaks one word at a time; before each next word he flips back through the stack and pulls out the few cards that matter right now. Saying "dog", he is holding the furry ones; saying "frisbee", a different few.
How are the two trained? On web pages. Countless images online ship with a line of caption text attached, which hands you hundreds of millions of "picture plus sentence" pairs for free. The catch is plain: most of that text is junk — "click to enlarge" and the like — and over half has to be thrown out before any of it is usable.
The crude way: have five people each write a reference sentence for the same photo, then see how much wording the machine's sentence shares with theirs. But a paraphrase gets unfairly marked down — "puppy" and "canine" do not count as a match. So there is a second trick: bring in a scorer that looks at the picture, one that never reads the reference sentences and answers only one question — do this sentence and this picture go together?
Seeing the picture is the entry fee; the hard part is that nobody can state what "described it correctly" means, so everything from training to sign-off runs on a stack of approximations propping each other up. And the dangerous failure is not silence — it is a fluent sentence naming something that was never in the picture, which the listener has no way to check.
Want the mechanisms, the notation and the diagrams? → Switch to the deep read
You think the hard part of image captioning is understanding the picture; in fact the vision half is largely solved by off-the-shelf pretrained models. What this chapter actually tackles is two things nobody can pin down: how to align a visual space with a language space into a chain you can generate through autoregressively, and how to score anything when one image has infinitely many correct descriptions. The engineering intuition is a single line: do not build a captioner from scratch — take two already-strong models and train only the thin translation layer between them.
The fourth of the ten case studies in Part II. It follows Ch4 (ChatGPT), where input and output were both text; here the book crosses modalities for the first time: image in, text out — the mirror image of Ch7–Ch11, which go text in, image out. In the real world it maps onto accessibility alt text, image search in stock libraries and e-commerce, content understanding ahead of moderation, and the "look and describe" step inside every multimodal assistant today.
First, separate it from the previous generation of tasks. Classification and detection return labels: person, dog, frisbee. But labels drop the relations — "the person throws the frisbee to the dog" and "the dog brings the frisbee back" have identical label sets. A screen reader announcing three nouns still leaves a blind user without the event. What this chapter wants is a sentence with a verb in it.
Second, scale. Writing alt text by hand is not impossible, it is merely unfinishable: a mid-sized image platform taking in 10 million new photos a day, at 30 seconds per human description, needs roughly 83,000 hours — about ten thousand person-days, every day. A model does one image in tens of milliseconds. The problem is automation-only from the first minute.
Third, and worst: "correct" has no single form. Translation at least has reference translations, classification a ground-truth label; here one image has unboundedly many acceptable descriptions, and coarse ones are useful for one purpose (search indexing) while detailed ones are useful for another (accessibility). Evaluation has to be rebuilt from scratch — which is why it takes up so much of the chapter's back half.
What happens if you skip it? Blind users hear the word "image" and nothing else as they scroll; and billions of pictures become a black hole that cannot be searched, moderated or filed.
The standard skeleton is vision encoder plus language decoder. A ViT cuts a 224×224 image into 16×16 patches, yielding 196 visual tokens, each a d-dimensional vector; the decoder is an ordinary autoregressive language model. Between them there must be a projection or adapter that moves the visual vectors into the language model's embedding space — the two sides speak different languages, and this layer is the interpreter.
At generation time the decoder runs cross-attention for every word it writes: it computes a weight over those 196 visual tokens, which is exactly "flipping back through the stack of cards for the ones that matter now". Writing "dog", the weight concentrates on the furry patches; writing "grass", it slides to the bottom half of the frame. This is precisely what labels cannot do: the attention moves as the sentence advances, and the relations get spoken in the course of that movement.
Why this design is startlingly cheap: the vision side is a one-off few hundred tokens, and a caption is usually 10–20 tokens, so the generated volume is two to three orders of magnitude below text-to-image — on the same GPU, an image takes seconds while a caption takes tens of milliseconds. The cost sits elsewhere: finer patches see more detail (448×448 gives 784 tokens), and attention cost grows with the square of the token count. Reading small text and dense scenes is bought with exactly that.
Training needs image–description pairs, and there are only two sources — opposite on nearly every axis.
Human annotation: COCO Captions gives about 120,000 images 5 human-written descriptions each. Five sentences per image is itself direct evidence that acceptable answers are plural. It is clean and well-formed, but capped in the hundreds of thousands and skewed towards everyday scenes.
Web alt text: billions of pairs for free, and filthy — file names, "click to enlarge", keyword stuffing, or text with no bearing on the image at all. So the real work is the filtering pipeline: drop non-sentences by length and part-of-speech, cross-check named objects against an off-the-shelf detector, and generalise proper nouns into hypernyms ("Obama" → "politician") so the model does not memorise names. Google's Conceptual Captions washed 3.3 million pairs out of web alt text exactly this way.
A third route is synthetic backfill: have a mediocre captioner rewrite descriptions for the dirty images, then let a scorer discard the ones that clearly do not match, producing something at near-web scale and near-human cleanliness. In essence it spends compute instead of annotation budget — at the price of writing the model's own preferences into its next training set, where errors reinforce themselves.
Training uses teacher forcing: at every step the correct prefix from the reference caption goes in, so the model only ever learns "given a correct prefix, what comes next". It trains fast and parallelises well. The hangover is exposure bias: at inference the prefix is the model's own, and once the third word goes wrong the model is in a situation it has never seen and keeps going wrong.
Decoding then becomes an honest trade. Beam search picks the highest-probability sentence, which makes captions uniformly safe and boring — an endless parade of "a man is …", because the most probable sentence is by construction the most generic one. Sampling (temperature, nucleus) makes captions vivid while pushing hallucination up: the model grows bolder about details that are not there. Beam search at k=3–5 with a length penalty is the usual landing spot for accessibility, where being wrong costs far more than being dull.
There is one more crack that is easy to miss: training optimises cross-entropy (is each word right?) while sign-off reads CIDEr (does the whole sentence look human?), and those are not the same thing. Hence the practice of optimising the metric directly as a reward (self-critical sequence training): let the model sample a few captions, score them with CIDEr, and shift towards the high scores. It does lift the metric substantially — and it opens the door to gaming the metric, which is Ch4's reward hacking wearing a different coat.
Automatic metrics come in three generations, each patching a hole in the last.
N-gram overlap (BLEU, METEOR, ROUGE): compare the generated sentence's word groups against several references. Paraphrases get punished unfairly, while boilerplate like "a man" collects points for free. CIDEr is the captioning-specific fix: it weights n-grams by TF-IDF so that phrases common to all images score near zero, and only what is distinctive about this image earns credit.
Structural (SPICE): parse the sentence into a scene graph — objects, attributes, relations — and reconcile it against the references' graphs. This catches exactly the layer labels dropped: "dog catches frisbee" and "frisbee hits dog" score almost identically on n-grams and are two different events on a scene graph.
Reference-free (CLIPScore): skip ground truth altogether, encode image and sentence into one shared space and measure how close they sit. It can credit correct descriptions no reference happens to contain — and it inherits CLIP's own blind spots: counting, spatial relations and negation.
All three together still do not replace human review. One more thing gets measured before launch: the object hallucination rate, the share of mentioned objects that are not in the image. It is unrelated to fluency, and it is the only fatal error class in accessibility.
The first decision is how to wire it up: strong pretrained models exist on both the vision and the language side, so the question is which part you train and which you freeze.
Table 1 · Four ways to wire it
| From scratch | Frozen towers + adapter | Full fine-tune | Call a general VLM | |
|---|---|---|---|---|
| Trainable params | everything | typically 1%–5% | everything | none |
| Data needed | tens of millions of pairs | hundreds of thousands to millions | millions, and high quality | none (a few examples at most) |
| Training cost | highest; out of reach for most | lowest; one multi-GPU box | high, and watch for catastrophic forgetting | no training cost |
| Domain fit | customise anything | swapping either tower only means retraining that layer | best fit to a vertical | often below bar on medical imaging, industrial inspection |
| Inference cost | self-hosted, quantisable, tens of milliseconds per image | billed per token, expensive at volume, plus a network hop | ||
| What it costs you | pure spend, unless you are a foundation-model shop | capped by the two frozen ends; fine-grained recognition often trails full fine-tuning | serves one domain at a time; a new domain starts over | data residency, rate limits and version drift all out of your hands |
| Fits | foundation-model teams | the default starting point for almost everyone | verticals with proprietary data | validation phase, low volume, general subject matter |
The second decision is what you sign off against. The commonest mistake here is shipping on a single CIDEr number.
Table 2 · Five ways to judge a caption
| Needs references | What it measures | What it misses | When to use it | |
|---|---|---|---|---|
| BLEU | yes (several) | n-gram overlap | punishes paraphrase; pays for boilerplate | a snapshot for regression tests only |
| CIDEr | yes (several) | TF-IDF-weighted n-gram overlap | still surface matching; scores well with relations reversed | the main offline iteration metric |
| SPICE | yes | whether objects, attributes and relations reconcile | depends on parsing; a bad parse poisons the score | when you suspect invented relations |
| CLIPScore | no | image–sentence fit in a shared space | counting, position, negation; inherits CLIP's bias | bulk monitoring in production, no ground truth |
| Humans + hallucination rate | no | usability, and the share of objects mentioned but absent | expensive, slow, not repeatable as a regression | the final gate before launch |
The third decision gets treated as a detail and in fact defines the product: how detailed the caption should be. A search index wants dense nouns (more matchable words is better; whether it reads like a human is irrelevant). Accessibility wants one complete, accurate, non-speculative sentence. Social contexts want a little tone on top. One model rarely optimises all three — so most systems keep one visual representation and vary the prompt and decoding parameters per surface, rather than training three models.
Three landing points already running at scale: accessibility (a platform auto-fills a caption for images that shipped without alt text, and the screen reader speaks it), image search (push generated descriptions into an inverted index and pictures become findable by text — the core requirement in e-commerce and stock libraries), and content understanding upstream (turn the image into text first, and moderation, tagging and recommendation can all reuse the existing text pipeline instead of building a parallel multimodal stack). The third is the valuable one: in many companies captioning is not a user-facing feature at all, it is the adapter that plugs images into the text world.
3.3 million pairs. The real work on the data side is not collection, it is filtering.google-research-datasets/conceptual-captions ↗1. The chapter is not about "understanding the image" — it is about cross-modal alignment and scoring without ground truth.
2. Labels drop relations: "person throws frisbee to dog" and "dog brings frisbee to person" share a label set — what you need is a sentence with a verb.
3. The skeleton is vision encoder + projection/adapter + language decoder, with cross-attention moving as the sentence advances.
4. Scale sense: 224×224 → 196 visual tokens, a caption of just 10–20 tokens, tens of milliseconds per image; go to 448×448 and attention cost grows with the square of token count.
5. The default start is freeze both ends, train the layer in between: a few percent of parameters, and swapping components is cheap.
6. Human data is clean but capped in the hundreds of thousands; web alt text is billions and filthy — the craft is in the filtering pipeline, and synthetic backfill spends compute instead of annotation.
7. Teacher forcing buys speed and leaves exposure bias; beam search is safe and dull, sampling is vivid and hallucinates — accessibility takes the former.
8. Evaluation runs in three generations: n-gram overlap (CIDEr suppressing boilerplate via TF-IDF) → scene-graph reconciliation (SPICE) → reference-free fit (CLIPScore). They prop each other up; none concludes alone.
9. The launch gate is human review plus the object hallucination rate: a fluent error is far more dangerous than a clumsy correct one.
10. The product red line: describe what is visible, never infer identity — models copy co-occurrence in the data straight back out as rule.