Text-to-Image: the hard part is not looking good, it is obeying

In one sentence

The previous chapter solved "fast and large"; this one supplies the other half: control. The answer is two moves — cross-attention injects the text into every denoising layer so each position can look back and query the sentence, and classifier-free guidance (CFG) extrapolates the "obedience direction" by a multiplier at sampling time, turning compliance into a dial you can turn live without retraining. What is most often underrated here is the layers outside the model: safety, rewriting, negative prompts and caching.

Glossary

Where it sits

Question 8 of the ten case studies in Part II, and the middle link of the book's image trilogy: Ch7 (GAN faces) solves looking real, Ch8 (latent diffusion) solves being affordable, this one solves being right, and it sets up Ch10 — where the condition stops being a sentence and becomes a handful of selfies. It maps onto the full pipeline behind Stable Diffusion, DALL·E, Imagen and Midjourney.

The problem this chapter attacks

The diffusion model from the last chapter gives you a beautiful but ungoverned image. What the product wants is "a shiba inu wearing an astronaut helmet, watercolour" — the model has to land subject, attribute, relation and style all at once, and missing any one of them means the user rerolls.

What happens if you do not solve it? You get the most expensive failure mode in generative systems: fluently wrong — no error, no blank, just a quiet omission of "wearing a helmet" at first-rate image quality. Worse, the automatic metrics cannot see it: FID measures whether a batch of images looks like real photographs, so a high-quality image that ignores the prompt entirely can still post a beautiful FID. So this chapter's problem is really a pair: how the condition gets in (mechanism), and how you measure whether it was heard (evaluation).

A cost constraint squeezes the design space further. A 512×512, 50-step request takes seconds on a single A100 — and CFG below runs the network twice on every step, so per-request cost is multiplied by 2. Here, "how obedient" is not an academic parameter; it is a dial billed by the multiple.

The core, properly

1. Turn the sentence into vectors: three choices of text encoder

Step one hands the prompt to a text encoder — usually an off-the-shelf language model kept frozen during training — which emits one vector per token (77 token slots in the Stable Diffusion line). That sequence is the "condition" everything downstream reads. Not training the encoder, and letting the diffusion network learn to read it, saves compute and makes the text side swappable as a block — which is exactly why picking an encoder is a genuine system decision. Three routes:

• CLIP's text tower: trained jointly with images, so it natively "sees" visual semantics — hear "golden retriever" and a picture is already there. Weakness: it mostly saw short captions, so long sentences, negation and grammatical structure are shaky.
• A pure text LLM (the T5 family): far better at long sentences, subordinate clauses, counting and spatial relations. Weakness: it has never seen an image, so the diffusion network must map language space onto vision by itself, which makes training harder.
• Both: concatenate the two outputs so the diffusion network gets visual intuition and grammatical precision together. This is SDXL's route.

A useful analogy: CLIP's text tower is an art director who has seen ten million captioned pictures — one word and they know roughly what it looks like, but they lose the thread of a long sentence. T5 is a literary editor who has only ever read books — impeccable at parsing structure, with no picture in their head.

2. Cross-attention: the condition is re-queried at every layer, not stated once

Given the text vectors, the naive move is to add them to the network's input. That works badly — it is a one-shot, global remark, and by the time the network is painting a local patch it no longer remembers.

What works is a cross-attention layer inside every block of the denoising network (U-Net or DiT): each position in the latent emits a Query, the text tokens emit Key / Value. In plain terms — while working on the top-left patch, the network goes back and asks the sentence again: "what should this patch look like?", then fetches information from each word by relevance.

This explains a lot. Why does "helmet" land on the dog's head rather than smearing across the canvas? Because attention is inherently per-position asking, so a soft "who listens to whom" alignment table grows on its own during generation; the hat region and the scarf region are querying different words of the same sentence.

The cost is small: text is only a few dozen tokens, so this layer is nearly free next to self-attention over the whole image. The trap is elsewhere — that alignment table is soft and unsupervised, so "red hat, blue scarf" often swaps into a blue hat and a red scarf, and "three cats" often comes out as two. Attribute binding and counting are structural weaknesses; no amount of tuning fixes them.

3. Classifier-free guidance: extrapolate the obedience direction

With cross-attention alone, the model treats the prompt more like a suggestion. To push compliance to the limit, the chapter's protagonist arrives.

Its predecessor is classifier guidance: train a separate model that can classify noisy images and use its gradient to steer sampling toward a target class. Two problems — you must train an extra noise-aware classifier, and it can only steer toward a "class", never an arbitrary sentence.

CFG's idea is to throw that classifier away entirely. During training, randomly drop the condition (about one sample in ten is fed an empty prompt), so one set of weights learns both "denoise while reading the prompt" and "denoise reading nothing". At sampling time you run both, get two predictions, and extrapolate along their difference:

ε̂ = ε_uncond + w · (ε_cond − ε_uncond)

In plain terms: the difference between the two predictions is the direction this sentence drags the picture in, and w is how many times you amplify that pull. w=1 is no amplification (ordinary conditional generation); w=7 means travelling six times further along it. Intuitively it tells the model: don't just consult the sentence — go further toward looking like it.

Three properties to remember. First, it lives at sampling time and never touches the weights — the same checkpoint can expose w as a user-facing slider. Second, it costs two forward passes per step, doubling compute and latency (batching the two branches together absorbs some overhead, but the FLOPs really are doubled). Third, turning it up is not free money: adherence rises while diversity and realism fall — oversaturation, formulaic composition, and at extreme w the numerical blowouts that Imagen's dynamic thresholding was introduced to suppress.

4. Who owns the prompt at the system layer

This is the most system-design-ish section of the chapter, and the easiest to dismiss as product detail. The user's sentence cannot go straight to the model; at least four layers sit in between:

• Safety on both sides. Filter the prompt on the way in (classifier plus rules), and filter the generated image on the way out (adult content, real people's likenesses, trademarks). Both are required: text filters are trivially bypassed — respell it, use a metaphor, split it into pieces, and the model still understands.
• Prompt rewriting / expansion. A user writes "a cat", while the captions the model trained on were long and specific. Putting an LLM in between to expand a short prompt into a detailed description improves adherence substantially. The cost is that you have altered the user's intent, so you generally echo the rewritten prompt back and leave a "use my wording" switch.
• Negative prompts. A counterintuitive free lunch: nothing says CFG's "unconditional" branch has to be an empty prompt. Point it at "blurry, extra fingers, watermark" and the extrapolation direction automatically becomes "away from these". It is not folklore, it is the same formula read backwards — one line of code, zero extra compute, since that pass runs anyway.
• Determinism and caching. Make (prompt, seed, steps, w, model version) the cache key: the same inputs must yield the same image. One design buys three things — reproducibility (support can replay what the user saw), auditability (you can trace which version produced it) and savings (a hit skips the entire denoising loop). In a product with viral prompts, this is the single biggest saving.

Trade-offs and how to choose

Two decisions in this chapter are actually actionable: what to use on the text path, and how high to set w.

Table 1 · Choosing a text encoder

Table 2 · Setting the CFG scale w

One trade-off will not fit in a table: which side rewriting lives on. Inference-side expansion ships fast and rolls back instantly, but adds a hop of latency and a token bill to every request; training-side (retraining on detailed synthetic captions) is more thorough but takes weeks and cannot be reverted quickly. Most teams ship the inference side first and only then consider moving it into training.

Why it matters / in real systems

Asked in an interview "how do you make it obey better", answering "use a bigger model" is not an answer. This chapter gives three separable handles: swap the text encoder (expensive, needs retraining), change how the condition is injected (expensive, touches architecture), tune w and add a rewriting layer (cheap, shippable today). Read right to left by cost, that is an action order you can recite.

Evaluation is equally examinable. Reporting one number is wrong: FID measures realism and is nearly blind to obedience; CLIP score measures adherence but can be gamed by painting the keywords in literally. Worse, the two move in opposite directions along w — turn w up and CLIP score rises while FID degrades. So you sweep w and report a curve, not a point, and still backstop it with human evaluation.

Pitfalls and disputes

• Treating CFG as a quality dial. It is an adherence dial. Turning it up often degrades quality — oversaturation, formulaic composition. When users complain the images look bad, check w before reaching for a different model.
• Assuming CFG is free. Two forward passes per step means compute ×2 per request. Capacity estimates that count one pass will overstate throughput by half.
• Attribute binding, counting and text rendering are known weaknesses. The cross-attention alignment table is soft and unsupervised, so "red hat, blue scarf" swapping colours and "three cats" coming out as two are structural, not tuning problems. Text inside images stayed weak for a long time, improved mainly by stronger text encoders and character-level handling.
• Treating negative prompts as folklore. They merely replace CFG's unconditional branch with a sentence; once you understand the formula you know what belongs in there (things to move away from) rather than copying incantations.
• Dated in two ways. The book was written in 2024; since then backbones have largely moved from U-Net to DiT, and few-step distillation has made 1–4 step generation routine — distilled models often bake guidance into the weights, so the external w no longer implies two passes. The skeleton of "cross-attention injection plus sampling-time extrapolation" has not changed.
• Training-data licensing remains unresolved. The technical fact that "the model stores parameters, not a library of images" is not an answer to the licensing question — that is a separate axis, and it should not be brushed aside with it.

Takeaways