Book Deep Read · Generative AI System Design Interview · Ch 7
Generative AI System Design Interview · Ch 7 · Ali Aminian, Hao Sheng & Alex Xu · 2024
You have seen these photos online: someone smiling naturally, every hair, pore and earlobe lit correctly — and that person has never existed. This chapter is about how a machine paints a real face out of nothing, and about a harder question: once it is painted, who says whether it is any good?
Every other task has a right answer. Cat or dog — right is right. But "paint a face" has none. You cannot point at any photograph and say "this is what you should have painted." No answer means no score, and no score means no way to tell the machine which way to move next.
The early idea was to have it paint toward the average of everyone. The result is absurd: average ten thousand faces position by position and you get a grey smear with the features melted together — averaging ten thousand faces never gives you a good face, only a face that is nobody. Want sharpness, stop averaging; stop averaging, and you are back to having no way to score.
The fix is strange: if nobody can be the judge, then train a judge too.
Put two people in a room. One is a forger, starting out by smearing paint around blindly. The other is an appraiser, with a stack of genuine photographs on the desk and whatever the forger just handed over, and exactly one job: point at the fake. The forger wants to fool the appraiser; the appraiser wants not to be fooled. The beauty is that the appraiser improves too — a blur gets through today; tomorrow the appraiser has learned to read the eyes; the day after, ears and hairlines. The yardstick keeps getting stricter, so the forger gets pushed all the way to indistinguishable.
Later the forger was given a layered palette: settle the big things first (which way the head faces, the shape of the face), then the middle (features, expression), and only then the fine things (skin tone, stray hairs, pores). To change the hairstyle without touching the face shape, you turn only the middle dial — that is where control comes from. There is one more handy trick: have it paint a little closer to the average face, so it makes fewer mistakes and looks more convincing; the price is that everyone it paints starts to look faintly related.
Still not by looking at one picture. Lay out fifty thousand real photographs and fifty thousand painted ones, and compare the character of the two piles — the real pile has old and young, profiles, people in hats; if the painted pile spreads just as wide and holds just as much detail, it passes. This is the general move against "no right answer": one picture cannot be graded, a crowd can.
The gain is speed: once trained, a face comes out in the blink of an eye, which is why live avatars, stock imagery and using synthetic faces in place of real ones for testing all lean on it. The cost is one sentence, and a hard one: the sparring can break down at any moment — most often by the forger discovering one face that always works and painting nothing else ever again.
You think the hard part is painting something lifelike. The hard part is that nobody can say what "lifelike" means. The whole cleverness of this chapter is turning an unstateable standard into an opponent that is also learning.
Want the mechanisms, notation and diagrams? → switch to the deep read
You think the invention behind the generative adversarial network (GAN) is the generator that paints, but the real invention is handing the loss function itself to a second network to learn — because "does this face look real" cannot be written as a formula, so you train an ever-stricter discriminator to serve as the yardstick. The engineering line worth keeping from the whole chapter: the yardstick is learned, so it wobbles, it can be gamed, and it cannot be read as a progress bar. GAN's three diseases (non-convergence, mode collapse, discriminator overfitting) and distribution-level metrics like FID are all the bill for that one choice.
512 dimensions is typical for faces) that seeds one image; the same z always yields the same face.ψ): at generation time, pull the latent toward the "average face"; the harder you pull, the safer and the more monotonous.The sixth of the ten case studies in Part II, and the opener of the image-generation trilogy (Ch7 faces, Ch8 high-resolution synthesis, Ch9 text-to-image). It follows Ch5 image captioning — image in, text out — and inverts it: nothing in, an image out. This chapter takes the GAN route; Ch8 switches to diffusion, and only reading both shows you the route war. In the real world it maps to avatars and stock imagery, de-identification of face data, and the risk engineering around deepfakes.
The core tension in one line: discriminative tasks have labels, generative tasks do not. "Does this X-ray show a lesion" has one answer you can check against. "Paint a face" has none — given a random seed, no photograph is what it should have become. No right answer means no error, which means no objective to descend.
What happens if you force a per-pixel error anyway? A blurry face. The reason is plain: the distribution of real faces is multi-modal (profiles and frontals, long hair and shaved heads), and the minimiser of pixel-wise mean squared error is the average over all possibilities — averaging ten thousand faces gives not a good face but a fog that is nobody. Sharpness requires giving up averaging.
So what this chapter must deliver is an optimisable objective that pins down two things which usually fight: fidelity (one image holds up) and diversity (a batch spreads out and covers the real population). What if you don't? Iteration falls back to people eyeballing grids, the feedback loop stretches from minutes to days, and two reviewers can reach opposite verdicts on the same batch.
What it is: two networks. The generator G turns a random vector z into an image; the discriminator D looks at an image and answers "what is the probability this came from the real dataset". They update in alternation: with G frozen, feed D real and fake images so it learns to separate them; with D frozen, push G in whatever direction raises D's score on its output.
The intuition: a forger sparring with an appraiser. What matters is not how strong the forger is but that the appraiser improves too — a blur passes today, tomorrow D has learned eye symmetry, the day after it starts picking at ear shapes and hairlines. The yardstick itself tightens, so there is never a day when the generator is "done".
Notation: usually written min_G max_D V(D,G), which in plain words is "D tries hard to separate real from fake, G tries hard to make it fail". Note this is not minimising a function; it is searching for an equilibrium between two opponents — and that single fact causes everything that follows.
Why design it this way: because nobody can write "looks like a real face" as a formula. Hand-written metrics (pixel difference, edge similarity) can all be fooled by something that satisfies the metric and looks nothing like a face. If you can't write it, learn it — that is GAN's central move, and the source of its entire bill.
Non-convergence and oscillation. Ordinary training walks downhill, so a falling loss means progress. Adversarial training is a tug of war, and a falling G loss usually just means D went soft this minute. Hence GAN's most counter-intuitive rule of practice: the loss curve is not a quality progress bar; judgement comes only from sampling images regularly plus distribution-level metrics.
Mode collapse. If one kind of face reliably fools the current D, G dives into it and maps every z onto those few faces. What makes it lethal is that fidelity is unharmed — every image is sharp and attractive, they are simply all alike, and a demo will never reveal it.
Discriminator overfitting. With little data, D starts memorising the specific training images instead of learning "realness"; once the yardstick is meaningless, training diverges.
The fixes, one line each: the non-saturating loss rewrites G's objective so it still has usable gradients while being crushed; WGAN-GP's gradient penalty and spectral normalisation cap how steeply D may change, keeping the yardstick from jumping; TTUR (two different learning rates) lets D learn slightly faster, maintaining an opponent that is useful but not overwhelming; discriminator augmentation applies random transforms to every image D sees — real and fake alike — forcing it to learn features rather than memorise pictures.
In a plain GAN, z is entangled: nudge one number and face shape, hair colour and pose all shift together — a box you can only shake as a whole. StyleGAN fixes this in three steps.
Disentangle first: z no longer enters the painting pipeline directly. It passes through a small mapping network and becomes w, a space in which attributes that were crowded together get spread apart, making "change only the hair colour" possible.
Then inject per layer: the image is grown from 4×4 up to 1024×1024, and w is injected once at every level (via AdaIN — w sets that layer's feature mean and variance, i.e. that layer's style). Control therefore comes pre-sorted by granularity: low-resolution layers govern pose and face shape, middle layers features and expression, high-resolution layers skin tone and hair. "New hairstyle, same person" works precisely because those things were always handled by different layers.
Finally, separate the randomness: pore placement and the drift of stray hairs — details that should differ on every draw without changing who this is — travel on their own per-layer noise channel instead of consuming w's capacity.
The truncation trick: at generation time, pull w toward the training set's average w, with the strength written ψ. A smaller ψ sits closer to the average face, visibly cuts the failure rate and reads as more convincing; the price is that everyone starts to look faintly related. This is the dial to remember from the whole chapter — there is no switch between fidelity and diversity, only a continuous slider — and almost every stunning face demo online is running a low ψ.
With no right answer for a single image, evaluation has to move up a level: compare distributions, not pictures.
The mechanism, in plain words: take 50,000 real images and as many generated ones, push both through an off-the-shelf image classifier (Inception) and pull 2048-dimensional features from a middle layer; treat each pile of features as one high-dimensional Gaussian, fitting a mean and a covariance; measure the distance between the two Gaussians. The smaller the number, the more the two piles look like the same crowd in that feature space.
Why design it this way: the metric punishes both failures at once — blur shifts the mean, collapsed diversity shrinks the covariance. One number, both ends covered.
The costs and traps: precisely because one number covers both ends, when it goes bad you cannot tell which end broke. Production therefore adds a layer: precision and recall measure fidelity and coverage separately (low precision = oddities, low recall = whole segments of the population missing). It also leans on an ImageNet-trained network as its yardstick, which need not be sensitive to facial detail; and change the sample count or the implementation and the number moves — FID is only meaningful compared within one evaluation pipeline; numbers copied across papers do not line up.
The first choice is the route war. When an interview asks "why a GAN", this table is the answer.
| Route | Cost per image | Training stability | Coverage / diversity | When to pick it |
|---|---|---|---|---|
| GAN | 1 forward pass; a 1024² face in tens of milliseconds on one modern GPU | Worst: all three diseases must be defended against | Weaker; mode collapse is a standing risk | Latency and unit cost are hard constraints: live avatars, super-resolution, on-device |
| Diffusion | 20–50 steps of iterative denoising; one to two orders of magnitude slower | Good: essentially a regression task, trains stably | Best; covers the distribution naturally | Quality and controllability first, seconds of latency acceptable: the text-to-image workhorse (Ch8–9) |
| VAE | 1 forward pass, fastest | Best | Good, but inclined to average | You want a compact representation more than top-tier fidelity; also the latent compressor under diffusion |
The costs sit in columns two and three: GAN buys fast inference with hard training; diffusion trades the other way.
The second choice happens inside one model — where you set ψ. It is not a tuning detail but a product decision:
| Scenario | ψ leans | What you get | What it costs |
|---|---|---|---|
| Landing page demo | Low (conservative) | Almost no oddities; every image presentable | Homogenised faces that all look related |
| Synthetic training data | High (varied) | The long tail gets covered, so downstream models don't skew | More rejects; you need automatic filtering behind it |
| Chasing FID in a paper | Tuned to the sweet spot | The best-looking metric | That ψ need not be the ψ your product should ship |
Same weights, different ψ, two different products — a common source of "great FID in the paper, ordinary feel in production".
The famous deployment is NVIDIA's StyleGAN family and the "this person does not exist" style demo page, which made "synthetic faces have passed the naked eye" common knowledge. Three uses are real work: avatars and stock imagery (games, e-commerce models, profile pictures — no shoot, no licensing); de-identification of face data (synthetic faces standing in for real ones in demos and tests, sidestepping consent and compliance); and data augmentation (long-tail samples for downstream models). Realism is itself the risk surface: the more convincing it gets, the cheaper impersonation becomes, so provenance and watermarking belong in the design up front, not as an afterthought.
The interview question this chapter really feeds is not "how do you train a GAN" but "how do you prove a model with no right answer got better" — and the answer is layered: a distribution metric (FID) for the whole, precision/recall to split fidelity from coverage, and blind human comparison as the final arbiter. None of the three is optional.
ψ with hand-picking. Evaluate unfiltered random batches — the cheapest and sharpest question in any due diligence.① The invention here is not the generator but handing the loss function to a second network, because "looks like a real face" cannot be written as a formula.
② Generative tasks have no right answer; forcing per-pixel error makes the average over all possibilities optimal, which is a blurry face that is nobody.
③ The objective must hold two things that fight: fidelity (one image holds up) and diversity (a batch spreads out).
④ G and D alternate toward an equilibrium, not a minimum — so a GAN's loss is not a quality progress bar.
⑤ Three diseases: non-convergence, mode collapse (cuts only diversity, invisible in a demo), and discriminator overfitting (divergence when data is scarce).
⑥ Each stabiliser covers one stretch: non-saturating loss keeps gradients, gradient penalty and spectral norm cap the yardstick's steepness, TTUR sets the pace, discriminator augmentation stops memorisation.
⑦ StyleGAN returns control in three steps: a mapping network to disentangle, w injected per layer (coarse for pose, fine for hair), and noise on its own channel.
⑧ Truncation ψ is a continuous slider between fidelity and diversity, not a switch; a beautiful demo is mostly a low ψ.
⑨ FID compares distributions, not images: fit a Gaussian to each pile and measure the gap — one number for both failure modes, which is also why it cannot say which one broke, and why it does not transfer across implementations.
⑩ Selection in one line: GANs buy fast inference with hard training (one forward pass, tens of milliseconds), diffusion buys stability and coverage with dozens of steps; and the more realistic the output, the earlier provenance and watermarking must enter the design.