High-Resolution Image Synthesis: the hard part is not painting finer — it is not painting at full size

In one line

This chapter turns the diffusion model into a system that can ship high-resolution images, and its central claim is counter-intuitive: the bottleneck at high resolution is not "can it paint fine detail" but "how large a canvas are you repeating those tens-to-hundreds of steps on". So the answer comes in two halves — first rewrite one-shot generation as a long chain of easy denoising steps (buying training stability), then move that whole chain onto a 64×64 latent (buying affordable compute). The mental model to walk away with: the design variables of a generative system are not "which model" but "how big the canvas, how many steps".

Glossary

Where it sits

This is the seventh of the ten case studies in Part II. It follows Ch 7 on face generation with GANs — where a generator and a discriminator fight, training is unstable and mode collapse looms; here the adversarial game is replaced by a plain regression task. It sets up Ch 9 on text-to-image, which bolts text conditioning and classifier-free guidance onto exactly this skeleton. Real systems it maps to: Stable Diffusion, DALL·E 2, Imagen, Midjourney — the whole current generation of image products.

What this chapter is trying to solve

Two pains: one about unstable training, one about unaffordable compute.

The instability is inherited from the previous chapter. A GAN optimises a game between two networks; whenever one pulls ahead the other's gradient goes flat, the balance gets harder as resolution climbs, and mode collapse (it only ever paints the same few faces) is always in reach. The engineering consequence is that training becomes unpredictable — same code, same data, a different random seed, and the run is wasted. For a team that has to schedule and reproduce work, that is fatal.

The compute problem belongs to this chapter. A 512×512 RGB image is 512×512×3 = 786,432 numbers. Diffusion runs the denoiser T times, and the original DDPM used T=1000 — a thousand full-size forward passes per image. Worse, self-attention inside the U-Net grows with the square of the number of positions (O(n²) — double the positions and the cost quadruples): 512×512 is over 262,000 positions, so global attention at that scale simply does not work in practice.

What happens if you do not solve it: you are stuck training 64×64 models and propping them up with a stack of super-resolution stages — each trained separately, deployed separately, each compounding the previous stage's errors. Image generation stays a game only for companies with large datacentres.

The core, properly

1. Rewrite "generate" as "denoise": one hard job becomes a thousand easy ones

A diffusion model defines two processes running in opposite directions. Forward: take a real image x₀ and, following a fixed recipe, mix in Gaussian noise step by step until after T steps it is pure noise x_T. This half has no learnable parameters at all — it is an addition formula. Reverse: start from pure noise and subtract the noise back out, step by step, until an image develops. That half is what the network learns.

Why is this rewrite the bedrock of the chapter? Because it makes the supervision signal carry its own answer: pick a random image, pick a random timestep t, mix in the noise yourself — and what the network ought to output (the blob of noise you just mixed in) is known. No human labels, and no adversary network acting as judge. The reason GANs are hard to train is precisely that their "right answer" comes from another network that is also moving; diffusion swaps that moving target for a fixed one.

2. What the network actually learns is a subtraction

The training objective is so plain it hardly looks like a generative model: L = ‖ε − ε_θ(x_t, t)‖². In words — the noise ε that was genuinely mixed in, versus the ε_θ the network guesses from the noisy image x_t; the gap between them is the loss. An ordinary regression problem, with stable gradients, reproducible runs and a loss curve you can schedule against. That is its engineering advantage over a GAN.

The backbone is a U-Net: downsample to capture large structure, upsample to restore detail, with skip connections handing high-frequency information straight across at matching scales. Because input and output share a shape, it fits per-pixel prediction naturally. The timestep t is embedded into a vector and injected into every block, which tells one set of weights which step this is, and therefore how hard to wipe. That matters: all steps share a single set of parameters, or the model size would be multiplied by a thousand.

3. Latent space: move those steps onto a sketch pad

This is the real turn of the chapter. You train a separate autoencoder first: encoder E compresses 512×512×3 down to 64×64×4, decoder D restores it. The information drops from 786,432 numbers to 16,384 — a ratio of 48:1. Then the entire diffusion process runs only on the latent, and you decode exactly once at the end.

Why does that work? Most of the bits in a natural image are spent on high-frequency texture the eye barely registers, and a deterministic decoder can reproduce that well enough. The diffusion network's capacity should go into semantics and composition — what is in the picture and where it sits — not into reproducing skin pores. It also settles the attention bill: positions drop from over 262,000 to 4,096, and a squared cost falls by three orders of magnitude, which is what makes global attention viable again.

The cost has to be stated plainly: the encode/decode round trip is lossy, and the more aggressive the compression (downsampling factor f=16, f=32), the more small structure dissolves. Small text, human hands and dense regular grids are the worst hit — a fair share of this generation's signature artifacts are not the diffusion network painting wrong, but the decoder failing to bring detail back.

4. Getting to 1024: cascaded super-resolution vs a bigger latent canvas

Latent space makes 512×512 practical, but products want 1024×1024 and beyond. Two roads.

Road A · cascaded: generate at low resolution, then chain one or two diffusion super-resolution stages to enlarge (say 64 → 256 → 1024). Each stage is small, can be trained and swapped independently, and the low-resolution stage gets to spend its compute on composition. The cost is that errors propagate down the chain — if stage one paints a bad hand, the next two only render that hand more sharply — total latency is the sum of every stage's steps, and you now operate three models.

Road B · enlarge the latent: keep the pipeline and move the latent from 64×64 to 128×128 (which decodes to 1024×1024). One pipeline, end to end, no seams between stages. The cost is that VRAM and per-step time scale with latent area — double the side and the area quadruples — and the model usually needs another fine-tuning round at the target resolution, or you get duplicated composition (the same subject painted twice).

In practice the two get mixed: the main pipeline takes road B to 1024, with one lightweight super-resolution stage bolted on to reach 2048 and above.

5. Sampling steps: the one dial you can turn live

Original DDPM sampling walks all T=1000 steps. DDIM rewrites sampling as a deterministic process that can skip ahead, reaching comparable quality in 20–50 steps — and note that the model is not retrained; only the sampler changes.

What this means for system design is underrated: step count is the only parameter in the whole pipeline you can set per request, live, and trade almost linearly for latency and cost. Same weights: 20 steps for a preview, 50 for the final take, a 2.5× cost difference, and no second model to deploy. Swapping models, by contrast, means reloading weights and re-warming — minutes, not milliseconds.

The other half is batching. Every diffusion step is the same large matrix operation, so packing the same step from several requests into one GPU call saturates utilisation and lifts throughput considerably — but forming a batch means waiting, which pushes single-request p99 up. The usual answer is a batching window of a few tens of milliseconds: take a smaller batch rather than let the tail get away from you.

The trade-offs, and how to choose

This chapter really comes down to two choices: which canvas you compute on, and how many steps you take. The first fixes your cost ceiling and your quality ceiling; the second fixes per-request latency.

Table 1 · Four generation routes, and when each one is right

Table 2 · Choosing a step count (same weights, switchable in production)

One more choice that tends to get skipped: the downsampling factor f. At f=4 the decoder is more faithful but the latent is larger (4× the compute); f=8 is the industry's balance point; at f=16 and above you save compute but detail collapses badly. Picking f is drawing a line between "how much the decoder can restore" and "how long diffusion takes" — and once set it is very hard to change, because it fixes the input shape of every diffusion weight you train.

Why it matters / real systems

This chapter is not about the internals of one model; it is about the skeleton every image product built after 2022 shares: an autoencoder that fixes the canvas, a U-Net that denoises, a sampler that fixes the step count. Stable Diffusion, DALL·E 2, Imagen and Midjourney differ mainly in data, in how conditioning is injected, and in their resolution strategy — those three bones are the same. Get the skeleton straight and Ch 9 (text-to-image), Ch 10 (personalisation) and Ch 11 (text-to-video) all read as things bolted onto it.

In interviews the follow-up is almost always about cost: given a million image requests a day at 1024×1024, how do you set step count, how do you batch, when do you switch to distilled weights, where in the pipeline does super-resolution belong. Getting "steps × latent area × batch size" right to the correct order of magnitude is worth far more than reciting the DDPM derivation.

Pitfalls and open arguments

• "More steps is always better" is wrong. Returns fall off fast; going from 50 to 150 steps is often visually indistinguishable at three times the cost. Step count should be set by the use case, not by "crank it to max to be safe".
• FID is not image quality. It measures the distance between two feature distributions and is nearly blind to what people actually notice — finger counts, spelling, object counts. It is also sensitive to sample size, running systematically high with few samples, so comparing FID numbers across papers usually does not hold; it only means something within one evaluation setup. Human evaluation stays mandatory before launch.
• Blaming every artifact on diffusion. The lossy compression is a separate layer: mangled small text and scrambled grids are often the decoder failing to restore, and a bigger U-Net will not help. You have to change f or change the decoder.
• "Diffusion models retrieve and collage training images" is a misconception — they store parameters, not a library of pictures. But that does not make training-data licensing a non-issue; it is a separate argument, and a technical rebuttal should not be used to wave it away.
• Dated in places. The book was written in 2024, and two things have visibly rewritten this chapter since. First, few-step distillation (consistency models, adversarial distillation) pushed usable step counts down to 1–4, so the top band of Table 2 moved from experimental to routine. Second, the backbone is shifting from U-Net to Transformer (DiT), which scales better. The core judgement — compress to a latent first, then iterate denoising — has not changed.

Takeaways