CS PAPERS DEEP-READ · PAPER 28
Ho, Jain & Abbeel · UC Berkeley · NeurIPS 2020
Today you can type a sentence and watch an AI paint a picture — Midjourney, Stable Diffusion, DALL·E. The machinery underneath is called a diffusion model. In 2020, three people at Berkeley (Ho, Jain, Abbeel) wrote this paper, DDPM, and made an old idea finally work: let a machine start from a screenful of TV static and slowly rub a photograph out of it — a photograph of something that never existed. It was the first time this approach beat the reigning champion on picture quality, and nearly every AI image and video product since has been built on it.
Before this, the dominant way to teach a machine to paint was forger versus inspector: one network fakes pictures, the other calls out the fakes, and the two fight their way to competence. The pictures came out gorgeous, but training was a tightrope walk — tip the balance slightly and it collapses — and the forger often learned to paint only a handful of things.
DDPM took a completely different route, one that is almost funny in its plainness: learn to destroy first, then play the tape backwards.
Take a sharp photo and sprinkle a thin layer of sand over it. Then another layer, and another. After a thousand layers the photo is gone — nothing but static. Nobody needs to learn how to do that; ruining a picture is free.
Here's the point: play that tape backwards and you have a photograph growing out of static. "Paint a picture from nothing" is impossibly hard, but "brush off the one thin layer of sand that was just sprinkled on" is easy. The hard job has been cut into a thousand nearly thoughtless small ones — and the small one is all the machine ever has to learn.
Training is almost cheating in its simplicity. Take a real photo, pick a grubbiness level at random, and sprinkle that much sand on in one go — and because we sprinkled it ourselves, we are holding the answer key. Hand the network the dirty picture along with "this is grubbiness level 400" and ask it exactly one question: what does the sand on here look like? It points; we compare against the answer key; we nudge it. No opponent, no game — one fill-in-the-blank question with a known answer. That is why it trains so calmly.
Generation runs the same machinery backwards. Grab a fistful of static, ask "which of this is sand," subtract a little, ask again, subtract again — a thousand rounds later, a photograph nobody ever took has surfaced. Why does something new come out instead of one of the training photos? Because the starting point is a random fistful of static, and each step only pushes the picture a little way toward "looking more like a real photograph." Different start, different ending — like staring at clouds: the clouds are random, but you always end up seeing something specific in them.
No more tightrope: training is stable, and the pictures come out both varied and clean. On the same dataset it overtook the best adversarial model of its day. More importantly the recipe kept growing — bolt on "brush according to this text description" and you get the type-and-get-a-picture products of today; video generation runs on the same idea.
The honest cost is a single line, but a heavy one: producing one picture means running the network back and forth a thousand times, hundreds of times slower than the one-shot rival it beat. Much of the follow-up research since has been about squeezing those thousand steps down to a few dozen.
Turn "paint a picture from nothing" inside out: take a thousand small steps to sprinkle a photo into static (no learning needed), then train a network to do one job — spot the noise added at each step. To generate, start from random static and repeatedly subtract the noise the network points at, and a picture grows. Stable to train, diverse in output; the price is a thousand passes per image.
Want the noising/denoising chain diagram, the objective, and the numbers? → Switch to the deep read
DDPM casts image generation as a fixed noising chain plus a learned denoising chain: a thousand small steps blur a real image into pure Gaussian noise, and a neural network is trained to walk that chain back. The paper's decisive move is reparameterizing the training objective into "predict the noise that was added at this step," which collapses an intimidating variational bound into a plain mean-squared-error regression. It was the first time a diffusion model matched and beat GANs on sample quality (unconditional CIFAR-10: Inception Score 9.46, FID 3.17), and it is the methodological foundation of every text-to-image and text-to-video system in use today.
By Jonathan Ho, Ajay Jain and Pieter Abbeel (UC Berkeley), NeurIPS 2020. It inherits from two lines of work: Sohl-Dickstein et al. (2015), who framed generation as diffusion borrowed from non-equilibrium thermodynamics — the idea was there, the samples were bad — and Song & Ermon (2019), whose score-based models sampled via Langevin dynamics. DDPM welded the two together and got them working. What it launched needs no introduction: DDIM's few-step sampling, classifier-free guidance, Latent Diffusion / Stable Diffusion, DALL·E 2, Imagen, and today's video generators.
Around 2020 generative modeling had three main roads, each with a hard defect. GANs produced the best-looking images, but training is a two-network game that collapses on a bad hyperparameter, tends toward mode collapse and poor coverage of the data, and offers no comparable likelihood number. VAEs and autoregressive models trained stably with a clean likelihood story, but were respectively blurry and painfully slow to sample. Diffusion had existed since 2015 with an elegant theory — the objective is an honest likelihood bound — but its samples were bad enough that almost nobody took it seriously.
So the real question was not "is there a third road," but: why does this theoretically clean road produce such poor images, and can we reach GAN-grade quality without adversarial training? DDPM's answer: the fault lies in how the model is parameterized and how the loss is written, not in the framework.
Take a real image x₀ and define a fixed T-step process (the paper uses T = 1000). Each step mixes in a little Gaussian noise and shrinks the remaining image slightly; the strength comes from a schedule β₁…β_T fixed in advance (linear from 1e-4 to 0.02). After 1000 steps the image is pure Gaussian noise x_T, carrying nothing of the original. This chain has no learnable parameters — it is simply the destruction we chose.
One property matters enormously: because every step is Gaussian with a fixed strength, any intermediate noisy image can be computed in one shot, with no iteration:
x_t = √ᾱ_t · x₀ + √(1−ᾱ_t) · ε, where ε is standard Gaussian noise and ᾱ_t is a coefficient derived from the schedule that decreases with t. In words: the image at step t is the original faded down plus a proportional dose of noise mixed in; the larger t, the fainter the original and the heavier the noise. This is what lets training pick a random t and instantly produce both the noisy image and its answer key.
Generation walks the chain backwards: start from pure noise x_T and repeatedly ask what the slightly cleaner previous image looked like. The good news is mathematical: when each forward step adds only a little noise, the reverse step is itself approximately Gaussian. So the network never has to emit a complicated distribution over whole images — only the mean of a small Gaussian. (The variance is simply fixed to a constant, β_t or a close variant; the paper found both give comparable sample quality.)
This is the entire payoff of chopping the problem up: modeling "the distribution of all real images" in one shot is hopeless, while "given a slightly dirty image, guess a slightly cleaner one" is local, gentle, nearly linear. A thousand of those small tasks in series is equivalent to the one impossible task.
In the variational bound, each term pushes the network's predicted mean toward some true posterior mean. The paper performs one algebraic rearrangement: since x_t is built out of x₀ and the noise ε, that true mean can be rewritten as a combination of x_t and ε. So let the network predict the noise ε directly and recover the mean by formula. The loss collapses to a single line:
L_simple = ‖ ε − ε_θ(√ᾱ_t·x₀ + √(1−ᾱ_t)·ε , t) ‖²
In words: draw a training image at random, a step count t at random, and a blob of noise at random; sprinkle it on; show the network the dirty image and t; ask it to point out the noise. The closer it points, the smaller the loss. No adversary, no sampling loop inside training, no partition function — an ordinary mean-squared-error regression, as stable as training a classifier.
Why does this rewrite matter so much? Two reasons. First, it makes the target a scale-stable quantity: whatever t is, the answer is always a unit Gaussian blob, so the network never has to chase wildly varying magnitudes. Second, the paper shows this form is exactly equivalent to denoising score matching: predicting the noise is, up to a constant, estimating "which direction makes this image more like a real image" — the score — and the sampling loop becomes annealed Langevin dynamics. One equation joins the variational-bound view and the score-matching view.
There is also an accidental gift. L_simple amounts to throwing away the t-dependent weights in the variational bound, and the effect is to down-weight the small-t terms — the nearly clean images that differ only in fine detail — letting the network spend its capacity on the large-t steps where the noise is heavy and the job is genuinely hard. The paper measures both: training the strict bound gives better likelihood numbers; training L_simple gives much better pictures.
Predicting noise is an image-in, image-out task, so the backbone is a U-Net: downsample to capture global structure, upsample to restore detail, with skip connections between matching resolutions so detail isn't lost. The paper adds group normalization, inserts self-attention at the 16×16 resolution to capture long-range relationships, and injects the step index t as a sinusoidal embedding into every residual block — all 1000 steps share one network, with t telling it how heavy the noise it is facing should be. Parameter count is therefore independent of chain length.
Sampling loops t = T … 1: feed the current x_t and t to the network, subtract the appropriate share of the predicted noise, rescale, and — except on the very last step — mix a little fresh random noise back in. That injected randomness is not optional: it is what lets the same starting point end at different images, and it is precisely the perturbation term of Langevin sampling. Run all T steps and an image comes out.
The main battleground is unconditional generation on CIFAR-10 (50,000 training images at 32×32). DDPM reaches Inception Score 9.46 and FID 3.17, better than every published model at the time, GANs included — the first time a diffusion model won on image quality outright, and the source of the paper's influence. On LSUN bedrooms and churches at 256×256, sample quality lands in the same league as ProgressiveGAN.
The paper is also honest about an unflattering number: as a likelihood model, its lossless codelength is about 3.75 bits/dim on the test set, no better than contemporary likelihood-focused models. The authors' explanation is the interesting part: broken down, the overwhelming majority of those bits go to imperceptible fine detail, while the coarse structure that decides whether an image "looks real" costs very little. That is a concrete demonstration that good likelihood ≠ good pictures, and it explains why the weight-discarding L_simple produces better samples. Two ablations also matter: predicting ε clearly beats predicting the mean directly, and fixing the reverse variance to a constant is sufficient — learning it was unnecessary. The paper further demonstrates progressive lossy compression and decoding, and shows that interpolating in the noised latent space yields semantically smooth transitions between images.
It took diffusion from "elegant but useless" to "state of the art" in one step, and the recipe transfers unusually well: the objective is plain regression, training is stable, scaling up helps, and mode collapse is structurally absent — the model is fitting the whole data distribution in a likelihood-style objective rather than out-maneuvering a discriminator. Within a year came Nichol & Dhariwal's improvements (learned variances, cosine schedule), DDIM's deterministic few-step sampling, and Diffusion Models Beat GANs with classifier guidance. Then classifier-free guidance made text conditioning simple, and Latent Diffusion (Stable Diffusion) moved the whole process into a compressed latent space, bringing the cost down to consumer GPUs. DALL·E 2, Imagen, Midjourney and today's video generators all trace their method back here. A decade of GAN dominance in generative modeling ended at this paper.
L_simple is empirically good rather than derived. Why that weighting works better had no principled answer at the time.① One sentence: a fixed thousand-step chain blurs images into Gaussian noise; a network is trained to play it backwards; the key is rewriting the objective as "predict the noise added at this step."
② The pain: GANs look great but train unstably and collapse modes; VAEs and autoregressive models are stable but blurry or slow; diffusion existed since 2015 with terrible samples.
③ The forward chain has no parameters and a closed form, x_t = √ᾱ_t·x₀ + √(1−ᾱ_t)·ε — training draws a random t, builds the noisy image instantly, and gets the answer key free.
④ Each reverse step is approximately Gaussian, so the network need only emit a mean with the variance fixed — one impossible task becomes 1000 gentle ones.
⑤ The decisive move: predict the noise, collapsing the loss to ‖ε − ε_θ(x_t,t)‖², a plain regression — and equivalently denoising score matching, making sampling an annealed Langevin walk.
⑥ Dropping the bound's t-weights down-weights the near-clean steps, focusing capacity on heavy-noise steps: slightly worse likelihood, clearly better images.
⑦ Implementation: U-Net with self-attention at 16×16 and a sinusoidal embedding of t; one parameter set for all 1000 steps; sampling mixes fresh noise back in at each step.
⑧ Results: unconditional CIFAR-10 IS 9.46 / FID 3.17, the first diffusion win over GANs on quality; LSUN 256×256 on par with ProgressiveGAN; likelihood ~3.75 bits/dim, not a lead.
⑨ Impact: DDIM, classifier-free guidance, Latent/Stable Diffusion, DALL·E 2, Imagen and video generation all descend from this method.
⑩ Limits: a thousand passes per sample; likelihood not competitive; the weighting simplification is empirical; the original is unconditional and small-scale; misuse is a real cost.