An autoregressive model like GPT behaves like a cursor: fetch one row, then the next, where row i must wait for row i-1 to come back. Masked diffusion writes differently: first INSERT one fixed-length record with every field NULL, then run several rounds of batch UPDATE, each round filling in the few fields it is most confident about, until no NULLs remain. Text is not "written out" — it is developed, like a photograph, from a blank canvas.
The pain point is a hard wall of sequential dependency: generating N tokens autoregressively requires N forward passes, because the input for token i contains token i-1. No amount of extra GPU compresses those N steps into 10 — latency is locked by serial step count, not by compute.
The diffusion idea comes from image generation, but text is discrete, so adding Gaussian noise is meaningless (nudging "cat" with noise does not produce "dog"). D3PM (Austin et al., 2021, arXiv:2107.03006) gave the usable discrete version: noise is not jitter but forgetting — the forward process replaces a fraction of real tokens with [MASK] according to noise level t, and [MASK] is an absorbing state (once in, never out); t=0 is the original text, t=1 is an all-mask canvas. The reverse process trains a Transformer with the causal mask removed — the most fundamental structural difference from GPT: it sees left and right simultaneously, and predicts the distribution at every [MASK] position in a single forward pass.
The training objective is the surprising part. MDLM (Sahoo et al., NeurIPS 2024, arXiv:2406.07524) showed this diffusion ELBO simplifies to a weighted masked-LM loss:
L(θ) = Et~U(0,1] [ (1/t) · Σi∈masked positions −log pθ(xi | xt) ]
Term by term: t is the mask ratio sampled this step (i.e. the noise level); xt is the sequence after removing a fraction t of tokens; the bracket is simply "predict the removed tokens" — exactly BERT's training task, except the mask ratio is drawn at random per example instead of being fixed at 15%. That 1/t weight is the key: when t is small only a few tokens are removed, nearly all context is present, the example is too easy and the signal too sparse, so 1/t amplifies it and levels out each noise level's contribution to the total loss — otherwise the model only gets good at the barely-masked regime, while sampling starts precisely at t=1. In other words: training a diffusion language model is training a BERT that tolerates any mask ratio. LLaDA (Nie et al., arXiv:2502.09992) took this route to 8B and reached in-context-learning performance comparable to a same-size LLaMA3.
import torch from transformers import AutoTokenizer, AutoModelForMaskedLM name = "bert-base-uncased" # any masked LM works as a one-step denoiser tok = AutoTokenizer.from_pretrained(name) mdl = AutoModelForMaskedLM.from_pretrained(name).eval() L, STEPS = 16, 8 # canvas of 16, only 8 steps (AR would need 16) ids = torch.full((1, L), tok.mask_token_id) # t=1: canvas all [MASK] ids[0, 0], ids[0, -1] = tok.cls_token_id, tok.sep_token_id for s in range(STEPS): masked = (ids == tok.mask_token_id) if not masked.any(): break with torch.no_grad(): logits = mdl(input_ids=ids).logits # bidirectional: all slots at once prob, pred = logits.softmax(-1).max(-1) # best token + confidence per slot prob[~masked] = -1 # committed slots stay (absorbing) k = max(1, int(masked.sum()) // (STEPS - s)) # spread remaining masks over steps for i in prob[0].topk(k).indices: # commit the k most confident ids[0, i] = pred[0, i] print(s, tok.decode(ids[0])) # watch text develop out of noise
Autoregression is task scheduling under a fixed topological order: the order is hard-coded, you get no choice. Diffusion is a scheduler that may pick any task: whichever slot it is most confident about goes first. That sounds like a win for the scheduler — but every real scheduler has this flaw: it keeps picking the easy tasks and pushes the critical-path node to the very end. By the time you get to that node, everything around it is already committed and its option space has been squeezed shut.
The masked-diffusion ELBO is in fact an expectation over all generation orders — what it learns is an any-order model, of which autoregression is one fixed-order special case. The standard sampling strategy is confidence-based unmasking: each round, commit the few slots with the lowest predictive entropy, i.e. the ones the model is surest about. The intuition is appealing: settle what can be settled, and let it serve as context for the rest.
The ICML 2026 Outstanding Paper The Flexibility Trap (Ni et al., arXiv:2601.15165) identifies a structural flaw here. In reasoning tasks, the real decision points are precisely the high-entropy tokens — "therefore", "since", "let x =". Their entropy is high exactly because several legitimate reasoning branches hang off them. Confidence-first decoding systematically routes around them, filling in the unambiguous surrounding words first; by the time it comes back, the left and right context have pinned the position down, and of those several branches only one still reads coherently.
The authors' fix is elegantly small: during RL, force left-to-right exploration order (autoregressive order as a training scaffold), while inference keeps full bidirectional attention and parallel decoding. This JustGRPO recipe reaches 89.1% on GSM8K and 45.1% on MATH-500 (generation length 256). The conclusion is counterintuitive but clean: a larger hypothesis space plus a greedy order-selector can be worse than a smaller hypothesis space.
import torch from transformers import AutoTokenizer, AutoModelForMaskedLM tok = AutoTokenizer.from_pretrained("bert-base-uncased") mdl = AutoModelForMaskedLM.from_pretrained("bert-base-uncased").eval() M = tok.mask_token # the middle slots are where a connective like "therefore" belongs text = f"Amy had 3 apples and bought 5 more , {M} {M} she has 8 in total ." ids = tok(text, return_tensors="pt").input_ids with torch.no_grad(): p = mdl(input_ids=ids).logits.softmax(-1) ent = -(p * p.clamp_min(1e-9).log()).sum(-1)[0] # predictive entropy per slot for i in (ids[0] == tok.mask_token_id).nonzero().flatten(): top3 = tok.convert_ids_to_tokens(p[0, i].topk(3).indices) print(f"slot {i.item()} entropy={ent[i]:.2f} candidates={top3}") # high-entropy slots = decision points with several valid branches; # confidence-first decoding schedules them last — that is the flexibility trap
Autoregressive decoding is the textbook N+1 query problem: 512 tokens means 512 serial requests, each paying a full round trip. Pure diffusion is the other extreme: fire the whole batch at once, but retry for many rounds, and recompute the entire batch every round. Block diffusion is the middle answer every DBA would give you — paged batching: one page of 256 at a time, parallel within a page, sequential across pages.
Pure masked diffusion has two hard problems, neither mathematical but both engineering. One, fixed length: you must allocate a canvas of fixed length, yet a real request's output length is not known in advance. Two, dead KV cache: bidirectional attention means every token depends on every other, so last round's keys/values are all invalid this round — autoregression's single biggest time-saver is gone.
Block Diffusion (Arriola et al., ICLR 2025, arXiv:2503.09573) solves both: cut the sequence into blocks, make it autoregressive across blocks (earlier blocks are committed, so they cache, and you can generate as many blocks as you like) and diffusion inside a block (parallel denoising). Block size becomes a continuous dial: block=1 degenerates to pure autoregression, block=full length to pure diffusion.
DiffusionGemma (Google DeepMind, 2026-06, open weights under Apache 2.0) follows exactly this route: a 26B-class MoE on the Gemma 4 architecture with only about 4B parameters active per step, denoising blocks of 256 tokens in parallel. The vendor's stated figures: decoding roughly 4× faster than Gemma 4, and over 1,000 tokens/s on a single H100.
Where does the speed actually come from? The one line worth remembering: latency ≈ NFE (number of forward passes) × time per forward pass. Autoregression's NFE equals the token count and not one step can be skipped. Block diffusion's NFE = number of blocks × denoising steps per block, and steps per block is far smaller than block length (256 positions might take only thirty or forty steps). Note it does not reduce FLOPs — each forward pass computes the whole 256-position block, so total compute does not drop and may even rise slightly. What is saved are the N−1 round trips of "must wait for the previous token".
Why is that trade worth it? Because during autoregressive decoding the GPU is actually waiting on memory: at batch=1, every single token requires moving all weights from VRAM into the compute units, leaving most of the arithmetic idle (memory-bound). Block diffusion spends that idle arithmetic processing a whole block at once, pushing memory-bound decoding back toward compute-bound — the same logic as rewriting an N+1 query as one batched SQL statement: round trips are the bottleneck, not arithmetic per query.
import math def nfe_ar(n): # autoregressive: one pass per token, no parallelism return n def nfe_block(n, block=256, steps=32): # sequential blocks, parallel inside return math.ceil(n / block) * steps N = 512 for fwd_ms in (10, 14): # time per forward (smaller with sparse MoE) a, d = nfe_ar(N), nfe_block(N) print(f"fwd {fwd_ms}ms | AR {a} passes={a*fwd_ms/1000:.2f}s | " f"block diff {d} passes={d*fwd_ms/1000:.2f}s | speedup {a/d:.1f}x") # turn the block/steps dials: block=1 degenerates to autoregression. # Key point: each block-diffusion pass computes the whole 256-position block, # so FLOPs do not drop. What is saved are N-1 "wait for the previous token" # round trips. Arithmetic traded for latency.
Autoregression is like single-leader strongly consistent writes: clear order, cacheable, every step sees everything already committed before it — at the price of serial writes. Diffusion is like multi-leader concurrent writes: high throughput, but within one round the slots cannot see what the others are writing this round, so you can get a combination where every field looks fine alone and the whole is inconsistent.
The fundamental tax on parallel decoding hides in that analogy: one forward pass gives you the marginal distribution p(xi | xt) at each masked position, so committing k positions at once quietly assumes those k positions are conditionally independent given the current context. But adjacent positions in language are strongly correlated — "New" and "York" could each be something else alone; only together is there one reading. The larger k, the further this assumption sits from the true joint. So "parallelism ↔ quality" is a real curve, not an implementation problem; adaptively controlling k with a confidence threshold is just picking an operating point on that curve.
Axis by axis. Likelihood: autoregression's chain-rule factorization log p(x) = Σ log p(xi|x<i) is an identity, so it yields an exact value you can use for scoring, perplexity comparison and reranking; diffusion's order is random, so it only gives an expectation-over-orders lower bound (the ELBO), and the two perplexities cannot be compared directly. Controllability: diffusion natively supports bidirectional constraints — fill the middle given both ends, fixed-format slots, right-to-left completion — which autoregression manages only awkwardly via fill-in-the-middle training; LLaDA also reports it mitigates the reversal curse (learning "A is B" yet failing at "B is A"), exactly the injury one-directional training leaves behind. Latency: diffusion wins, at the cost of the fixed-length canvas and rebuilding the KV cache.
When not to use it, honestly: when you need an exact likelihood for scoring or model comparison; when you want peak accuracy on long reasoning chains (the numbers above are still a gap); and when you need token-by-token streaming — tokens inside a block are not committed left to right, so the "typewriter effect" UX contract must be redesigned. The mainstream judgment today is not replacement but complementarity and convergence: block diffusion is itself an interpolation between the two poles, and block size is the slider.
import torch from transformers import AutoTokenizer, AutoModelForMaskedLM tok = AutoTokenizer.from_pretrained("bert-base-uncased") mdl = AutoModelForMaskedLM.from_pretrained("bert-base-uncased").eval() # bidirectional infilling: prefix AND suffix given — AR cannot see the right side M = tok.mask_token prompt = f"Cache stampede is usually fixed with {M} {M} , at the cost of write latency ." ids = tok(prompt, return_tensors="pt").input_ids slots = (ids[0] == tok.mask_token_id).nonzero().flatten().tolist() for _ in range(len(slots)): # commit one at a time → dodge the independence assumption with torch.no_grad(): p = mdl(input_ids=ids).logits.softmax(-1)[0] left = [i for i in slots if ids[0, i] == tok.mask_token_id] if not left: break best = max(left, key=lambda i: p[i].max().item()) # surest slot goes first ids[0, best] = p[best].argmax() print(tok.decode(ids[0], skip_special_tokens=True)) # switch the loop to commit all slots in one round to see the cost of independence
Engineering counterpart → super-individual D16 (Cost: the latency / throughput / cost triangle)