Text-to-Video: the Hard Part Isn't Looking Real, It's the Next Frame Still Recognising the Last

One line

Text-to-video is not text-to-image ×N. The chapter's core claim: temporal consistency has to be written into the model's structure as a first-class citizen, not patched on afterwards — and the moment you write it in, compute explodes along two axes at once, frame count and resolution. So the whole chapter has exactly two engineering threads: how to build time in, and how to spread the compute out. Nearly every architecture in video generation today is a compromise between those two.

Glossary

Where it sits

This is the closing problem of Part II's ten interview questions in Generative AI System Design Interview, and the most compute-hungry of them. It stands directly on Chapter 8 (high-resolution image synthesis · latent diffusion) and Chapter 9 (text-to-image · cross-attention and classifier-free guidance): "how to paint one frame well and make it obey the prompt" is taken as given here, and the chapter adds exactly one new thing — time. In the real world it maps to Sora, Veo, Runway and friends, and to any generative backend where you submit a job, queue, and collect the output later.

What this chapter is solving

Start with the scale. A 5-second clip at 24 fps is 120 frames. Generate it frame by frame the way Chapter 9 would, and — taking a single image on one A100 to be a few seconds — a hundred and twenty frames is minutes, at best. What you get for those minutes is unwatchable: each frame is solved afresh from independent random noise, so the content drifts semantically with nothing holding it down.

What happens if you don't solve it? Two bad things at once. One, the quality ceiling is nailed shut: people are far more sensitive to breaks in time than to per-frame sharpness — an extra finger in one frame may go unnoticed, but a sofa pattern that changes every frame is obvious to a child. Two, the cost ceiling is nailed shut: a video's pixel count is frames × height × width, so moving the same model onto video multiplies the compute, and nobody can pay that. The real problem of this chapter is therefore: buy temporal consistency out of an exploding compute budget.

The core content, properly

Why "per-frame generation plus interpolation" cannot save you

One distinction has to be drawn first. Frame interpolation can only make already-consistent motion smoother; it assumes the two neighbouring frames were two instants of the same scene to begin with. The failure of independent per-frame generation is not "not smooth enough" — it is semantic drift: the cat in frame 40 and the cat in frame 41 are simply not the same cat, and an interpolated frame between them is just a smeared blend of two cats. The test is blunt: interpolation is a smoother. It fixes jitter; it cannot fix "the thing got swapped out".

So the conclusion is hard: time has to enter the generative process itself, not sit behind it.

Inflating a 2D model into 3D: factorized space-time attention

The mainstream move is called inflation: take a trained text-to-image model (the latent U-Net or DiT of Chapters 8 and 9), leave every spatial layer untouched and insert new temporal layers into each block — convolution or attention running along the axis of "same spatial position, different frames." The payoff is direct: image quality and prompt adherence are inherited wholesale, the new parameters only have to learn "how things move," and the amount of paired text-video data you need drops sharply — which neatly patches the fact that video data is far scarcer than image-text data.

Why not just do full 3D attention? Cost. Say a clip is T latent frames of S tokens each. Flatten the whole thing into one attention pass and you compute (T·S)² pairs; split it into "spatial attention within a frame + temporal attention across frames" and you compute T·S² + S·T². Plug in realistic numbers — T=21, S=1024 — and the first is about 460 million pairs, the second about 22 million: a factor of 20. That factor of 20 is the entire reason most video models today are factorized.

The cost has to be stated honestly: factorizing means no token can see "another place and another moment" in a single step. Long-range space-time relationships are only built indirectly, by stacking layers. So factorized models break down first on big camera moves and objects streaking across the frame — exactly the cases that need position and moment to shift together. That is precisely what motivated the later route of cutting space and time into joint patches and running full attention over them, with Sora as its best-known example.

Time gets compressed too: the video VAE and that little "+1"

Chapter 8 already moved denoising into latent space — the picture is squeezed spatially by an autoencoder, denoised there, then decoded. Video has to squeeze one more axis: time. The typical device is a causal 3D autoencoder: 8×8 spatially, 4× temporally, which together take a clip down to roughly 1/256 of its size (channels aside). Without this step, neither cascades nor long context are on the table.

One detail worth remembering: the official CogVideoX repository hard-codes the generatable frame count as 8N+1 (the 5B model) or 16N+1 (the 1.5 model, default 81). That +1 is not padding — it falls out of a causal design in which the first frame is encoded on its own and the rest are compressed in fixed-size groups. The first frame has to be independently recoverable for the most common use of all to work: hand me one image, extend it into a clip. A constraint that looks arbitrary in the API is really the shape of the autoencoder showing through.

The cascade: do the hard step once, on the smallest canvas

This is the engineering structure most worth taking away. Rather than running the whole denoising process at target resolution and target frame rate, split it into a relay:

• Base model: generate a "skeleton" at low resolution and low frame rate — content, composition and rough motion are all decided here. Hardest and most expensive.
• Temporal SR: push the frame count up by inserting in-between frames. This only has to be locally coherent, a far easier job.
• Spatial SR: enlarge each frame. This barely needs to understand semantics at all.

Why it pays: compute scales roughly as frames × side². Run the expensive full denoising once on the canvas where both frame count and side length are smallest, and the two later stages — large in area though they are — can use shallower models and fewer sampling steps. Google's Imagen Video (2022) was exactly a base model feeding a chain of interleaved temporal and spatial super-resolution models; and Vertex AI shipping "upscale to 1080p / 4K" today as a capability decoupled from generation is the same structure projected onto the product surface.

The cost is equally clear: a cascade is a pipeline that accumulates error — whatever motion the base model gets wrong, the two later stages simply render more crisply. And several models means several trainings, several deployments and several monitoring stories, with end-to-end tuning all but impossible.

Long video: if you can't generate it in one go, keep writing

What comes out of the above is usually a few seconds. For more, the chapter's main road is autoregressive extension: take the last few frames of the generated segment as the condition for the next one. Stability AI's official repository states the scale plainly: Stable Video Diffusion was "trained to generate 14 frames," and SVD-XT is "finetuned for 25 frame generation" — fourteen to twenty-five frames, two or three seconds at ordinary frame rates. Everything longer is stitched.

The price of stitching is drift: each segment takes the previous one's output as input, error accumulates down the chain, and colour, lighting and character identity slide away. Extension is also inherently serial — more GPUs will not parallelise it. The mitigation is to re-inject the global conditioning at every segment (prompt embedding, first frame, reference image) so each one anchors back to the same source rather than to its predecessor.

Key trade-offs and choices

The soul of this chapter is how you buy consistency with compute — three routes, none of them free.

Table 1 · Choosing among the three generation routes

Table 2 · When you want a longer clip

The choice compresses to one sentence: if your setting is "a few seconds, previewable, regenerable," a cascade plus fixed-length clips is almost always right; only when "one long take with a moving camera" is itself the selling point is it worth paying the full space-time attention bill. This is a different calculus from text-to-image, where the dominant variables are fidelity and prompt adherence. Here the dominant variable is how long consistency holds.

Why it matters / in real systems

The product shape is basically dictated by compute. Video generation is almost always an asynchronous job — submit, queue, poll, collect. That is a different backend from text-to-image, where you wait synchronously for a few seconds: you need a job queue, preemptible GPU scheduling, recoverable job state, and a two-stage delivery of "low-resolution preview first, full render after the user confirms," because one regeneration costs GPU-minutes.

Cost is a design constraint, not an after-the-fact invoice. Per-second pricing is the norm externally; on the training side, the open-source Open-Sora project publishes the training cost of its 11B model as $200K. The value of that number is not that it is cheap — it is that "can we train this?" has become a budget question with an answer.

Safety and compliance bind harder than for images. Prompt-side filtering, output-side detection and provenance marking (watermark / C2PA) are all mandatory: a forgery that moves is far more convincing than a still, so likeness and copyright risk are not in the same league.

Generating is only the start. The finished clip still has to be transcoded into multiple bitrates and shipped over a CDN — no different from an ordinary video-on-demand system, and routinely missed in design answers.

Pitfalls and open questions

• Believing interpolation fixes flicker. Interpolation is a smoother; it only works when things were already coherent and merely rough. Per-frame generation fails on semantic drift, and interpolating across that just smears two different things together.
• Treating FVD as the judge. FVD compares feature distributions and is insensitive to small but glaring temporal artifacts like flicker and jitter, while being highly sensitive to sample size and preprocessing. A good score and a good viewing experience are often different things — which is why the chapter's evaluation section still lands on human review.
• Reading a CLIP score as "it obeyed." CLIP score measures text-image similarity frame by frame and ignores time entirely: a clip where every frame matches the prompt but the motion is nonsense can score very well.
• Treating "longer" as the headline metric. In products, "fixed 4–8 seconds plus controlled extension" usually beats "60 seconds in one go," because what users actually want is another try — and nobody wants to pay for another try at 60 seconds.
• Forgetting the old data problem. High-quality paired text-video data is far scarcer than image-text pairs, and much of the labelling is auto-generated captioning whose noise turns directly into "it doesn't follow the prompt." This is the root motivation for the "keep the image model, train only the temporal layers" route.
• Period limits and open argument. The book was written in 2024, when the default was U-Net plus factorized attention plus a cascade; as compute grew, "space-time patches, long-context full attention, joint audio-video" became the new default and cascades got shorter. The bigger argument is whether these models have learned physics or merely pixel statistics that look like physics — failures on unusual physical interactions remain common.

Takeaways