Introduction and Overview: the seven-step framework for generative systems

In one sentence

You think a generative system design question is about picking a model. It is about everything around the model. This chapter is the book's spine: it breaks "design a system that generates content" into seven steps — clarify requirements, frame it as a generative task, prepare data, develop the model, evaluate, deploy and serve, monitor and iterate — and names the three root differences from its sibling volume on discriminative ML systems: evaluation has no answer key, cost is billed per token, and failure sounds fluent. Those three decide nearly every trade-off in the ten case studies that follow.

Glossary

Where it sits

This is the only chapter in Part I of Aminian, Sheng and Xu's Generative AI System Design Interview (2024), and the template the ten case studies in Ch2–11 all follow. It inherits the seven-step framework of the sibling Machine Learning System Design Interview, but changes the subject: that volume designs discriminative systems whose output is a label or a score, this one designs generative systems whose output is a sequence, an image or a video. In the real world that maps onto smart compose, machine translation, assistants, RAG question answering, text-to-image and text-to-video.

The problem this chapter solves

Handed "design ChatGPT" or "design a text-to-image product", the usual failure is to dive straight into architecture — how many Transformer layers, how many diffusion steps. Not wrong, but it skips what actually decides the outcome: what is the latency budget (a completion product must land before the user's next keystroke, commonly tens of milliseconds to about a hundred; a chat product can spend a few hundred milliseconds on the first token as long as words keep flowing), how much may one request cost, on what grounds is this version better than the last, and how will you notice it quietly getting worse.

The deeper pain is that the failure mode changed. When a discriminative system is wrong, it is wrong visibly: probabilities skew, metrics drop, alerts fire. When a generative system is wrong, it is wrong fluently: HTTP 200, normal latency, zero error rate — the content is simply invented. No infrastructure metric will ever tell you. Ignore that and you ship a system that demos beautifully and gets screenshotted onto social media three months later.

The core, explained

Difference 1: no single answer, cost per token, fluent failure

A discriminative model maps an input into a finite output space. A generative model's output space explodes combinatorially: a 500-token answer has on the order of vocabulary-size-to-the-500th possibilities (|V|^500 — all you need from the notation is "too many to enumerate"). Three consequences run through the whole book.

(1) There is no single correct answer, so there is no accuracy to compute. Two answers can both be right and both be good, differing only in style — evaluation has to be designed, not selected off a shelf.

(2) Cost moves from "one inference" to "per token". A discriminative model produces its answer in one forward pass, at roughly constant cost. A generative model emits N tokens autoregressively, which is N forward passes — output length is cost, and output length is latency. This constraint comes back to bite in every later chapter.

(3) Failure is fluent. An uncertain discriminative model returns a probability near 0.5, and that signal can drive a threshold or a handoff to a human. An uncertain generative model keeps writing smoothly; its confidence is not externalized. So a generative system must build a layer of visibility that does not come for free — citations, groundedness checks, guardrails, sampled review — before you can know when it was wrong.

Difference 2: the seven steps, and how they differ from the sibling volume

Against the sibling framework, two changes stand out.

Feature engineering largely leaves the stage. The most labour-intensive step in a discriminative system — turning raw data into columns of features — is replaced by two things: data curation on the training side (dedupe, filter, mix) and context construction on the inference side (what to retrieve, where in the prompt to put it, how to fit it in the window). The context window is the new feature budget: it has a hard ceiling, and what goes in is a design decision.

Evaluation goes from one step to a through-line. In the sibling volume evaluation is "offline metrics → online A/B", two steps and done. Here, because there is no answer key, the evaluation itself must first be designed: define what good means, then decide who judges, how many cases, and how to keep the cost down. Nearly every one of the ten case studies needs its own.

Difference 3: step one is worth the most — nail down four budgets

Clarifying requirements matters more here than in a discriminative question, because what you are nailing down is four budgets that fight each other:

• Latency: completion products (tens of milliseconds, so the model must be small), chat products (a few hundred milliseconds to first token plus steady output, where streaming hides the total), offline generation (a text-to-video job may run for minutes). Three or four orders of magnitude apart, which directly decides how big the model can be and whether you can afford another retrieval hop.
• Cost: per-token billing means "one more sentence" has a price — here cost is not a finance question but an architectural constraint (see the industry evidence below).
• Quality: say what "good" means first — faithful to the supplied documents (groundedness), correctly formatted, or on-brand? Different definitions lead to different evaluations and different technical routes.
• Safety: what must never come out, what must be blocked on the way in, and who is accountable when it slips.

The test is blunt: until those four are nailed down, talking about model selection is empty.

Difference 4: where capability comes from — a four-rung ladder

Step four, "model development", rarely means training one from scratch. It means choosing a rung on a capability ladder: prompt engineering → RAG → fine-tuning → continued pretraining or training from scratch. Each rung up buys control and costs an order of magnitude more in both time and unit price.

The test for picking a rung is startlingly plain: ask whether that rung treats your actual disease. "It doesn't know anything about our company" is a knowledge gap and calls for RAG. "It knows, but the tone is wrong, the format is wrong, it won't follow our procedure" is a behaviour gap and calls for fine-tuning. Getting these backwards — fine-tuning to inject facts, prompting to fix behaviour — is the most common and most expensive mistake in the field.

Difference 5: evaluation when there is no answer key

Three routes, used together. Automatic metrics (BLEU / ROUGE against a reference, perplexity, FID for images) are cheap enough to run on every commit, but they only measure surface overlap and nearly break down on open-ended answers — say the same thing differently and the score drops. LLM-as-judge scores against a rubric with a strong model, scaling to tens of thousands of cases in minutes, at the price of its own biases: it prefers long answers, is swayed by option order, and favours outputs from its own family. Human review is the most trustworthy and costs tens of seconds to minutes per case, which means it can only ever sample.

The working answer is to layer them: the model judge runs the full set as the daily regression, humans sample to calibrate the judge itself, and online A/B or side-by-side battles (Elo) show what real people prefer. One trick you can copy directly comes from KDDI's RAG application: turn the judge's 1–5 rating into a binary pass / fail to cut variance, and sample on a two-dimensional grid of file format × media composition, which cut the evaluation workload by 75%.

Difference 6: deployment and monitoring — latency is two numbers

Latency for a generative service cannot be one number. Split it at least into TTFT (time to first token, which decides whether it feels responsive) and TPOT (time per output token, which decides whether the words keep up with reading — a comfortable reading pace is on the order of a dozen or so words per second). That is why streaming is the default in almost every chat product: it compresses perceived waiting from total duration down to TTFT.

The three serving tricks all revolve around tokens: continuous batching (splice decode steps from different requests into one batch so the GPU never idles), KV and prefix caching (compute the shared system prompt once), and quantization and distillation (trade a smaller model for a lower unit price per token). The open-source engine vLLM puts the first two straight into its pitch: PagedAttention for "efficient management of attention key and value memory", plus "continuous batching of incoming requests, chunked prefill, prefix caching".

What monitoring watches is not just 5xx either: quality drift (judge scores sliding slowly), cost drift (mean output length creeping up), guardrail hit rate and user rejection rate (retries, thumbs-down).

Key trade-offs

Table 1 · Where capability comes from: the four-rung ladder (timelines and costs are common industry magnitudes, not figures from the book)

Table 2 · Three evaluation routes: who judges, and how much you can afford to judge

• Is a longer context always better? Stuffing the window is the easiest mistake to make: it raises latency and the bill at once, and dilutes the relevant passage until quality actually drops. Spend the context budget as carefully as a feature budget.
• Is a bigger model always better? The latency budget pins model size first: in a completion product, a small model that emits in tens of milliseconds beats a twice-as-good model that arrives too late.
• Which layer holds safety? Training it into the model (alignment) is more thorough but slow to change; a guardrail layer iterates fast but is easier to route around. Real systems run both.

Why it matters / in real systems

The value of this chapter is a shared vocabulary across products: whether you are building completion, translation, chat, RAG question answering or text-to-image, it comes down to the same handful of questions — what are the latency and cost budgets, which rung of the ladder supplies the capability, who judges quality, and how does breakage become visible. It also maps onto how real teams divide the work: retrieval and indexing, inference serving (engines like vLLM and TensorRT-LLM), evaluation platforms, and safety guardrails are four separate jobs today.

Pitfalls and disputes

• Treating model choice as the substance of the design. The most damaging interview opening is to name an architecture immediately. The model is one of seven steps, and usually the replaceable one — the evaluation plan and the data are not.
• Using BLEU / ROUGE to sign off open-ended tasks. They measure literal overlap with a reference: say the same true thing differently and you score low; paraphrase the reference badly and you score high.
• Trusting the model judge too far. LLM-as-judge carries length, position and same-family biases; it must be calibrated against human samples, and swap the judge model and your historical scores stop being comparable — record the judge version alongside every result.
• Trying to "fix" hallucination as a bug. It is intrinsic to autoregressive sampling. Engineering can push it down with retrieval grounding, verifiable citations, guardrails and sampled review, but never to zero. A vendor promising zero is paying for it somewhere else.
• Ignoring how context length compounds. Every extra 1000 tokens in the system prompt is paid on every request; at a million daily users that writes your architectural decision straight onto the invoice.
• Period limits. The book was written in 2024. Reasoning models with long chains of thought, very long contexts and agentic loops have since broken the implicit assumption that one request equals one generation — the cost model moved from "output length" to "thinking length plus number of tool-call rounds". The seven steps and the three root differences have held up.

Takeaways