ChatGPT: the hard part isn't making it smarter, it's turning a text continuer into an assistant

In one sentence

You think the hard part of ChatGPT is making the model smarter; in fact the knowledge is largely fixed by the end of pretraining. What this chapter actually solves is an objective mismatch — pretraining teaches "predict the next token", while the product needs "follow the instruction, answer the way people prefer, refuse when it must". The answer is three stages (pretraining → supervised fine-tuning → preference alignment), and the engineering intuition behind it fits in one line: when "good" cannot be written as a loss function, turn human pairwise comparisons into a scorer you can optimise against.

Glossary

Where it sits

The third of the ten case studies in Part II. It follows Ch3 on translation — there at least a reference translation exists; here there is no reference answer at all, so evaluation has to be replaced wholesale. It leads into Ch5 on image captioning, pushing generation beyond plain text. In the real world it maps to ChatGPT, Claude and Gemini, and to any product backend built from multi-turn dialogue, tools and a safety boundary.

What the chapter is solving

Pretraining has exactly one objective: given what came before, predict the next token — so what it learns is what internet text tends to look like next, and on the internet a question is often followed by another question, an ad, a table of contents. The characteristic failure of a base model is therefore not "can't", it is answering beside the point, or continuing the list of questions.

The second difficulty is that "good" cannot be defined. Translation has a reference; code has tests. A good resignation letter has infinitely many forms, and its quality depends on length, tone, whether to ask a clarifying question, whether it oversteps — none of which reduces to a rule or a single labelled answer.

The third is that safety becomes a system requirement for the first time: a fluent model states harmful content just as fluently. The failure mode is being wrong fluently, not throwing an error.

What happens if you skip all this? You get a knowledgeable model nobody can use: unstable formatting, instruction-following by luck, and one sentence away from saying what it shouldn't. The whole chapter is about turning "can talk" into "gets things done and stays inside the lines".

The core, explained

1. Three stages, three different purchases

Pretraining buys knowledge and language: unlabelled text at the trillion-token scale, and the overwhelming majority of the compute. Supervised fine-tuning (SFT) buys format and intent: human-written "instruction + ideal reply" demonstrations drag the output distribution onto the "answering" side. It is cheap and fast, and its ceiling is obvious — demonstrations must be written one at a time, and they can only say "this is right", never "that one is worse".

Preference alignment buys ranking: people write nothing, they only pick the better of several replies the model produced itself. Throughput is an order of magnitude higher, and the signal is exactly the one SFT cannot give. The design in one line: write demonstrations while you can, and switch to picking when you can't.

2. The reward model: compressing an undefinable "good" into one number

The recipe: sample several replies per prompt (commonly 4–9), have people rank them, and train a scoring model on those pairs with a blunt objective — the chosen reply must score above the rejected one.

Why comparisons rather than scores? Because absolute scores are not comparable across people: the same reply gets a 3 from a strict annotator and a 5 from a lenient one, and the noise swamps the signal, while agreement on "is A or B better" is far higher. This is the one piece of methodology worth taking away: when no absolute standard exists, step back and ask only for relative judgments.

The cost lives in the same place: the reward model is only a finite fit to human preference, and its own biases — toward length, toward pretty formatting, toward a confident tone — are amplified faithfully by the next step.

3. PPO, and the leash

The RL loop is: the policy model generates → the reward model scores → the weights move a small step toward higher scores. PPO's "small step" matters, but the KL penalty in the objective matters more: it measures how far the new output distribution has drifted from the frozen SFT reference, and charges for the distance.

Without it, the model attacks the reward model rather than pleasing people: if long answers score higher it pads endlessly, if boilerplate scores higher every sentence becomes boilerplate — while other capabilities quietly degrade, a loss known as the alignment tax. The KL coefficient is therefore the knob that matters most here: too small and it drifts, too large and it barely learns.

A sense of cost: in Microsoft's published DeepSpeed-Chat numbers, a 13B model finishes all three stages on 8 A100s in about 13.6 hours, of which PPO alone takes 10.8. The bulk of alignment compute is not reward-model training, it is that sample–score–update loop — which is also the heaviest to engineer: policy, reference, reward and value models all sit in GPU memory at once.

4. Memory in a conversation: the context window is a hard constraint

The naive approach re-assembles the whole history into the prompt every turn. The problem isn't forgetting, it is arithmetic: 500 tokens a turn and 20 turns is 10k tokens; attention cost grows with the square of length (double the length, roughly quadruple that part of the cost) while KV-cache memory grows linearly and every concurrent session holds its own copy — what you hit first is usually not the context limit but the concurrency your GPU memory allows.

So real systems split "memory" into layers: the system prompt (fixed, cacheable), recent turns verbatim (the last k, preserving tone and reference), compressed history (a rolling summary of everything earlier), plus an optional long-term memory: stable user preferences written to external storage and retrieved on demand.

Two traps that get overlooked: summarisation is lossy and irreversible — what it drops cannot be asked for later; and information sitting in the middle of a long context is more likely to be ignored, so put what matters first or last.

5. Tool calling: outsourcing what it cannot do

Three things a model is structurally bad at: events after its training cutoff, exact computation, and private data it has never seen. One answer covers all three — have it emit a structured call request, let the system execute it, feed the result back into the context, and let generation continue.

Three things to watch in production. Latency: every tool round-trip is one full forward pass plus the tool's RTT, so two or three of them take you from a few hundred milliseconds to several seconds; cap the number of rounds and fire independent calls in parallel. Reliability: models call the wrong tool and invent arguments, so validate arguments against a schema and have retries and a fallback script. Trust boundary: once tool output is in the context it sits next to the user's instructions — this is exactly where prompt injection comes in, so external content must be labelled by origin and never granted instruction authority.

6. Safety and evaluation: signing off with no right answer

Safety is three layers stacked: refusal behaviour trained in with safety preference data, an input-side classifier that blocks clearly disallowed requests, and an output-side pass on the way out. Streaming makes the third layer hard — the text is already on the screen and retracting it is expensive, so you either validate in delayed chunks or accept the "emit then withdraw" experience.

Evaluation has to be replaced too: perplexity barely correlates with user satisfaction. The workable combination is human pairwise comparison (blind A/B, ranked by win rate or Elo) + LLM-as-judge (the same pairwise verdicts at scale — cheap and repeatable, but with position bias and a length preference that has to be corrected, e.g. by swapping order) + red teaming + online signals (thumbs-up rate, regeneration rate, human-handoff rate).

And one product-level trade-off you cannot dodge: over-refusal. Raise the safety threshold and violations fall, but the share of legitimate requests that get refused rises — the most typical "hurts at both ends" curve in alignment.

Key trade-offs

Table 1 · What each of the three stages buys

Table 2 · Four routes to alignment

Table 3 · Four ways to hold conversational memory

Why it matters / in real systems

This skeleton reaches well past chatbots: every product whose quality only a human can judge looks like this — copy generation, coding assistants, support QA, AI review. Fix format with demonstrations, fix preference with pairwise comparisons, scale human judgment with a stand-in scorer, then watch like a hawk for the day the stand-in gets over-optimised.

The most valuable sentence in an interview: translate "the model answers badly" into "what went wrong with our preference signal". Saying "this bad case isn't missing knowledge, it's a reward model that prefers long answers, so I'd add a batch of 'short and correct beats long and correct' pairs" carries far more weight than naming architectures.

Pitfalls and open arguments

• Thinking RLHF teaches knowledge. Knowledge is set during pretraining; alignment changes behaviour and expression, adding almost no facts. What it doesn't know, alignment can't fix.
• Thinking SFT is enough. A demonstration can say "this is right" but never "that is worse" — and writing one costs roughly ten times what picking a winner costs.
• Forgetting the KL constraint. Optimisation without a leash slides into reward hacking: answers get longer, boilerplate piles up, the metric climbs while the experience sinks.
• Treating the reward model as truth. Over-optimising a fitted stand-in must diverge — Goodhart's law in textbook form: once a measure becomes a target, it stops being a good measure.
• Treating the context window as a disk. It is a workbench, wiped after the turn; cross-session memory must be stored and fetched explicitly, and users should be able to inspect and delete it.
• Underrating retraction under streaming. By the time disallowed text is appearing character by character, blocking it is too late — the product's shape adds a hard constraint to safety design.
• Signing off on perplexity or single-point scores. The former barely correlates with satisfaction; the latter isn't comparable across annotators.
• Ignoring sycophancy and over-refusal as twins. Both come from the same source: the model learns "what people will like", and people like being agreed with while annotators reward cautious replies. They are not bugs, they are what this objective produces; only data mix and dedicated evals hold them down.
• Period limits. The chapter takes the PPO three-stage pipeline as its spine. Around its publication, DPO-style methods that skip a separate reward model, and RLAIF where a model does the labelling, had already displaced much of it, and reasoning models put automatically verifiable rewards back into RL. The underlying constraint hasn't moved: open-ended generation has no single right answer, so quality can only be defined by comparison.

Takeaways