CS PAPERS DEEP-READ · PAPER 14
Ouyang et al. · OpenAI · NeurIPS 2022
In early 2022, a team at OpenAI (Ouyang et al.) introduced InstructGPT, using a method called RLHF (training with human feedback) to turn GPT-3 — which only knew how to "keep talking" — into an assistant that "does what you ask." The result is striking: a small model with just 1.3 billion parameters had its answers preferred by people over the original 175-billion-parameter GPT-3. A model 100× bigger lost to one that "listens better." ChatGPT, which arrived a few months later, uses the same recipe.
GPT-3 has read the whole internet and is stuffed with knowledge, yet it often "answers the wrong question": ask it to "explain the moon landing in one sentence" and it may spin out a plausible-sounding but off-topic paragraph; ask a serious question and it may confidently make things up, or blurt out something offensive. It isn't dumb — it simply wasn't ever "answering you." It was playing a "guess the next word" game. It learned "what usually comes next on the internet," not "what this person actually wants." Those two things are often not the same.
The old way to make models better was to feed them more text and have them imitate it. But "imitating internet text" can never teach a model to "answer a real person thoughtfully." The idea here: instead of more imitation, have people rate its answers, then have the model adjust toward "what people prefer." Like training a chef — memorizing recipes isn't enough; a diner has to taste and say "this is good, that's too salty" before the chef knows which way to tune. Handing the model that yardstick — human preference — is the first time it learns what a "good answer" even looks like.
Step 1 · Ask people to demonstrate. First, a few dozen trained people write ideal answers by hand for a range of prompts, and the model imitates them — like an apprentice watching the master do it a few times.
Step 2 · Ask people to rank, and train a "judge." For the same prompt, let the model produce several answers, and have people rank them from best to worst. Use those rankings to train a dedicated "judge model" that learns human taste — so no human need be present later; it can score any answer on its own.
Step 3 · Let the model practice, judged by the judge. The model keeps trying new answers, the judge scores them in real time, and the model nudges itself toward "higher scores." To stop it from talking nonsense just to please the judge, it's kept on a "don't drift too far" leash — new answers may not stray too far from its old self.
The model now actually "answers you": more willing to follow instructions, less prone to making things up, less likely to say something offensive, and better at declining what it shouldn't answer. More surprisingly, this path proved that "making a model more useful" doesn't require making it bigger — tune it to listen better, and a small model can beat a giant 100× its size. Nearly every chat AI you use today stands on this method.
One honest note: it is aligned to the taste of those few dozen labelers and researchers, not to "humanity's" values; the model still makes mistakes, can be worked around, and may even learn to "say what sounds nice" rather than what's true.
GPT-3 only knew how to "guess the next word," not how to "answer you thoughtfully." InstructGPT's fix: have people demonstrate, have people rank answers to distill a "judge," then let the model improve itself against the judge's scores — teaching it human preference. So a 1.3B model became more popular than 175B GPT-3, and ChatGPT came from the same recipe.
Want the per-step losses, the KL constraint, and the numbers? → switch to the deep read
InstructGPT uses a three-step RLHF (reinforcement learning from human feedback) pipeline — supervised fine-tuning → train a reward model → optimize with PPO — to redirect a pretrained language model's objective from "predict the next word" toward "follow user intent, be helpful and harmless." The result: a 1.3B InstructGPT whose answers labelers prefer over the original 175B GPT-3, while being more truthful and less toxic. This recipe became the standard alignment step behind ChatGPT, Claude, and later conversational models.
By Long Ouyang, Jeff Wu, and colleagues at OpenAI, the paper "Training language models to follow instructions with human feedback" appeared in March 2022 and at NeurIPS 2022. It picks up directly from GPT-3 (2020) — taking that "big but unruly" completion model and aligning it — and builds methodologically on a line of OpenAI/DeepMind work since 2017 on "learning rewards from human preferences." What it launches is immediate: ChatGPT, a few months later, is its sibling, and RLHF became the universal step for turning a large model into an assistant.
Large models like GPT-3 are trained with a single objective: predict the next word. That objective is not the same as what users actually want — "help me, following my instructions, usefully and safely." The authors name the gap: the model is imitating the distribution of internet text, not serving a person with an intent.
The consequences are concrete and annoying: GPT-3 often ignores instructions (you ask for one sentence, it gives a page), fabricates facts (hallucinates), produces toxic or biased content, and is hard to simply "direct." Prompt engineering (crafting cleverer instructions) helps somewhat but treats the symptom, not the cause. The authors call this the "alignment problem": the model has the capability, but its objective isn't aimed at human intent. Their target is to make the model helpful, honest, harmless (the "3H").
The hard part: a "good answer" has no ground truth and can't easily be written as a loss function — there's no formula for "is this passage useful and appropriate." The only reliable judge is a human. So the problem becomes: how do you get "human judgment" into the training loop?
The answer is a three-step pipeline. Two insights anchor it: (a) rather than have people write ground-truth answers (expensive, and impossible to cover exhaustively), have them do something easier — pick which of several candidates is better; (b) distill those human preferences into a "reward model" that becomes an automatic scorer — a proxy for human preference that RL can query an unlimited number of times.
First, about 40 screened and trained labelers write ideal answers by hand for real prompts drawn from the OpenAI API, producing "demonstration data." Standard supervised fine-tuning of GPT-3 on this yields the SFT model. This step is the "hands-on demonstration," getting the model roughly into an "answering" mode — but the demo data is limited, far from enough to teach every nuance.
The pivotal step. For one prompt, sample K answers from the model and have labelers rank them best-to-worst (rather than assign absolute scores — people rank far more consistently than they score). Each "better vs. worse" pair is a training signal, and a pairwise ranking loss trains the reward model to give the human-preferred one a higher scalar score r(x,y). Intuitively, the RM compresses dozens of people's taste into one neural network — after which no human need be present; it scores any answer instantly.
A pragmatic design detail: they use a 6B reward model, not the largest 175B — a huge RM is unstable to train and unnecessary.
With an automatic judge in hand, training becomes an RL problem: the policy is the language model, which generates answers; the reward model scores them; and PPO nudges the policy in small steps toward higher scores. But optimizing this directly has a big trap — the model will "game the judge" (reward hacking): the reward model is only an approximation of people, and the model can find answers that trick a high score while being actually bad.
Two constraints keep it in check. The objective can be written roughly as:
objective = E[ r(x,y) − β·KL(π_now ‖ π_SFT) ] + γ·E[language modeling on pretrain text]
In plain terms: the first term is "score as high as possible"; the second term is a per-token KL penalty — the new model's output distribution may not drift too far from the SFT model, the "don't stray too far" leash that blocks talking nonsense to please the RM; the third term mixes back a slice of the original pretraining objective (denoted PPO-ptx) to stop the alignment tax — the tendency, when single-mindedly aligning, to degrade general capability. Together, the model moves toward human preference without flying off or regressing.
The headline: on human-preference evaluations, the 1.3B InstructGPT's outputs are preferred by labelers over the original 175B GPT-3 — 100× fewer parameters, yet more popular because it's more aligned. At equal size, 175B InstructGPT is preferred over 175B GPT-3 by an overwhelming margin.
Other dimensions: on TruthfulQA (truthfulness) it answers truthfully far more often; on RealToxicityPrompts, when asked to "be respectful," toxic output drops markedly; it follows explicit instructions better and fabricates less. Generalization holds up too: it works for "held-out" labelers who weren't in training, and for non-English instructions and code requests that were rare in training. As for cost, plain RLHF regresses on some public NLP benchmarks (the alignment tax), but PPO-ptx, which mixes in pretraining gradients, largely erases that regression.
It gave the field the standard recipe for the one step that turns a "raw large model" into a "usable assistant." Before this, the instinct was "if the model isn't good enough, make it bigger"; this paper shifted the emphasis from scale to alignment, and made the point undeniable with "1.3B beats 175B": bigger alone isn't enough — you have to aim it at human intent. ChatGPT, months later, is a product of this method, and nearly every conversational model since — Claude, Llama-Chat, Gemini — adopted RLHF or a variant. It also established "reward model + KL constraint + policy optimization" as the general skeleton of alignment engineering, with later RLAIF, DPO, and others evolving inside the same framing. In a real sense, that everyone can converse fluently with AI today rests on this paper's threshold from "completion engine" to "assistant."
① In one line: use three-step RLHF (SFT → reward model → PPO) to redirect a large model's objective from "guess the next word" toward "follow user intent, be 3H."
② The pain: the pretraining objective (predict next word) ≠ user intent, so GPT-3 ignores instructions, hallucinates, and is toxic; a "good answer" can't be written as a loss, and only humans can judge it.
③ Insight: have people do the easier task of "ranking" rather than writing ground truth, then distill preference into an auto-scoring reward model.
④ Three steps: SFT aligns roughly using human demos; the RM learns taste via a pairwise ranking loss (6B, not 175B); PPO improves the policy against the RM's scores.
⑤ Two constraints: a per-token KL penalty leashes the model against drift and reward hacking; PPO-ptx mixes back pretraining gradients to erase the "alignment tax."
⑥ Results: 1.3B InstructGPT preferred over 175B GPT-3; more truthful (TruthfulQA), less toxic, better at instructions, and it generalizes to held-out labelers and non-English/code.
⑦ Impact: the same recipe as ChatGPT; RLHF became the standard alignment step for turning large models into assistants (later RLAIF, DPO evolved from it).
⑧ Limits: aligned to a few labelers' preferences, not humanity's; the RM can be gamed; it breeds sycophancy; it still hallucinates / can be jailbroken; human labeling is costly and hard to scale.