CS PAPERS DEEP-READ · PAPER 12
Kaplan, McCandlish et al. · OpenAI · 2020
In early 2020 a team at OpenAI (Kaplan, McCandlish and colleagues) studied one question systematically: as you make a language model (the kind of "guess the next word" model behind ChatGPT) bigger and bigger, exactly how much better does it get? They found a startlingly clean rule — a model's prediction skill improves along a smooth, predict-ahead-of-time curve as you increase three things: how big the model is, how much text you feed it, and how much compute you spend. How clean? Before training even starts, you can calculate how good a ten-times-bigger model will be. This rule gave OpenAI the confidence to build GPT-3 — and lit the fuse on the whole "bigger is better" arms race that followed.
Your gut says making a model bigger should hit a ceiling: fast gains at first, then diminishing returns, then a plateau. Yet as they took models from tiny to huge and data from little to lots, the improvement curve turned out to be a straight downhill line — no kink, no plateau, straight across several orders of magnitude. It's as if every time you spend ten times the effort, your score reliably drops by the same fixed step, again and again.
People assumed how you "wire up" the network mattered a lot — how deep, how wide, how many attention units. It turned out that for the same total "part count" (number of parameters), whether you build it tall-and-thin or short-and-wide gives nearly the same score. What really decides the outcome is just three plain numbers: how big, how much data, how much compute. In other words, getting stronger is less about clever design and more about how many resources you're willing to pour in.
It turns a guessing game into arithmetic: given a fixed compute budget, should you buy a bigger model, or feed it more books to read? They computed the split — put the lion's share into making the model bigger, and don't wait for it to read its books cover to cover: read the gist and stop, then spend the next batch of compute. In short: rather than let a small model grind a little data to death, let a big model skim a vast amount. It's exactly this arithmetic that let OpenAI leap straight to GPT-3's unprecedented scale.
One honest caveat: that "split" was later found by DeepMind to be miscalibrated — for the same compute you should actually pick a less extreme model and feed it far more text. The shape of the law was right; the sweet spot was placed wrong.
It turned big models from alchemy into engineering: how much to spend, how to split it, what you'll get back can now be worked out on paper first. Today every lab runs a "scaling law" prediction of cost versus payoff before launching a big model — that calculate-first, spend-second playbook starts here.
A language model's skill improves along a smooth, predictable curve as you scale "model size + data + compute" — straight across several orders of magnitude, with the exact wiring of the network barely mattering. So for the first time you can calculate in advance whether to invest more and how to split it. That was GPT-3's confidence, and the quantitative basis for the whole "bigger is better" era.
Want the exact power-law exponents, the compute-optimal split, and Chinchilla's later correction? → switch to the deep read
This paper shows, with many controlled experiments, that a Transformer language model's test loss falls as a smooth power law in each of three factors — model size N, dataset size D, and training compute C — holding as a straight line across seven orders of magnitude, while structural details like depth-vs-width barely matter. It follows that, for a fixed compute budget, the optimum is to train very large models and stop early, before converging on the data. These "scaling laws" turned big-model spending into predictable engineering and directly underpinned GPT-3.
y = a·x^k. Its signature is that on log–log axes it's a straight line with slope k; it means "each 10× multiplies the improvement by a fixed factor."The authors are an OpenAI team led by Jared Kaplan and Sam McCandlish (with Tom Brown, Dario Amodei, Alec Radford and others), released January 2020. It inherits the GPT-2 (2019) line of "large models + unsupervised pretraining" and turns "bigger is better" from a slogan into a quantitative law. It launches that same year's GPT-3 — which bet on 175 billion parameters following exactly this arithmetic — and the whole scaling era. Two years later DeepMind's Chinchilla (Hoffmann et al., 2022) corrected its optimal split, making the two a must-read pair.
Around 2019 people vaguely sensed that "bigger model, more data → better results," but this was only empirical intuition: nobody could say how much better, where it would plateau, or whether money was better spent on "a bigger model" or "more data." Every large model is enormously expensive to train, so trial-and-error amounts to gambling with cash.
The authors set out to make it a science: fix everything else, vary one factor, and measure how loss responds. They trained a large family of Transformers from thousands to a billion-plus parameters, sweeping dataset sizes, model shapes, and training lengths, to see whether a clean law would emerge from the scatter — so people could "calculate before committing."
The central result: when the other two factors aren't the bottleneck, test loss L falls as a smooth power law in parameters N, data D, and compute C, respectively: L(N) ∝ N^(−0.076), L(D) ∝ D^(−0.095), L(C) ∝ C^(−0.050). Don't be scared by the numbers — in plain terms: plotted on log–log axes they are straight lines, running all the way out to seven orders of magnitude without bending — no kink, no sign of a plateau. Those tiny exponents (0.05–0.1) say the returns are steady but you must invest by multiples to buy each small improvement.
Why is that striking? Because it means it's extrapolable: fit the line on small models and you can fairly accurately forecast how good a much larger one will be — knowing the result before training, a rarity in deep learning.
They swept network "shapes" — same parameter count, made deeper vs wider, more vs fewer attention units, different feed-forward ratios. Across a wide range, shape barely affected loss; what truly decided the score was total parameter count N. That carries huge weight: it demotes "should we hand-tune the architecture," long a heavy investment, to a second-order concern, and pushes all attention onto scale.
Reading the same batch of data, bigger models learn faster and use fewer samples to reach a given loss. Combining the N and D effects into one formula of the form L(N,D) = [ (Nc/N)^(αN/αD) + Dc/D ]^αD: intuitively, too small a model makes parameters the bottleneck; too little data starts to overfit. To avoid overfitting, data need only grow sublinearly with the model (about D ∝ N^0.74), not blow up proportionally.
Pushed to a compute budget, this yields the famous recommendation: given fixed compute C, the optimum is to put most of it into a bigger model and stop early on the data, before convergence. By this paper's fit, optimal model size grows fast with compute (about N ∝ C^0.73) while data grows only slowly (about D ∝ C^0.27). In plain words: rather than let a small model grind a little data to death, spend on a bigger model and let it skim a vast amount. It's exactly this arithmetic that let OpenAI leap to GPT-3's scale.
The evidence is hundreds of training runs falling onto clean straight lines: the power laws in N, D, and C each hold across six to seven orders of magnitude with tiny fit error. The combined L(N,D) formula predicts both the "not enough parameters" and "not enough data" bottlenecks and their optimal trade-off. They also found the optimal batch size is predictable from the "gradient noise scale." Most convincing is the extrapolation check: a curve fit at small scale predicted larger models' loss quite accurately — precisely the basis on which GPT-3 placed its bet the next year.
It moved big models from "alchemy" toward "engineering": how much to invest, how to split it, what you'll get can now be calculated ahead of time. Three deep effects — one, it gave GPT-3 its quantitative justification and established the now-standard practice of running scaling-law forecasts before training; two, it shifted the field's focus from "designing cleverer architectures" to "scaling up," shaping the "scale is all you need" creed; three, its methodology — fit a law on small experiments, extrapolate to large models — was adopted widely (scaling laws are now an essential tool for training any frontier model).
① In one line: a Transformer LM's loss is a smooth power law in parameters N, data D, and compute C — straight across many orders of magnitude, and extrapolable.
② The pain: "bigger is better" was only intuition — no one could say how much better, where it plateaus, or whether to spend on model or data; every run was expensive trial-and-error.
③ Power laws: L(N)∝N^−0.076, L(D)∝D^−0.095, L(C)∝C^−0.050 — straight lines on log–log axes; small exponents mean steady returns but multiplicative investment.
④ Scale beats shape: at fixed parameter count, depth-vs-width and other shapes barely affect loss; total parameters decide the score.
⑤ Big models are more sample-efficient; given compute, the optimum is "train a very large model and stop early on data" (this paper: about N∝C^0.73, D∝C^0.27, model-heavy).
⑥ Validation: hundreds of points fall onto lines holding across six–seven orders of magnitude; small-scale fits extrapolate to large models — GPT-3's direct basis.
⑦ Impact: turned big models from alchemy into engineering, shaped the "scale is all you need" era; scaling laws became a standard tool for frontier training.
⑧ Limits: Chinchilla corrected the split (model/data should scale equally, ~20 tokens/param), so many models were undertrained; loss ≠ downstream ability; extrapolation isn't unlimited; it explains what, not why.