CS PAPERS DEEP-READ · PAPER 13
Jason Wei et al. · Google / Stanford / DeepMind · TMLR 2022
Remember how a big model like GPT-3 could "learn a new task from a few examples"? This paper (2022, led by a team at Google) noticed something even stranger: certain skills that small models simply cannot learn appear "all at once" once a model is scaled past some threshold — not a gradual climb, but a jump from "gets nearly everything wrong" to "does it competently." They gave the phenomenon a name: emergence.
Think of boiling water. You add heat degree by degree and it stays just "hot water" — 90°, 95°, 99°, no obvious change in kind. Then at 100° it suddenly boils, turning from liquid to steam. That last degree doesn't buy "a bit hotter" — it buys a new state. Some abilities of large models are like this: as you scale up bit by bit nothing seems to happen, then past a line the ability appears as if by a "phase transition."
The prior consensus was: bigger models do smoothly and predictably a little better across the board (exactly what the "scaling laws" paper described). You could even draw a line through small-model scores and extrapolate roughly how good a big one would be. But emergent abilities break that rule: before the threshold the curve sits flat at the "random guessing" level, and you simply cannot predict the jump ahead. Whether and when it jumps is invisible in advance.
The method is plain: take a broad batch of tasks of varying difficulty (from three-digit arithmetic and obscure-language QA to reasoning-heavy exams), have a whole lineup of models from small to large attempt them, and plot "how big the model is" against "how many it got right." The same shape kept appearing: a long stretch hugging the floor (about as good as guessing), then a sharp lift-off past some scale. They also found that some "problem-solving tricks" are emergent too — for example, having the model "write out its reasoning step by step before answering" (chain-of-thought) actually hurts small models and only starts helping once the model is big enough.
An exciting and unsettling conclusion: making a model bigger may unlock a batch of new abilities you didn't see coming at all — which is both the strongest motive to "keep scaling" and exactly what safety researchers worry about (harmful abilities could appear unannounced too). A large model is no longer "the same thing, scaled up" — scaling may turn it into a different thing.
Some abilities that small models entirely lack appear "suddenly" once a model crosses a scale threshold — like water hitting its boiling point — and cannot be predicted in advance. That is emergence. One honest caveat, though: later work argued the "jump" is sometimes an illusion of how you score the test — measure with a gentler ruler and the cliff flattens into a slope.
Want the emergence curve, which abilities exactly, and the "is it real?" debate? → switch to the deep read
This paper formally names and systematizes an observation: some abilities are entirely absent in small models (scores hug the random baseline) yet appear "suddenly" and improve fast once a model crosses a scale threshold — the authors call these emergent abilities. Their defining feature is unpredictability: you cannot foresee the jump by extrapolating small-model performance (the smooth "scaling law" curve). The paper backs this with a dozen-plus tasks across GPT-3, LaMDA, Gopher, Chinchilla, and PaLM, and notes that even "prompting tricks" like chain-of-thought and instruction following only work on large models.
Lead author Jason Wei, with collaborators from Google Research, Stanford (including Percy Liang, Rishi Bommasani, Tatsunori Hashimoto), and DeepMind; published in TMLR in 2022. It inherits the "bigger is better" quantitative picture of the "scaling laws" paper (Kaplan et al. 2020) but deliberately chisels at its non-smooth, unpredictable side. It quickly became a household term for understanding large models — and drew the famous 2023 rebuttal "Are Emergent Abilities a Mirage?" (see the last section). It proposes no new model or architecture; it is a work of observation and questioning — influential precisely because it asked the right question.
Around 2020, scaling laws gave the field a comfortable worldview: scale model and data together, and loss falls predictably along a smooth curve. That curve is elegant and useful — but it answers "how much better on average." The authors pressed a finer question: if you shift your gaze from "average loss" to "can a specific task actually be done," is the picture still smooth?
The answer is no. On many specific tasks, scores don't climb slowly with scale — they sit for a long time at "no better than guessing," then shoot up at some scale. Such "absent in small models, present in large ones" abilities cannot be extrapolated from small-model curves — an important complement to the impression that scaling laws make "everything predictable," and a prod to rethink: how should we even measure or anticipate what a bigger model will be able to do?
The authors give a deliberately simple definition: "An ability is emergent if it is not present in smaller models but is present in larger models." The point isn't the truism "big models are stronger," but two things: first, a phase-transition shape — flat and near-random below the threshold, a steep rise above it; second, unpredictability — from the flat stretch alone you cannot tell whether or where the jump will come. That is exactly what separates it from the smooth scaling-law curve.
The method has no gimmicks: take a lineup of models from the same family that differ only in size (so architecture, data, and training are as comparable as possible), run the same batch of tasks with a uniform few-shot prompt, put training FLOPs or parameter count on the x-axis and accuracy on the y-axis. What emerges over and over is the "flat line + cliff" of Fig 1. The authors stress that either FLOPs or parameters work as the x-axis; what matters is the shape of the curve, not any precise critical number.
The other half of the paper is the emergence of prompting strategies: some tricks are useless or even harmful on small models and only start helping past some scale. The clearest case is chain-of-thought (CoT) — having the model write reasoning before answering actually lowers accuracy on small models, and only clearly beats "answer directly" above some scale. Instruction tuning is similar: it barely helps (or hurts) small models, and only large-enough models use it to broadly learn to follow instructions. In short, both "abilities" and "ways of using abilities" can have thresholds.
Why does this happen? The paper candidly has no settled answer, offering only candidate intuitions: perhaps some tasks require several sub-steps to all click into place, and only large models assemble every piece; perhaps the strict "all-or-nothing" metrics used in evaluation display gradual progress as a jump (a thread that, tellingly, seeds the later debate). The authors explicitly leave "why emergence happens" an open question.
The paper compiles a dozen-plus few-shot tasks that show emergence, spanning arithmetic (e.g., three-digit add/subtract), phonetic transliteration, word unscrambling, Persian QA, truthfulness judgments, and a dozens-of-subjects exam (MMLU), with models drawn from the GPT-3, LaMDA, Gopher, Chinchilla, and PaLM families. Their common trait: scores hug the random baseline until roughly 10²²–10²⁴ training FLOPs, then shoot up — the exact threshold varies by task. Chain-of-thought's benefit likewise only turns positive on large models (around the ten-billion-plus-parameter range). From this the authors argue: emergence is a general phenomenon recurring across many tasks and models, not a one-off coincidence. (Note: the numeric ranges shift with model family and task; these are order-of-magnitude figures.)
This paper's weight is not technical but in how it shaped the way the field talks about large models. It pushed "emergence" into common vocabulary and made people realize: a bigger model may not be "the same thing, stronger" but may unlock new abilities that can't be foreseen. That picture both fuels the drive to "keep scaling" (a surprise may lurk past the next threshold) and puts safety researchers on high alert — if useful abilities can appear unannounced, so can harmful ones. It also elevated evaluation: to know whether a model "can" do something, average loss isn't enough — you must test task by task. Nearly all later discussion of "does a large model really understand" and "what else might emerge" takes it as a reference point.
① In one line: some abilities are absent in small models and appear "suddenly" past a scale threshold, unpredictably — the authors name these "emergent abilities."
② The pain: scaling laws say "overall error falls smoothly and predictably," but on specific tasks scores are often "a long flat line + a cliff," not extrapolable.
③ Definition: absent in small, present in large; the key is the phase-transition shape + unpredictability, not merely "bigger is stronger."
④ Method: a lineup of same-family models of different sizes, uniform few-shot prompts on the same tasks, scale on the x-axis and accuracy on the y-axis, yielding "flat line + cliff" repeatedly.
⑤ Tricks emerge too: chain-of-thought and instruction tuning are useless or harmful on small models and only work once big enough.
⑥ Results: a dozen-plus tasks (arithmetic, obscure-language QA, MMLU, etc.) emerge around 10²²–10²⁴ FLOPs; recurs across GPT-3, LaMDA, Gopher, Chinchilla, PaLM.
⑦ Impact: pushed "emergence" into common vocabulary, fueling the "keep scaling" drive, igniting safety concern over unpredictable risks, and raising the stakes of task-by-task evaluation.
⑧ Limits: Schaeffer et al. 2023's "mirage" rebuttal — many jumps are artifacts of discontinuous metrics, smooth under continuous ones; "unpredictability" weakened; the paper only observes, without explaining the mechanism.