The line you are reading now is not being swept over at a steady pace. Your eyes stop, jump ahead, stop again. Across a paragraph, roughly a third of the words never get a stop at all — you skip them. An eyetracker records every one of those stops and jumps to the millisecond.
How long you stop depends largely on how guessable the word is. That piece of common sense can be turned into a formula accurate to the millisecond: take a language model's probability for "the next word is this one", take its negative logarithm, multiply by an exchange rate, and you have the extra milliseconds.
The formula works well enough to be suspicious of. So the real business of this piece is the second half of the question: where it holds, where it breaks, and what the correlation does and does not show.
In 1981 Ehrlich and Rayner did something very direct. They placed the same target word into two kinds of context — one where the preceding text pinned it down almost completely, one where nothing constrained it. Readers skipped the word entirely more often in the high-constraint version, and when they did stop on it, the stop was shorter. The researchers also quietly misspelled part of the target, and found that the tighter the context, the less likely readers were to notice the error. Part of what you see, you supply yourself.
So prediction is not repair work done after a sentence ends. It runs while you read, and it starts before the word reaches your eyes.
Mandarin makes this unusually visible, through its classifiers — the small words that sit between a number and a noun. Say 一把 (yī bǎ) and the following noun jumps into the class of things with a handle: 伞 (sǎn, "umbrella"), 刀 (dāo, "knife"), 椅子 (yǐzi, "chair"). Say 一只 (yī zhī) and you land among animals and one-of-a-pair items: 猫 (māo, "cat"), 鞋 (xié, "shoe"). Say 一条 (yī tiáo) and it is long thin things: 鱼 (yú, "fish"), 路 (lù, "road"), 腿 (tuǐ, "leg"). The English article "a" tells you none of this. Nor is this only a claim from grammar books. In brainwave studies, a classifier that clashes with the upcoming noun elicits an N400 — the electrical response that appears when a word's meaning does not fit — at the moment the classifier itself is read, and the response is graded: the closer the classifier is to what the reader expected, the smaller it gets. A 2023 study using representational similarity analysis found that patterns of brain activity already diverge before the noun arrives, depending on whether the classifier points toward animate or inanimate nouns.
The betting, then, is real betting. The next question follows on its own: how good is the bet, and can it be turned into a number?
In 2001 Hale and in 2008 Levy each turned this into a quantity. Given the preceding context, compute the probability p of the word that actually turns up, and take minus its logarithm. Call it surprisal. The lower the probability, the larger the number; when the probability is 1, it is 0. Take logarithms base two and the unit is the bit.
What matters is not the definition but the shape of its relation to milliseconds. In 2013 Smith and Levy pushed probability from near one down past one in a million — six orders of magnitude — and watched what reading time did. The relation is logarithmic, not linear. Put plainly: the extra time it costs to drop from "near certain" to "a coin flip" is the same as the extra time it costs to drop from one in a thousand to one in two thousand. Every halving adds the same small charge. How small? Around two milliseconds per bit in naturalistic reading, roughly four in self-paced reading. A few milliseconds per bit is not much, but it is stable enough to pull out of the noise.
In 2024 Shain and colleagues scaled the question up — several eyetracking and self-paced reading corpora, probabilities re-estimated with modern language models — and the logarithmic shape still beat the linear one. In 2023 Wilcox and colleagues checked it from another direction: eleven languages across five families (Korean, Turkish, Hebrew, Finnish, plus Dutch, English, German, Greek, Italian, Russian and Spanish), using the multilingual eyetracking corpus MECO. All three predictions held — surprisal predicts reading time, expected surprisal does too, and the link to surprisal is linear. The evidence here can be labelled fairly firmly: many corpora, several language families, independent groups.
If you need a probability, something has to supply it. The usual measure of how well a language model guesses the next word is perplexity, which can be read loosely as "on average it is picking from about this many equally likely options" — the smaller the number, the better the guess.
In 2018 Goodkind and Bicknell reported a clean regularity: the lower a model's perplexity, the more of human reading time its surprisal explained, and the two traded off along roughly a straight line. The natural inference at the time was that bigger and stronger models would keep getting more humanlike.
In 2023 Oh and Schuler extended that line and found it bends. Within a single model family — they tested the GPT-Neo and OPT series against two reading corpora — the variants with more parameters and lower perplexity fit human reading times worse, and strictly monotonically: perplexity and goodness of fit stand in a stable log-linear relation, just running opposite to what people expected. The same year brought a sharper landmark: Transformer surprisal is at its most humanlike at around two billion training tokens, and drifts off past that.
The favoured explanation is sheer volume of training text. Having seen too much, a model is no longer surprised by rare personal names or unusual collocations — while people are still stumbling there. It guesses too well to guess the way a person does.
One step further out, things get messier. In 2021 Kuribayashi and colleagues ran the same comparison on a Japanese eyetracking corpus and found that "lower perplexity is more humanlike" does not hold in Japanese. Japanese puts its predicate last, so information is laid out through a sentence differently than in English; whether word order is the cause remains unsettled. Label this layer weaker: one language, one corpus.
English has a famous sentence: "The horse raced past the barn fell." Reading "raced", you almost certainly take it as the main verb — the horse ran past the barn — and sail on until "fell" collapses everything, forcing you back to reread "raced past the barn" as a modifier: the horse that was raced past the barn fell. Another of the same kind: "While the man hunted the deer ran into the woods", where "the deer" is first taken as the object of "hunted" and only at "ran" turns out to be the subject of the second half. These are garden-path sentences — you walk down an inviting path and reach the end only to find you must back out the way you came.
Surprisal does well here in one respect: it really does assign high values at "fell" and at "ran", pinpointing where a reader will stall. The trouble is the size. In 2021 van Schijndel and Linzen tested three classes of garden-path construction and found that surprisal drastically underpredicts how badly people actually stall — and that the size of the shortfall differs across constructions, which is the more damaging half, because it means no single exchange rate of milliseconds per bit can fit all three at once. In 2024 Huang and colleagues ran a much larger benchmark to the same conclusion, adding that surprisal cannot predict the variation among individual sentences within one construction either.
The sensible reading is that reading time carries at least two separate bills. One is the cost of a violated expectation; surprisal handles that one, and handles it well. The other is the cost of tearing down a structure already built and rebuilding it, which surprisal does not touch at all. The first explains where; the second explains how much.
So what does the correlation actually show? That people are sensitive to the statistics of their language, and that the sensitivity is logarithmic, incremental, and stable across language families. It does not show that a Transformer runs inside the head. Within this theory a model is only an estimator of probabilities, and how good an estimator it is stands apart from whether its internals resemble a brain. The reversal in the section on models that get too good separates those two things precisely: if goodness of fit measured brain-likeness, stronger models would fit better, and measurement says otherwise.
One family of answers treats reanalysis as its own step: it is not merely that the probabilities changed, but that a structure already built has to be dismantled. Another points to memory — retrieving an earlier constituent has a cost that has nothing to do with probability and everything to do with distance and interference. The two predict different things: the first says the cost tracks how much has to be torn down, the second says it tracks how far back you have to reach.
Because being humanlike does not demand accuracy, it demands being wrong in the same direction people are. People have not read that much text, and they rate rare words as far less likely — that is, far more surprising — than a large model does. A model as ignorant as a person is more useful for this particular job, which also warns against treating goodness of fit as a stand-in for model quality.
Two headaches. First, you must carve out "a word" before you can assign it a probability, and the way you carve changes the probability — the same problem as in the piece on word segmentation. Second, classifiers release part of the information before the noun arrives, so booking the cost only on the noun loses it. Cross-linguistic comparison has to be careful here: differences in fit may come from the bookkeeping rather than from the readers.