You say "can you pass the salt" at dinner, and nobody answers "yes". Yet grammatically it really is a question, and "yes" would be a perfectly sensible reply. Meanwhile the thing you actually wanted — the salt pushed toward you — appears nowhere in what you said.
This happens dozens of times a day, smoothly enough that you never notice. It asks the listener to do something laborious inside half a second: work out what you said, then work out why you chose to say it that way, then work out what you want.
How that step is done, when people install it, and whether machines have it — the third question is the hardest, and the difficulty is not in the machines. It is in the test.
In 1979 Herbert Clark had someone telephone fifty restaurants and ask the same thing each time: "Do you accept credit cards?" What he wanted was the shape of the replies. They fell into three piles: 44% answered only the literal question ("yes"), 38% answered the literal question and then filled in the rest ("yes, Visa and Mastercard"), and 16% skipped straight to treating it as a request and listed the cards. One sentence, three treatments — the literal layer does not vanish, and the step from wording to intention is not bolted on automatically; it is a move you may or may not make.
How is the move made? The account Grice gave in 1975 is still in use: a listener assumes the speaker is cooperating in the conversation, and therefore asks why this particular wording was chosen over a plainer one, recovering the unsaid part from the gap. What gets recovered that way is called conversational implicature. How that reasoning itself runs was the subject of the topic on saying less than you mean; what this one needs is its price — without a price on it there is no way to compare people and machines.
But "can you pass the salt" is worn smooth — worn enough that nothing needs computing. In 1983 Gibbs had people read that sentence and then immediately judge a target related to the literal question of ability: if the literal reading really came first and was then discarded, that target should have been primed and the judgement should have sped up. He saw no speed-up. Frozen indirect requests are recognized whole.
So the same phenomenon runs on two paths: ready-made forms take the shortcut, novel wordings get computed on the spot. And which forms count as ready-made is not constant across languages — the Cross-Cultural Speech Act Realization Project led by Blum-Kulka and colleagues in 1989 ranked requests in several languages by directness, with Argentinian Spanish coming out most direct, Australian English least, and German more direct than any of the English varieties. Do not read that as a ranking of politeness: one of the project's own conclusions is that indirectness and politeness are not the same thing.
"Some elephants have trunks." The sentence grates, and not because it is false — all elephants have trunks, so it is plainly true. What grates is that the speaker could have said "all elephants have trunks" and picked the weaker word instead, from which you read off "not all of them". What gets inferred from a stronger alternative having been passed over is called scalar implicature.
In 2001 Noveck put sentences like this to children aged eight to ten and to adults, asking only whether each one was true. 87% of the children said yes; among adults only 41% did. The children are the more logical party here: "some" does not logically conflict with "all", so judging by the literal meaning gets you the right answer. It is the adults who computed an extra step.
What does that extra step require? It requires having a list of the wordings the speaker could have used, knowing which entries are stronger and which weaker, and registering that the strong one was skipped. In other words, it requires a model of the other person's mind.
The classic psychological test of that model runs like this: show a child a sweets box containing pencils, then ask what a child who is about to walk in will think is inside. Children under four mostly answer "pencils" — what they know, they assume everyone knows. The task is usually passed around age four.
Set the two side by side: working out what was meant, and knowing what someone else does not know, take the same machinery. So indirect speech is not ornament but an extra inferential bill — and the bill is visible: children install the step late, and even adults run scalar implicature only about 40% of the time.
In 2023 Hu and colleagues assembled a set of items covering seven pragmatic phenomena: white lies, indirect requests, irony, conversational maxims, metaphor, humour and coherence. English materials, expert-written items, no examples given — just the question — with humans and models answering the same set. The largest models of the day beat chance on every phenomenon and came close to human level on some; the three weakest were humour, irony and the maxims.
That same year Ruis and colleagues tested conversational implicature alone, compressing items until only "yes" or "no" was possible, which puts chance at 50%. Pretrained models without instruction tuning scored around 60%; humans scored 86%. But the same base models, once instruction-tuned at the level of individual examples, jumped a long way up. That result carries weight: if hearing what was meant were an ability that simply arrives with scale, it should not be this sensitive to fine-tuning strategy — same parameter count, same corpus, and a different late-stage alignment recipe makes the score look like a different system's.
A third axis of wobble comes from language. Translate the same battery of theory-of-mind items into several languages, as the XToM line of work did, and models answer "what is actually in the box" stably across languages, while "what does he think is in it" drops, by differing amounts in different languages. How strong is the evidence? Each of these is a single benchmark with zero-shot prompting, and any one alone counts as weak. But the three point the same way: what is being measured is not a stable ability, it is a performance that swings widely with materials, language and training recipe.
In 2023 a paper reported that GPT models reached school-age performance on classic false-belief items. That same year Ullman altered the wording slightly while leaving the logic untouched: the original item has popcorn inside a bag labelled "chocolate", and he made the bag transparent — if you can see inside, the label cannot fool you. The models got it wrong. Across several such minor alterations, the strongest models of the day all fell.
In 2024 Strachan and colleagues ran a far larger comparison: a full battery of established psychological items (false belief, the hinting task, Strange Stories, faux pas recognition) against roughly 1,900 human participants. GPT-4 matched or exceeded humans on irony, hinting and the Strange Stories, with faux pas the one test it did not match — and LLaMA2-70B, the weakest model overall, beat humans on exactly that test.
The authors dug out the reason themselves: in the original faux pas items the correct answer always points the same way, so a model biased toward that answer scores perfectly by accident. They then rephrased the question for GPT-4 and found it had in fact inferred that the speaker did not know — it simply would not commit to the conclusion. They call this hyperconservatism.
So two quite different confusions are in play. A wrong answer may mean the inference could not be made, or that it was made and not stated. A right answer may mean understanding, or a set of items whose answers all point one way. Neither is a flaw in the model; both are flaws in the test, and both surfaced only after an extra step of work.
The argument today runs roughly three ways. One camp says these are behavioural tests by construction, that humans are credited with theory of mind on behavioural grounds too, and that holding machines to a different standard is unfair. A second says the human versions of these items and their answer keys sit in the training data in bulk, and evidence that cannot clear a contamination check is not evidence. A third wants tasks where beliefs must be updated across turns of interaction instead. Each camp has a strong version, and there is no settled answer.
Here is a genuinely testable difference. If a model runs the same inference people run, its errors should point where human errors point — human failures on scalar implicature are retreats to the literal reading, not random answers. Error patterns therefore carry more information than totals, which is exactly what Hu and colleagues reported. On the human side there is a trace of cost with no counterpart in a model: the same person under time pressure, or working in a language they know less well, retreats to literal readings across the board — the retreat described in the topic on the foreign-language effect. A system that pays no such bill may be scoring the same total while doing something else entirely.
By result, yes; by process, no. That creates a measurement problem: a test built entirely from ready-made forms is really measuring how many of those forms you have met. To measure on-the-fly inference you need freshly invented detours — but those are harder for people too, so the human baseline drops along with everything else.
Scalar implicature depends on context by nature: whether the speaker could plausibly know the full picture, and whether they had reason to say so, both change whether the inference should be drawn — and a bare "some elephants have trunks" removes exactly those cues. So 40% reads less like a score for human ability and more like evidence that the inference is optional.
The first is easy: give every item a logically mirrored twin with the correct answer flipped, and look only at the gap between the two versions. The second is much harder, since the classic items and their keys have long been online. One route is to write items fresh and never publish them; another is to make the task interactive, so the system has to revise a judgement across turns in response to what the other party does — a process that is hard to have memorized.