Ask a model for a paragraph of English and it writes more smoothly than most people. Ask it for one sentence of Kalamang — a language spoken by around 130 people in two villages on the Karas Islands off West Papua, Indonesia — and it cannot produce a word: there is almost no text about this language online.
So somebody took the grammar a linguist wrote after living in those two villages for eleven months, fed the whole book to a model, and had it read and translate on the spot. The setup is more interesting than the score, because it lays out on the table what the model is actually running on.
Can a discipline be made obsolete by a tool that outperforms it? No declaration settles that. You have to go item by item: which jobs really were taken over, which were not, and who defined the ones that were.
In 2024 Tanzer and colleagues built a benchmark called Machine Translation from One Book. Every piece of material comes from a single source: Eline Visser's 2022 grammar of Kalamang — several hundred pages of grammar, a word list with part-of-speech tags and English descriptions, and 375 paired Kalamang-English sentences. That is the model's whole input; then it translates sentences it has never seen.
Scores are the character-fragment overlap between the output and a reference translation (the measure is called chrF; higher is closer). The best set at the time reached 44.7 for Kalamang into English and 45.8 for English into Kalamang, against 51.6 and 57.0 for a single human learner working from the same materials — neither direction was caught up with.
The informative part is not the score but one specific error. The word list has an entry for teba, described as a progressive aspect marker — a particle that does the work of English "be ...-ing" and carries no action of its own. The model turned it into the English adverb "progressively" and wrote that straight into its translation. The shape of that error says it was consulting the word list, not applying the rule in the grammar.
This language can be processed at all because somebody went there for eleven months and recorded over a thousand example sentences. Of the world's roughly 7,000 languages, the overwhelming majority have about as much web text as Kalamang. The evidence strength needs stating plainly: one benchmark, one language, a very small test set, and weak support for any claim about what models can do. But "the model's entire input is a product of descriptive linguistics" is not a measurement — it is the precondition that lets the experiment exist.
In April 2023 a database called Grambank went public. It is one very large table: 195 logically independent grammatical features across the top (most of them yes-or-no questions), 2,467 languages down the side, over 400,000 cells, filled in by more than a hundred linguists working from each language's reference grammar.
Why could this never grow out of text? Because no cell holds a sentence; each holds a statement about a language. Whether a language puts its locational particles before or after the noun is not written inside any sentence of that language — it can only be summarised by someone who has read the whole of it.
The questions such a table makes askable look like this. In 1963 Greenberg noticed a correlation: languages that put the object before the verb tend to put locational particles after the noun (those are postpositions) rather than before it (prepositions). Japanese matches on both counts: the verb comes last, and in 東京に (Tōkyō ni, "to Tokyo") the particle に (ni) follows the noun; English puts the verb before its object and says to Tokyo. The correlation holds up well across the several-hundred-language sample in the World Atlas of Language Structures — though it is a statistical tendency rather than an iron law, and counterexamples exist.
The table's own result: grammatical features are predicted more by genealogy — which languages a language is related to — than by geography or environment; and counting by endangerment, close to half of the world's grammatical diversity is at risk of being lost.
The topic on whether large models learned grammar covered the minimal pair test: two sentences differing in one place, one licit and one not, and you check which one the model gives higher probability. Here we look at the other side of it — who wrote those items, and what they cover.
In 2020 Mueller and colleagues extended subject-verb agreement items from English into German, French, Russian and Hebrew, generating sentences in bulk from hand-written miniature grammars. The shape of the result: with no distractor between subject and verb, every language did well; once the agreement had to reach across an object relative clause (one of those chunks wedged in the middle) accuracy generally dropped; languages with richer morphology scored higher rather than lower; and multilingual models looked good on English and fell off noticeably elsewhere. That last observation became sayable only once the other languages had items at all.
In 2025 a set called MultiBLiMP pushed coverage to 101 languages and more than 128,000 minimal pairs, generated by a fully automatic pipeline. But the raw material is two hand-made things: Universal Dependencies (sentences annotated for syntactic structure under a shared scheme) and UniMorph (hand-compiled inflection tables). The way to widen an evaluation was not to crawl more text; it was to draw on more hand-assembled linguistic resources.
So the evaluation is the fulcrum: the question of whether a system handles some construction can only be asked once somebody has named that construction.
In 2023 Piantadosi published a piece titled "Modern language models refute Chomsky's approach to language". The core claim: a model is itself a theory of language, and a more successful one — it does produce grammatical sentences, while the rule system the older programme set out to write was never finished.
There have been at least two kinds of reply. Kodner, Payne and Heinz, in "Why Linguistics Will Thrive in the 21st Century", argue that models are a tool and not a theory, their hardest reason being data volume: a child gets there on far less than 100 million words of input, a model needs three or four orders of magnitude more, so the original puzzle has not been touched. Katzir, writing in Biolinguistics, takes another line: models learn patterns humans cannot learn and stumble where humans find things effortless, which makes them poor theories of human linguistic cognition.
These are argument papers, not new data, and the evidence strength should be labelled that way. But one strand really was turned into an experiment: the data budget. A public competition called BabyLM fixed the training budget at 100 million words — roughly the scale of what a child has heard by their teens, with a 10-million-word track as well — and asked who could do best inside it. In the first round a submission using the LTG-BERT architecture beat models trained on trillions of words on that evaluation suite, while intuitively appealing ideas like ordering the data by difficulty mostly did nothing.
The ledger should be kept honestly. What got displaced really was displaced — how hand-written grammar rules lost at translation is the subject of the topic on rules giving way to statistics. And in the BabyLM competition, the referee is exactly the kind of minimal-pair item from the previous section. Setting the strongest version of each position side by side: a model is currently the only artefact that produces language whole; it has not done so at a human data budget; and whether it counts as a theory of people probably cannot be answered before saying what it is supposed to explain. That one is still argued over.
Plentiful text can support "what the sayable sentences of this language look like" but not "which ones do not work". Absence from a corpus does not mean illicit, and the two cannot be distinguished inside text — while a fair share of a grammar is devoted to exactly the second.
You would first have to agree on what it is meant to explain. If the target is "on how much input do people get there", a fixed-data-budget competition is the test. If the target is "why languages come in only the observed shapes", you need a learner that is defeated by shapes that should not exist. The two targets are tested differently, so the question probably has to be split before it can be answered.
There is, and it is a real problem: the phenomena the items cover are the phenomena current theory considers important. One way out is to let model failures point back at what was left out (reading error patterns rather than totals); another is to draw material from languages no theory has looked at yet.