你让一个模型写一段英语，它写得比大多数人顺。你让它翻一句卡拉芒语（Kalamang）——印度尼西亚西巴布亚卡拉斯群岛上两个村子里大约一百三十个人说的语言——它一个字都翻不出来：网上关于这门语言的文本几乎是零。
于是有人把一位语言学家在那两个村子住了十一个月写成的那本语法书整本塞给模型，让它现读现翻。这个设置比成绩更有意思——它把「模型到底靠什么」摊在了桌面上。
一门学科会不会被一个比它更能干的工具淘汰？靠表态答不了，得一项一项看：哪些活真被接走了、哪些没有、被接走的那些当初是谁定义的。
2024 年 Tanzer 等人做了个叫「从一本书学翻译」的基准。全部材料只来自一个来源：Eline Visser 2022 年出版的《卡拉芒语语法》——几百页语法书正文、一份带词类标注和英文释义的词表、375 对卡拉芒语和英语的对照句。模型的输入就这些，然后翻没见过的句子。
成绩按译文和参考译文的字符片段重合度算（这个指标叫 chrF，越高越接近）。当时最好的一组是卡拉芒语译英语 44.7、英语译卡拉芒语 45.8；只读了同一批材料的一个人类学习者是 51.6 和 57.0——两头都没被追上。
最有信息的不是分数，是一个具体的错。词表里有个词 teba，释义说它是进行体标记——相当于汉语里「正在」那类成分，自己不表示任何动作。模型把它当成了英语副词 progressively，直接写进了译文。这个错的形状说明它在查词表，没在用语法书里那条规则。
这门语言能被处理，是因为有人去那里住了十一个月、记了一千多条例句。全世界约七千种语言，绝大多数的网络文本量和卡拉芒语同一个量级。证据强度得说清：单一基准、单一语言、很小的测试集，拿它评判模型能力相当弱。但「模型全部的输入都是描写语言学的产物」不是测量结果，是这个实验能成立的前提。
EN In 2024 Tanzer and colleagues built a benchmark called Machine Translation from One Book. Every piece of material comes from a single source: Eline Visser's 2022 grammar of Kalamang, a language spoken by roughly 130 people in two villages on the Karas Islands off West Papua, Indonesia. Three resources — several hundred pages of grammar, a bilingual word list with part-of-speech tags and English descriptions, and 375 paired Kalamang-English sentences. That is the model's entire input; then it translates sentences it has never seen. Scores are character-level overlap with a reference translation. The best baselines reached 44.7 for Kalamang into English and 45.8 for English into Kalamang, against 51.6 and 57.0 for a human who studied the same materials. Not bad, but short of the person in both directions. The informative part is not the score but one specific error. The word list has an entry for teba, described as a progressive aspect marker — a particle that carries no action of its own. The model rendered it as the English adverb "progressively" and put that straight into its output. The shape of that error says it was consulting the word list rather than applying the rule in the grammar. Text in this language is not on the web. It can be processed at all because somebody lived there for eleven months and recorded over a thousand example sentences. Of the world's roughly 7,000 languages, the overwhelming majority have about as much web text as Kalamang does. On evidence strength: this is one benchmark, one language, a small test set, and it is weak support for any claim about model ability. But the fact that the model's entire input is a product of descriptive linguistics is not a measurement — it is the precondition that lets the experiment exist.
2023 年 4 月公开了一个数据库，叫 Grambank。它是一张巨大的表：195 个互相独立的语法特征横着排（多数是「有没有」这种是非题），2467 种语言竖着排，四十多万个格子，由一百多位语言学家按各语言的参考语法书填进去。
为什么这东西不可能从文本里长出来？因为每一格都不是一句话，而是一句关于一门语言的话。「这门语言把方位词放在名词前还是后」不写在任何一句该语言的句子里——它只能被一个读遍了那门语言的人总结出来。
有了这种表才问得出来的问题长这样。1963 年 Greenberg 注意到一条关联：宾语放在动词前面的语言，倾向于把表方位的小词搁在名词后面（叫后置词 postposition），而不是前面（叫介词 preposition）。日语两头都对上：动词在句末，「東京に」（Tōkyō ni，「到东京」）里的 に（ni）跟在名词后头；英语动词在宾语前，于是说 to Tokyo。这条在 WALS 那套几百种语言的样本上很稳——但它是统计倾向不是铁律，反例有。
这张表自己的结果是：语法特征更多由谱系（跟哪些语言有亲缘）预测，而不是由地理或环境预测；按濒危程度算，接近一半的语法多样性正面临丢失。
EN In April 2023 a database called Grambank went public. Its shape is a very large table: 195 logically independent grammatical features across the top, most of them yes-or-no questions; 2,467 languages down the side; over 400,000 cells, filled in one at a time by more than a hundred linguists working from each language's reference grammar. Why could this never grow out of text? Because each cell is not a sentence but a statement about a language. Whether a language puts its locational particles before or after the noun is not written inside any sentence of that language — it can only be summarised by someone who has read through the whole of it. The questions such a table makes askable look like this. In 1963 Greenberg noticed a correlation: languages that put the object before the verb tend to put locational particles after the noun (those are postpositions) rather than before it (prepositions). Japanese matches on both counts: the verb comes last, and in 東京に (Tōkyō ni, "to Tokyo") the particle に (ni) follows the noun; English puts the verb before its object and says to Tokyo, with to in front. The correlation holds up statistically across the several-hundred-language sample in the World Atlas of Language Structures — though it is a statistical tendency, not an iron law, and counterexamples exist. Grambank's own headline result is worth stating too: a language's grammatical features are predicted more by its genealogy — which languages it is related to — than by its geography or environment; and counting by endangerment, close to half of the world's grammatical diversity is at risk of being lost. The loss described in the topic on disappearing languages gets a concrete accounting unit here.
「大模型学到语法了吗」那期讲过那种最小对（minimal pair）测试：给出一对只差一个地方的句子，一句合法一句不合法，看模型给哪句更高的概率。这期要看的是它的另一面——这些题是谁出的，以及覆盖了什么。
2020 年 Mueller 等人把主谓一致的题从英语扩到德语、法语、俄语和希伯来语，办法是用人工写的小语法成批生成句子。结果的形状：主语和动词之间没有干扰词时各语言都做得好；一旦一致关系要跨过一个宾语关系从句（插在中间的那种成分）就普遍变差；词形变化更丰富的语言成绩反而更高；多语种模型在英语上分数漂亮，换到别的语言明显掉。最后这句只有在别的语言也有了题之后才说得出口。
2025 年有一套叫 MultiBLiMP 的题集把覆盖推到 101 种语言、十二万八千多对最小对，全自动流水线生成。但原料是两样人工的东西：Universal Dependencies（按统一标准人工标注过句法结构的句子库）和 UniMorph（人工整理的词形变化表）。把评测扩大的办法不是多爬文本，是去用更多手工攒出来的语言学资源。
所以评测是支点：「这个系统会不会某个结构」这个问句，得先有人把那个结构命名出来才问得出来。
EN The topic on whether large models learned grammar covered the minimal-pair test: a pair of sentences differing in one place, one licit and one not, and you check which one the model assigns higher probability. Here we look at the other side of it — who wrote the items, and what they cover. In 2020 Mueller and colleagues extended subject-verb agreement items from English into German, French, Russian and Hebrew, generating sentences in bulk from hand-written miniature grammars. The shape of the result is instructive: with no distractor between subject and verb, models did well in every language; once the agreement had to reach across an object relative clause — a chunk like "that I repaired yesterday" wedged in the middle — accuracy generally dropped; languages with richer morphology scored higher rather than lower; and multilingual models looked good on English while falling off noticeably elsewhere. Note that last observation: it became sayable only once the other languages had items at all. In 2025 a set called MultiBLiMP pushed coverage to 101 languages and more than 128,000 minimal pairs, generated by a fully automatic pipeline. But the raw material is two things: Universal Dependencies, a collection of sentences hand-annotated for syntactic structure under a shared scheme, and UniMorph, a set of hand-compiled inflection tables. The way to widen an evaluation, in other words, was not to crawl more text but to draw on more hand-built linguistic resources. So the evaluation is the fulcrum. The very question of whether a system handles some construction requires that somebody first name that construction and state when it holds and when it does not. Whoever writes the items decides what "learned it" means.
2023 年 Piantadosi 的一篇文章题目就叫《现代语言模型驳倒了乔姆斯基式的语言研究路线》。核心主张是：模型本身就是一种语言理论，而且更成功——它确实产出合乎语法的句子，而过去那条路线想写的那套规则始终没写完。
回应至少两类。Kodner、Payne 和 Heinz 的《语言学会在二十一世纪繁荣起来》说模型是工具不是理论，最硬的理由是数据量：孩子在远少于一亿词的输入下就学成，模型要多出三四个数量级，所以它没触及那个最初的难题。Katzir 发在《生物语言学》上的那篇走另一条：模型在人学不会的模式上也能学，在人毫不费力的地方反而会掉，所以拿它当人类语言认知的理论是差的。
这几篇都是论证文章不是新数据，证据强度要这么标。但其中一条真被做成了实验：数据量。有个叫 BabyLM 的公开比赛定了固定训练预算：一亿词，接近一个孩子长到十几岁听到的量级（另有一千万词的小档），比谁能在预算内把模型做得最好。第一届里一个用 LTG-BERT 架构的参赛作品，在那套评测上超过了用上万亿词训出来的模型；而「按难度给数据排顺序」这类直觉办法大多没效果。
账要诚实地记。被淘汰的那部分是真被淘汰了——手写语法规则去跑翻译这条路怎么输的，「从规则到统计」那期讲的就是它。而 BabyLM 这场比赛，裁判用的正是上一节那种最小对。三方最强的版本并排放：模型是目前唯一能完整产出语言的东西；它确实没在人类的数据量上做到；而「它算不算一种关于人的理论」，多半得先说清要解释什么才答得了。这个今天还有争论。
EN In 2023 Piantadosi published a piece titled "Modern language models refute Chomsky's approach to language". The core claim: a model is itself a theory of language, and a more successful one than what came before, because it does produce grammatical sentences while the rule system the older programme set out to write was never finished. There have been at least two kinds of reply. Kodner, Payne and Heinz, in "Why Linguistics Will Thrive in the 21st Century", hold that models are a tool and not a theory, their hardest argument being about data volume: a child gets there on far less than 100 million words of input while a model needs three or four orders of magnitude more, so the original puzzle has not been touched. Katzir, writing in Biolinguistics, takes a different line: models learn patterns humans cannot learn and stumble where humans find things effortless, which makes them poor theories of human linguistic cognition. These are argument papers, not new data, and the evidence strength should be labelled accordingly. But one strand did get turned into an experiment: the data budget. A public competition called BabyLM fixed the training budget — 100 million words, roughly the scale of what a child has heard by their teens, with a 10-million-word track as well — and asked entrants to do the best they could inside it. In the first round a submission using the LTG-BERT architecture beat models trained on trillions of words on that evaluation suite, while intuitively appealing ideas like ordering the data by difficulty mostly did nothing. In the second round, among 31 submissions, the winner mixed causal and masked training objectives, and a strong relationship turned up between training compute and average score. The ledger should be kept honestly. What got displaced really was displaced: how hand-written grammar rules lost at translation is the subject of the topic on rules giving way to statistics. And the referee in the BabyLM competition is exactly the minimal-pair kind of item from the previous section. Setting the strongest version of each position side by side: a model is currently the only artefact that produces language whole; it has not done so at a human data budget; and whether it constitutes a theory of people probably cannot be answered before saying what it is supposed to explain. That one is still argued over.
文本多能撑起「这门语言里说得通的句子长什么样」，撑不起「哪些说不通」。语料里没出现不等于不合法，这两件事在文本里没法区分，而语法书有不少篇幅恰恰在讲后者。
得先约定它要解释什么。如果要解释「人在多少输入量下学成」，固定数据预算那类比赛就是检验；如果要解释「语言为什么只长成观察到的那些形状」，那就要一个会被不该存在的形状难住的学习器。两个目标的检验方式不同，这问题多半得拆开答。
会，而且这是真问题：题覆盖的现象就是现有理论认为重要的那些现象。一条出路是让模型的失败反过来提示被漏掉的现象（看错误模式而不只看总分），另一条是从还没进过任何理论视野的语言里取材。