大模型学到语法了吗

把句子里的实词全换成胡编的词，模型照样挑得对动词的单复数

中间那个捣乱的名词叫吸引项（attractor，「吸引的东西」）。Bock 和 Miller 让人边看材料边说话，发现一个很稳的不对称：主语单数、后面跟一个复数名词时，说错的比例明显上升；反过来错得少得多。这个不对称三十多年里在好几门语言上被反复复制过。

2016 年，Linzen、Dupoux、Goldberg 把同一道题端给机器：从维基百科捞真实句子，训一个循环网络（recurrent network，按顺序一个词一个词往后读的网络）判断动词该用单数还是复数。中间没有干扰名词时它几乎不错；干扰名词一个个加进去，错误率一层层往上抬——形状和人那边像得让人意外。

但他们真正想说的是另一句：把训练目标从「专门判断单复数」换成「只预测下一个词」——今天所有大模型唯一在做的那件事——同一个网络明显退步。结论写得很直接：只靠语言模型这一个信号，不够。

两年后这句话被顶了回去。2018 年 Gulordava 等人训的就是普通语言模型，四门语言：英语、意大利语、希伯来语、俄语。关键在测试句——实词全换成同类的胡编词，像「The colorless green ideas I ate with the chair sleep furiously」这种，意思全毁，只剩骨架和词尾。模型还是能可靠地挑对动词的数，意大利语上跟人的直觉差得不多。

换词是整件事的枢纽：不换的话，模型不懂结构也能答对——「钥匙……是」这类搭配在语料里本来就常一起出现，词的意思由它的邻居决定那期讲的那套统计能替它蒙掉一大半。生造词一上，共现统计就断了。意大利语的例子长这样：le chiavi dell'armadio sono sul tavolo（字面「钥匙 的-柜子 是 在桌上」），动词 sono（是，复数）跟 chiavi（钥匙，复数）配，不跟紧挨着它的 armadio（柜子，单数）配。

证据强度得说清楚：四门语言都带着丰富的词形变化，谈不上跨语言类型的样本；两组结论的张力也没被「谁对」解决——数据量、网络大小、测试句来源都不同。今天公允的说法是：模型越大这类测试做得越稳，分歧于是整体挪了位置，从「做不做得到」挪到「做到了意味着什么」。

EN The troublemaker in the middle has a name: an attractor. Bock and Miller (1991) had people produce sentences and found a stable asymmetry — a singular subject followed by a plural noun draws far more errors than a plural subject followed by a singular one. The asymmetry has been replicated across several languages over three decades. In 2016 Linzen, Dupoux and Goldberg handed the same problem to a machine: a recurrent network, trained on real Wikipedia sentences, asked to say whether the verb should be singular or plural. With no intervening noun it almost never erred; errors climbed step by step as attractors were added, in a shape strikingly like the human one. Their real point, though, was different: swap the training objective from "predict the number" to "predict the next word" — the only thing today's large models do — and the same network gets visibly worse. Their conclusion was blunt: the language-modelling signal alone is not enough. Two years later that was pushed back on. Gulordava and colleagues (2018) trained plain language models in four languages — English, Italian, Hebrew, Russian — and tested them on nonce sentences, with content words replaced by invented words of the same class, as in "The colorless green ideas I ate with the chair sleep furiously". Meaning is destroyed; only the skeleton and the endings remain. The models still picked the right number reliably, and in Italian came close to human intuitions. That substitution is the hinge of the whole enterprise: without it a model can answer correctly while knowing nothing about structure, since a pairing like "key … is" co-occurs often in the corpus, and the distributional statistics discussed in the issue on a word being known by the company it keeps would carry much of the load. Replace the words with invented ones and the co-occurrence crutch is gone. The Italian case looks like this: "le chiavi dell'armadio sono sul tavolo" — the verb "sono" (are) agrees with "chiavi" (keys, plural), not with the adjacent "armadio" (wardrobe, singular). Evidence strength matters here: four languages, but all four richly inflected, so this is no typological sample. And the tension between the 2016 and 2018 results was never resolved by one side being right — data volume, network size and test material all differed. The fair summary today is that larger models do these agreement tests reliably, and the disagreement has moved from whether they can to what it means that they can.

探针能从模型内部读出句子的骨架 —— 可给它随机标签，它照样学得会

行为测试只看得到进去什么、出来什么。想看里面，办法叫探针（probe，探测器的意思）：拿模型内部状态当原料，在上面训一个很小的读出器，看某种信息能不能被读出来。2019 年 Hewitt 和 Manning 做了个很漂亮的版本。句子结构可以理解成一组距离：任意两个词之间隔着几步，直接搭在一起的近，绕好几层才挂上的远。他们找了一个线性变换（对所有向量做同一次拉伸和旋转），让两个词的向量距离对应上这个步数。在 BERT、ELMo（两个当时常用的模型的名字）上，句子层级能还原出相当一部分，对照的表示上还原不出来。同年 Tenney 等人发现层与层像条流水线：靠前的管词类，中间管句子结构，指代更靠后才出来。

同一年，Hewitt 和 Liang 给这套方法装了个照妖镜，叫对照任务（control task）：给每个词型随机指派一个标签。这种标签没有任何语言信息，能学会只可能是探针自己硬背下来的。结果当时流行的几种探针在随机标签上也拿到了很高的准确率。于是「探针准确率高」本身什么都不说明——要看差值：真任务减掉对照任务，他们管这叫选择性（selectivity）。

更要紧的一层是：读得出来不等于模型自己在用。2019 年 Lakretz 等人把一个循环语言模型拆到单个神经元去看，发现跨长距离的单复数信息主要压在极少数几个单元上，这几个单元又受另一批跟结构相关的单元管着；把它们关掉，长距离一致性就塌了。关掉就坏是因果证据，比「能读出来」硬一个档次。证据强度：单个模型、单个架构，搬不到别的模型上。

差别有多大值得掂量：一份内部状态里能读出来的东西，远多于模型那条路上真正用到的。体温里读得出季节，可体温不是你决定穿什么的线路。

EN Behavioural tests see only what goes in and what comes out. To look inside, the method is probing: take the model's internal states as raw material, train a very small readout on top, and see whether some kind of information can be recovered. In 2019 Hewitt and Manning built an elegant version. Think of sentence structure as a set of distances: any two words are some number of steps apart, close if they attach directly, far if the link runs through several layers. They found a single linear transformation — one uniform stretch and rotation applied to every vector — under which the distance between two word vectors matches that number of steps. On BERT and ELMo (names of two models common at the time) a substantial part of the hierarchy could be recovered, and on the control representations it could not. The same year, Tenney and colleagues found the layers behaving like a pipeline: earlier layers handle word classes, middle layers sentence structure, and coreference surfaces later still. Also in 2019, Hewitt and Liang fitted the method with a mirror that exposes fakes: the control task, which assigns every word type a random label. Such labels carry no linguistic information, so the only way to learn them is for the probe itself to memorise them. Several popular probes scored high on the random labels too. High probe accuracy therefore says nothing on its own; what counts is the gap — real-task score minus control-task score — which they call selectivity. And there is a deeper layer: decodable is not the same as used. In 2019 Lakretz and colleagues opened up a recurrent language model at the level of single neurons and found that long-distance number information is carried mostly by a very small number of units, themselves governed by another set of units that track structure; switch those units off and long-distance agreement collapses. Breaking it by switching it off is causal evidence, a notch stronger than mere decodability — though this is one model and one architecture, and does not transfer automatically. The size of the distinction is worth weighing: far more can be read out of an internal state than the system's own pathway ever consults. Your body temperature is enough to read off the season, but body temperature is not the circuit deciding what you wear today.

把英语改造成一门人学不会的语言，再训一遍，模型学得明显更费劲

有一种很有力的主张：大模型对「人能学的语言」和「人学不会的语言」一视同仁，所以它学到的跟人的语言能力无关。这句话的好处是能直接做实验。2024 年 Kallini 等人拿英语造了 15 门「不可能的语言」，分三类：按固定规则打乱词序；把句子整段或部分倒过来；把动词的单复数标记改成「往后数几个词」的规则。每门都用同一套 GPT-2 架构从头训，和正常英语作对照。结果这些语言都学得更费劲——衡量用困惑度（perplexity，粗略说是模型对下一个词猜得有多准，越低越准）。

结论立刻被两头夹击。一头说对照组不公平：2025 年有篇文章专门论证它没拿真正按成分结构造出来的语言来比。另一头指出表现最差的那门其实是句内完全随机打乱，那门「语言」几乎没有规律可学，赢它不算赢；也有工作直接主张概率、困惑度这类指标根本分不开可能与不可能。现在的状态是：范式站得住，具体结论还在打。

两边在一件事上不吵：模型不是白板，它有偏好；至于是不是人的那种偏好，还没被测出来。要测干净得让两边吃差不多的东西——BabyLM 挑战就是这个思路，把训练数据卡在一亿词以内，那大致是一个孩子长到十三岁听过的量级。今天的大模型见过的是这个数目的几千倍。

最后一条常被混掉的线：成套的最小对（minimal pair，一对只差一处的句子，一个合语法一个不合）测试。BLiMP 有 67 组、每组一千对，人对这些标签的整体认同率约 96%。结果形状很一致：跟词形一致性有关的题模型很稳，更细的现象明显差——比如英语的 any 只能出现在某些否定或疑问环境里（「I didn't see any」行，「I saw any」不行）。所以「学到语法了吗」不是一个数能回答的问题，是几十条各自独立的曲线。

EN One forceful claim is that large models treat learnable and humanly unlearnable languages alike, so whatever they acquire has nothing to do with the human language capacity. Its virtue is that it can be tested. In 2024 Kallini and colleagues built 15 "impossible languages" out of English, in three classes: words shuffled within sentences by fixed rules, sentences reversed in whole or part, and verb number marking rewired into a rule that counts a few words ahead. Each was trained from scratch on the same GPT-2 architecture and compared against unmodified English as a control. All of them came out harder to learn, measured by perplexity — roughly, how well the model guesses the next word, lower being better. The conclusion was immediately squeezed from two sides. One argues the comparison is unfair: a 2025 paper makes the case that no genuinely constituency-based language was among the comparisons, so the gap cannot support the claim. The other points out that the worst-performing case was full within-sentence shuffling, a "language" with almost no regularity to learn, so beating it proves little; and other work argues that probability and perplexity metrics cannot separate possible from impossible at all. The paradigm stands; the specific conclusion is still being fought over. Notably, both sides agree on one thing: the model is not a blank slate, it has a bias — and whether that bias is the human one has not been measured. Measuring it cleanly requires feeding both sides comparable input, which is what the BabyLM challenge does by capping training data at 100 million words, roughly the order of what a child has heard by thirteen. Today's large models have seen thousands of times that. One last line often collapsed into the others: batteries of minimal pairs, sentence pairs differing in one place where one is grammatical and one is not. BLiMP has 67 sets of a thousand pairs each, with overall human agreement with the labels around 96 percent. The shape of the results is consistent: models are solid on items involving morphological agreement and visibly weaker on finer phenomena — for instance that English "any" is licensed only in certain negative or interrogative environments ("I didn't see any" works, "I saw any" does not), or whether a phrase can be extracted out of certain structures. Which is exactly why "did it learn grammar" is not a question one number can answer; it is dozens of separate curves.