切词这件小事

六个母语者给同一百句中文划词，两两之间只对上七成六

歧义句其实少，上下文通常也能定。难的是普通句子。

1996 年，Sproat、Shih、Gale、Chang 四个人做了件很朴素的事：从语料里随机抽 100 句中文、共 4372 个字，请六位母语者（三位来自大陆、三位来自台湾）各自划出词的边界，再把每两个人的答案对一遍。两两之间的一致度平均只有约 76%。

分歧不在读懂没读懂，在「词」这个单位算到哪一层。「北京大学」是一个词还是两个？「吃饭了」里那个「了」算不算进去？这些地方没有对错，只有约定。约定不统一的后果直接可见：2003 年第一届国际中文分词评测，参评的标注语料来自北京大学、台湾中研院、香港城市大学等好几家，划法互不相同，评测只能按不同标准分别算分——同一个系统换一份标准，分数明显变化。「分词准确率 98%」这种数字，从来都是对着某一份约定说的。

最容易被忽略的是：这不是中文独有的困境。英语有空格，可 don't 是一个空格块装了两个词，New York 是两个空格块合成一个词。越南语更直白：它写空格，但空格隔开的是音节不是词——thuế thu nhập cá nhân 五个音节其实是两个词，thuế thu nhập（所得税）加 cá nhân（个人）。泰语干脆不在词之间写空格，空格留给句子，于是 ข้าวเย็น（罗马转写 khâao yen，字面「饭」加「凉／晚」）既可以是「晚饭」这个词，也可以是「凉了的饭」这个短语。

所以空格是排印约定，不是语言事实：它把边界问题外包给了几百年前定下书写习惯的人；没空格的语言只是把题留在原地，谁要处理谁自己答。

EN Set the funny sentences aside: 乒乓球拍卖完了 (pīngpāngqiú pāimài wán le) can be cut as "ping-pong paddles / sold out" or "ping-pong balls / auctioned / finished", but such strings are rare and context usually settles them. The hard cases are ordinary sentences. In 1996 Sproat, Shih, Gale and Chang took 100 Chinese sentences at random, 4372 characters in all, asked six native speakers (three from the mainland, three from Taiwan) to mark word boundaries, and compared every pair: average agreement was only about 76 percent. The disagreement is not about comprehension but about what counts as one word. Is 北京大学 (Běijīng Dàxué, "Peking University") one word or two? Does the aspect marker 了 (le) in 吃饭了 (chīfàn le, "has eaten") belong inside the word? There is no right answer, only convention — and conventions differ. The first international Chinese word segmentation bakeoff in 2003 had to score systems against several separate gold standards, from Peking University, Academia Sinica, City University of Hong Kong and Microsoft Research Asia, because no single one existed; swap the standard and the same system's score moves visibly. "98 percent segmentation accuracy" is always relative to one convention. None of this is peculiar to Chinese. English has spaces, yet "don't" packs two words into one space-delimited block, "New York" spreads one word across two, and dictionaries disagree about "ice cream". Vietnamese is blunter: its spaces separate syllables, not words, so "thuế thu nhập cá nhân" is five syllables but two words — "thuế thu nhập" (income tax) plus "cá nhân" (individual). Thai writes no spaces between words at all, reserving them for clauses, so ข้าวเย็น (khâao yen, literally "rice" plus "cool/evening") is either the single word for dinner or the phrase "cold rice". Spaces are a typographic convention, not a linguistic fact.

一个 1994 年为了压缩文件写的算法，成了今天每个大模型的第一道工序

1994 年，Philip Gage 在一本面向 C 语言程序员的杂志上发了篇很短的文章，讲一个压缩文件的土办法，后来叫字节对编码（byte pair encoding，常简称 BPE）。做法一句话讲完：找出文件里出现最多的那一对相邻字节，换成一个没用过的新符号，记一笔账，然后重复。高频组合越并越长，文件就短了。

拿一小堆词演示。假设手上只有 low、lower、newer、wider，先全拆成单个字母。数一遍相邻字母对：e 后面跟 r 出现三次，最多，于是把 er 并成一块；再数，再并。几千次之后，常见的词整块进了表，罕见的词留成碎片。整个过程只用一件信息：谁和谁常挨着。

2016 年，Sennrich、Haddow、Birch 把这个压缩算法搬进机器翻译，治的是罕见词：词表就那么大，遇到表外的词，老系统只能吐一个「未知」；切成碎片就没有表外这回事了。

有个关键容易滑过去：BPE 切出来的不是语素（morpheme，就是还带着意思的最小单位，像 un- 和 -ing）。它不知道 un- 是前缀，只知道哪两块常挨着——有时碰巧对上，有时完全对不上，一个生僻人名可能被切成六块，每块单独什么也不表示。词是拼出来的那期讲的有意思的零件，跟这里的碎片只偶尔重合。

而它最狠、也最容易被误解的一招是：它压根没回答「词是什么」。划词边界那节里让六位母语者吵起来的那道题，BPE 一个字都没答，它换了个不需要答案的问题——哪些字符串常连在一起。这跟手写语法输给数数那期是同一种赢法：胜出的招不是更懂语言，是承认自己不懂然后绕开。

后来的模型走得更远，干脆从字节切起，好处是任何文字都不会出表外；代价对中文特别明显——UTF-8 里一个汉字占三个字节，不常见的汉字会被切成三个谁也不表示什么的 token。同一句「乒乓球拍卖完了」，按词三块、按字七块、按字节二十一块。哪个是「自然」单位？没有一个是。

EN In 1994 Philip Gage published a short piece in a magazine for C programmers describing a crude file-compression trick, now known as byte pair encoding, or BPE. The rule fits in a sentence: find the most frequent adjacent pair of bytes, replace it with an unused symbol, record the substitution, repeat. Take a tiny vocabulary — low, lower, newer, wider — split into single letters. Counting adjacent pairs, "e" followed by "r" occurs three times, more than anything else, so "er" becomes one unit; count again, merge again, and after a few thousand rounds common words are single units while rarer ones survive as fragments. The only information used is what tends to sit next to what. In 2016 Sennrich, Haddow and Birch moved this compression algorithm into machine translation to cure the rare-word problem: with a fixed vocabulary anything outside it became a single "unknown" token, whereas fragments make every word representable. What is easy to miss is that BPE does not produce morphemes. It has no notion that "un-" is a prefix or "-ing" a suffix; it knows only co-occurrence. Sometimes the pieces coincide with meaningful units, often they do not, and an unusual proper name may come out as six fragments that individually denote nothing. The sharpest and most misread thing about it is that it never answers the question the six native speakers disagreed about. It substitutes a question that needs no answer: which character strings habitually occur together. Later models went further and cut from raw bytes, so no script is ever out of vocabulary — at a cost conspicuous for Chinese, since UTF-8 encodes each character in three bytes, so an uncommon character can become three tokens that individually mean nothing. The same seven-character string 乒乓球拍卖完了 (pīngpāngqiú pāimài wán le, "the ping-pong balls have been auctioned off") is three units by word, seven by character, twenty-one by byte. None of these is the natural one.

给数字加上千分位逗号，模型的算术就准了一截

切法在训练开始之前就定下来，定完冻住。冻住的那几刀，会一路长成模型的能力和缺陷。

最好量的是长度。2023 年 Petrov 等人把同一段文本翻成很多种语言，量各家切法下的 token 数：最长和最短差到 15 倍。这直接换算成按 token 计费的价钱、生成的延迟、和同样的上下文窗口能装多少内容。别把这读成语言的高下：token 表是在一批语料上拟合出来的，那批语料里谁的文本多，谁的切法就便宜——这是语料构成的事实，不是语言的事实。

更精巧的是数字。GPT-3.5 和 GPT-4 那套 token 表给一位到三位的数字串都留了位置，切的时候从左往右贪心地切：12345 被切成 123 和 45。于是同样是「123」这一块，在一个数里站千位、在另一个数里站个位——块边界跟着数字总长度走，不跟着位值走。2024 年 Singh 和 Strouse 试了一招：给数字加上千分位逗号，写成 1,234,567。逗号是硬边界，每一块就都对齐到固定位值上了。加了逗号，这两个模型做加法的正确率明显上升——同一个模型、同一道题，只改了怎么切。

还有那个被玩坏的例子：问模型 strawberry 里有几个 r。它答不上来不是不会数，是它看见的不是十个字母，是两三个整块，字母封在块里面。这也不是绝对做不到——2025 年有工作发现，字符层面的能力在训练中确实会冒出来，只是很晚很突然。两件事的证据强度不同：加逗号是做过对照的干预，因果比较硬；「字母被封住」解释得通、有旁证，但不是唯一解释。

最古怪的一类是故障 token：token 表和模型是分两步训出来的，有的字符串在训 token 表时够多、进了表，训模型时那批数据却被清掉了。2023 年有人往 GPT 里喂 SolidGoldMagikarp 这串字符，模型的反应整个失常——表里有位置，经验里没有内容。

人也有自己的格子——大脑给声音画的格子那期讲过，压掉的区分自己就听不见了。但这里有个方向明确、能验的差别：人的格子可以重开。分不清英语里 l 和 r 的成年人经过针对性训练能把辨别正确率练上去；你也随时能把一个词一个字母一个字母拼出来。模型的 token 表在见到第一个训练样本之前就写死了，之后只能绕，绕不开的地方就成了那些傻得离奇的错误。

EN Segmentation is decided before training starts and then frozen; those cuts grow into the model's abilities and its defects. The easiest thing to measure is length. Petrov and colleagues (2023) took one passage translated into many languages and counted tokens under standard tokenizers: longest to shortest differed by up to fifteen times, which converts directly into per-token price, latency, and how much fits in a fixed context window. This is not a ranking of languages: a token vocabulary is fitted to a corpus, and whoever's text dominates that corpus gets the cheap encoding. It is a fact about corpora, not about languages. Numbers are subtler. The GPT-3.5 and GPT-4 vocabulary holds entries for one-, two- and three-digit strings and cuts digit runs greedily left to right, so 12345 becomes 123 and 45 — the same chunk "123" therefore sits in the thousands place in one number and the units place in another, because chunk boundaries follow total length rather than place value. Singh and Strouse (2024) inserted thousands separators, writing 1,234,567: the comma is a hard boundary, so every chunk aligns to a fixed place value, and addition accuracy rose measurably for both models. Same model, same problem, only the cutting changed. Then the worn-out example: asked how many r's are in "strawberry", the model sees not ten letters but two or three blocks, with the letters sealed inside. This is not flatly impossible — work in 2025 found that character-level ability does emerge during training, but late and abruptly. The evidence differs in strength: the comma result is a controlled intervention, so the causal claim is firm, whereas "the letters are sealed in" is a plausible, well-supported explanation rather than the only one. The strangest class is glitch tokens. In 2023 people found that feeding GPT the string "SolidGoldMagikarp" derailed it completely, because tokenizer and model are trained in two separate steps: the string was frequent enough in the tokenizer's data to earn a vocabulary slot, but the model's training data had been cleaned of it, so the model had almost never seen it. A slot in the table, nothing in experience. People have a grid too — as the topic on the grid the brain draws over sound described, the distinctions squeezed out of it become inaudible to you. But there is a clear, testable difference in one direction: a human grid can be reopened. An adult who cannot hear the difference between English "l" and "r" can raise their discrimination score with targeted training, and you can spell any word letter by letter on demand. A model's token vocabulary is fixed before it sees its first training example; after that it can only work around the cuts, and where it cannot, you get the errors that look absurdly stupid.