乒乓球拍卖完了 (pīngpāngqiú pāimài wán le) — how did you just read that? It can be cut as 乒乓球拍 / 卖完了, "the ping-pong paddles are sold out", or as 乒乓球 / 拍卖 / 完了, "the ping-pong balls have been auctioned off". Seven characters, two places to cut, both readings perfectly ordinary Chinese, and two completely different pieces of news.
Sentences like that look like party tricks. What they expose is something that happens tens of thousands of times a day in front of your eyes without your noticing: when you read Chinese you are constantly cutting, and the cut is not printed on the page. You do it fast and you do it right, but if someone asked you to write down the rule you use, you could not. And the surprise is that this is not a Chinese problem. Languages with spaces have merely hidden the same question inside the spaces, hidden it so well that everyone assumes it isn't there.
Ambiguous sentences are actually rare, and context usually settles them. The hard cases are ordinary sentences.
In 1996 Sproat, Shih, Gale and Chang did something very plain: they took 100 Chinese sentences at random, 4372 characters in all, asked six native speakers (three from the mainland, three from Taiwan) to mark the word boundaries, and then compared every pair of answers. Average agreement between pairs was only about 76 percent.
Where does the disagreement come from? Not from comprehension, but from how far up the unit "word" reaches. Is 北京大学 (Běijīng Dàxué, "Peking University") one word or two? Does the aspect marker 了 (le) in 吃饭了 (chīfàn le, "has eaten") belong inside the word or not? There is no right answer here, only convention — and the consequences of unshared convention are directly visible. At the first international Chinese word segmentation bakeoff in 2003, the annotated corpora came from several institutions — Peking University, Academia Sinica, City University of Hong Kong and others — each with its own way of cutting, so systems could only be scored against each standard separately. Move the same system to a different standard and its score moves visibly. A number like "98 percent segmentation accuracy" is always stated relative to one convention.
What is easiest to overlook is that none of this is peculiar to Chinese. English has spaces, yet don't packs two words into one space-delimited block and New York spreads one word across two. Vietnamese is blunter: it writes spaces, but the spaces separate syllables rather than words — thuế thu nhập cá nhân is five syllables and two words, thuế thu nhập (income tax) plus cá nhân (individual). Thai writes no spaces between words at all, reserving them for clauses and sentences, so ข้าวเย็น (romanised khâao yen, literally "rice" plus "cool/evening") is either the single word for dinner or the phrase "rice that has gone cold".
So a space is a typographic convention, not a linguistic fact: it outsources the boundary question to whoever fixed the writing habits some centuries ago. Languages without spaces simply leave the question where it was, and whoever wants to process them has to answer it.
In 1994 Philip Gage published a very short piece in a magazine for C programmers, describing a crude way of compressing files, later known as byte pair encoding, or BPE. The rule fits in one sentence: find the pair of adjacent bytes that occurs most often in the file, replace it with a symbol not yet in use, write the substitution down, and repeat. High-frequency combinations get merged into longer and longer units, and the file shrinks.
Take a small pile of words. Suppose all you have is low, lower, newer, wider, split into single letters to begin with. Count the adjacent letter pairs: "e" followed by "r" occurs three times, more than anything else, so "er" is merged into one unit; count again, merge again. After a few thousand rounds, common words are in the table whole and rarer ones survive as fragments. The whole process uses one piece of information: what tends to sit next to what.
In 2016 Sennrich, Haddow and Birch moved this compression algorithm into machine translation, to cure the rare-word problem: with a fixed vocabulary, a word outside it left the old systems nothing to emit but "unknown", whereas once words are cut into fragments there is no outside.
One thing here is easy to slide past: what BPE produces is not morphemes (a morpheme being the smallest unit that still carries meaning, like un- or -ing). It has no notion that un- is a prefix; it knows only which pieces sit together often. Sometimes the pieces happen to line up with meaningful units, sometimes they do not at all, and an unusual proper name may come out as six fragments that individually denote nothing. The meaningful parts discussed in the topic on how words are assembled coincide with these fragments only occasionally.
And its sharpest, most misread move is this: it never answers the question "what is a word". The question that had six native speakers disagreeing in the section above is one BPE does not touch. It substitutes a question that needs no answer — which character strings habitually occur together. This is the same kind of victory as in the topic where hand-written grammars lost to counting: the winning move was not understanding language better, it was admitting you don't and going around.
Later models went further and cut from raw bytes, so that no script is ever outside the table. The cost is conspicuous for Chinese: UTF-8 encodes each character in three bytes, so an uncommon character becomes three tokens that individually denote nothing. The same string 乒乓球拍卖完了 (pīngpāngqiú pāimài wán le), "the ping-pong balls have been auctioned off", is three units by word, seven by character, twenty-one by byte. Which of those is the natural unit? None of them.
How the cutting is done is settled before training begins, and then frozen. Those frozen cuts grow into the model's abilities and its defects.
The easiest thing to measure is length. In 2023 Petrov and colleagues took one passage translated into many languages and counted the tokens under standard tokenizers: longest to shortest differed by as much as fifteen times. That converts directly into the per-token price, the latency of generation, and how much a context window of fixed size can hold. What matters is not to read this as a ranking of languages: a token vocabulary is fitted to a corpus, and whoever's text dominates that corpus gets the cheap encoding. It is a fact about corpora, not about languages.
Numbers are subtler. The GPT-3.5 and GPT-4 vocabulary holds entries for digit strings of one, two and three digits, and cuts them greedily from left to right: 12345 becomes 123 and 45. So the very same chunk "123" sits in the thousands place in one number and the units place in another — chunk boundaries follow the total length of the number rather than place value. In 2024 Singh and Strouse tried one thing: write the numbers with thousands separators, as 1,234,567. The comma is a hard boundary, so every chunk lines up with a fixed place value. With the commas in, both models' accuracy on addition rose measurably — same model, same problem, only the cutting changed.
Then the example that has been worn out: ask a model how many r's are in "strawberry". It cannot answer, not because it cannot count, but because what it sees is not ten letters but two or three whole blocks, with the letters sealed inside. This is not flatly impossible either — work in 2025 found that character-level ability does emerge during training, just late and abruptly. The two findings differ in evidential strength: the comma result is a controlled intervention, so the causal claim is firm, whereas "the letters are sealed in" is a plausible and well-supported explanation rather than the only one.
The strangest class is glitch tokens. The tokenizer and the model are trained in two separate steps, so some strings are frequent enough in the tokenizer's data to earn a slot in the table while the model's own training data has been cleaned of them. In 2023 people fed GPT the string "SolidGoldMagikarp" and its responses went entirely off the rails — a slot in the table, nothing in experience.
People have a grid of their own, of course. As the topic on the grid the brain draws over sound described, the distinctions squeezed out of that grid become inaudible to you. But there is a difference here that runs in a clear direction and can be tested: a human grid can be reopened. An adult who cannot hear the difference between English "l" and "r" can raise their discrimination score with targeted training, and you can spell any word out letter by letter whenever you like. A model's token vocabulary is fixed before it sees its first training example; after that it can only work around the cuts, and where it cannot, you get the errors that look absurdly stupid.
Agreement with one set of annotations, not agreement with a fact; move to another standard and the same system loses a good many points. This is not gaming a metric — the metric's definition is bound to a convention by construction. To compare across standards you have to switch to downstream tasks instead: does retrieval find the right thing, do the pauses in speech synthesis fall in the right places.
It does; it has only moved. A model can decline to answer "what is a word"; someone building a product cannot. Search indexing needs word boundaries, the pauses in speech synthesis need them, and so do braille transcription, spacing in children's books and dictionary-making. The question the model walked around has been handed to the people downstream.
No. The token vocabulary is fitted, and whoever's text dominates the fitting corpus gets the shorter encoding; give the same language a tokenizer trained for it and the token count drops noticeably. Reading "expensive under this tokenizer" as "this language is inefficient" mistakes the composition of a corpus for a property of a language.