What surfaced in your head after those four sentences was probably not a definition but a position: it stands beside wine, beer and rice wine, a little further from water and milk, and very far indeed from cars. (Tezgüino is real — a corn-based fermented drink among some Indigenous communities in northwestern Mexico.)
The interesting part is not that you guessed right. It is that you guessed right holding nothing: no picture, no taste, nobody pointing. All you had were the other words around it — and that was enough. The idea has a name, the distributional hypothesis. It is half a century older than deep learning, every text-processing system in use today is built on it, and it carries a blind spot that has never been removed.
Unpack those four sentences and the cues are all there: "a bottle of" makes it a liquid with a volume; "on the table" places it at meals; "everybody likes" makes it a consumable rather than a tool; "makes you drunk" alone pushes it into the alcoholic class; "out of corn" supplies the raw material. Together they fence off a small region — and beer and rice wine occupy that same region. What you know is who it stands next to.
In 1954 the linguist Zellig Harris turned this into a claim: how far apart two words are in meaning can be read off how far apart their environments are. Three years later J. R. Firth left the endlessly quoted line — you shall know a word by the company it keeps. Both say the same thing: stop asking what the word points at in the world, and look at its neighbours. At the time this was only a fieldwork trick, but it had a side effect that took sixty years to matter: neighbours can be counted, and pointing-at-the-world cannot.
Now the blind spot. Complete this: "the coffee is ___." Hot and cold both fit. "This coat is very ___": expensive and inexpensive both fit. "He drove ___ for two hours": east, west, north, south, take your pick. Words with opposite meanings turn out to be the most freely interchangeable of all, because they demand almost identical neighbours.
So what the distributional hypothesis measures has to be stated precisely: it measures whether two words can be swapped into the same slot, not whether they mean the same thing. Near-synonyms are interchangeable; so are antonyms; the method files them together. This is not a bug in some program but a consequence of the idea itself — and there is a whole line of research devoted to injecting "these two are opposites" back into the result afterwards, which pushes the problem down only a little.
How do you turn neighbours into something a machine can compute? The crudest way is also the earliest: build a huge table, one row and one column per word, each cell holding how often the two occur together within a short span of text. Tezgüino's row carries counts under "bottle", "drunk" and "corn" and a zero under "screwdriver"; wine's row looks much the same. Similarity is how alike two rows are.
The trouble is that the table is both enormous and almost entirely empty: most words are rare, most word pairs never co-occur at all, and under one percent of a fifty-thousand-square table carries anything.
So it gets squeezed. In 1990 a group working on document retrieval proposed latent semantic analysis: compress that vast empty table into a few hundred numbers per word. The squeeze pulls together rows that keep the same company, so "car" and "automobile" land near each other even if they never shared a sentence. In 1997 Thomas Landauer and Susan Dumais put it through an exam: trained on some thirty thousand encyclopedia articles, about 4.6 million words, it took the synonym section of the TOEFL, eighty four-way multiple-choice items. It scored 64.4%, while applicants from non-English-speaking countries averaged around 65% on the same items. Read that number carefully: it shows that counting co-occurrence alone can reach a population's average on one particular item type. It does not show that the program knows English — synonym choice is the home ground of a distributional method.
In 2013 the approach changed from count-then-compress to guess-and-adjust. Hide a word in a sentence, have the program guess it from its left and right neighbours, and when it guesses wrong, nudge the string of numbers attached to each word. After a few billion guesses, that string is the word's position. This is word2vec, and it became famous for something that looks like a conjuring trick: take the numbers for "king", subtract "man", add "woman", and the nearest landing point is "queen".
The trick is overrated. In 2016 Tal Linzen pointed out something plain: the standard procedure first removes the three input words from the candidate answers, because otherwise the nearest position is usually "king" itself. Much of the success therefore comes from "queen" already being close to those three, not from a route from male to female stored inside the vectors. The phenomenon is real; the explanation was inflated.
The same year brought a sturdier result: in spaces trained separately on English and Spanish, number words have a similar internal arrangement, and so do animal words, so a single transformation lines the two up well enough to guess dictionary entries that were missing. The alignment problem from the machine-translation story reappears here wearing a different face — there it was counted sentence pairs, here it is the shape of two spaces.
In 2017 Aylin Caliskan and colleagues took the standard psychological test for implicit association and ran it, unchanged, on word vectors trained from web text, publishing the result in Science. The findings lined up one by one: flower words sit closer to pleasant words and insect words closer to unpleasant ones, a much-replicated human result; and how close an occupation word sits to female terms correlates positively with the actual share of women in that occupation.
The right reading is not "the vectors are biased" but that co-occurrence is all they have, so they cannot separate two things. In "nurse occurs near female terms", half is a fact about the labour force and half is an inertia in how people write, and counting treats the two alike. Separating them needs exactly what this method lacks by design: a world outside the word list.
There is a more basic limit too. One word gets one string of numbers, and that string can only be the average of all its uses. Mandarin 打 (dǎ) is a hard case: 打电话 (dǎ diànhuà, "make a phone call"), 打水 (dǎ shuǐ, "fetch water"), 打毛衣 (dǎ máoyī, "knit a sweater"), 打车 (dǎ chē, "hail a taxi"). The neighbours fall into several clusters, and one averaged position resembles none of them. English "bank" has the same problem, the riverside and the institution sharing a spot. Later models give every occurrence its own position, so the same 打 (dǎ) lands in different places in different sentences.
And people? People use distribution too — as the statistical-learning story described, infants who cannot yet speak are already tracking which syllables keep each other's company. The difference lies elsewhere, and it is concrete enough to compare. In 1978 Susan Carey and Elsa Bartlett told three- and four-year-olds "bring me the chromium tray, not the blue one, the chromium one"; the children brought the olive one, and a week later still linked the word to that colour. This is fast mapping, and it does not run on counts — it runs on the child seeing two trays and ruling out the one that already has a name. The difference is therefore testable: a word usually needs dozens to hundreds of occurrences before it has a decent position in a co-occurrence space, and a child needs one; conversely, no person will ever read the millions of documents a count can cover. (On replication: a single exposure yields partial knowledge, and how long it lasts is still disputed.)
In 2020 Emily Bender and Alexander Koller drew the line harder with a fable. An octopus taps an undersea cable and eavesdrops on two people ashore; in time it can hold up its end of the conversation seamlessly, having never seen anything on land. Then one day it is asked how to fend off a bear with a few sticks, and it has nothing. Their claim: meaning cannot be learned from form alone. The counter is not weak either — most speakers of "inflation" or "justice" have likewise never seen what those words point at, and use them perfectly well. The argument is unsettled. What is certain is narrower: the distributional hypothesis gives a word's position inside language, and says nothing about where that position is pinned to the world.
It is, and that is what critics point at most often: the system is internally consistent with no point pinned to the world. One reply is that circularity is not uselessness — a map with only relative positions still gives directions, provided you are standing somewhere on it. The other reply is more modest and more honest: co-occurrence statistics never claimed to deliver meaning. They deliver similarity, and similarity was always relative.
Scarce material does mean poor vectors, but be clear about what kind of problem that is. The amount of text reflects whether publishing, administration or religion created a demand for writing in that language; it has nothing to do with its structure or expressive power. Reading "the model does badly on this language" as "this language is deficient" mistakes institutional history for linguistic fact.
This is one of the strongest lines of objection. Someone who has never seen colour still uses colour words correctly and can discuss red being darker than pink, and nobody says they do not understand those words. The other side has an answer too: that person still lives among objects, temperatures, weights and other people's reactions, so language is not their only input. The question is therefore not whether eyes are required, but how much besides words is required — and nobody has produced that quantity.