In 2014 the best neural approach to translation was read-then-retell: one network read the English sentence all the way through and squeezed it into a fixed-length string of numbers, and a second network wrote the French out from that string. Translation quality was scored with BLEU, an automatic metric for how much a machine translation overlaps a human reference — higher is better. Past about twenty words, BLEU fell off a cliff, and it kept falling no matter how big you made the network.
The consensus read was that the networks were not yet large enough. Dmitry Bahdanau disagreed. He was an intern in the lab of the deep learning researcher Yoshua Bengio, and his read was that the flaw was the phrase "fixed length" itself. Sixty words of text, allowed one small note of a fixed size, then reconstructed in full detail from that note. Of course it fails.
His way out came from thinking about how people do the job. No human translator memorizes the paragraph before picking up a pen; you translate a stretch, you look at that stretch, and the source sits open on the desk the whole time. So he let the model work that way: before writing each output word, sweep back across every word of the source, score each one for relevance, draw more from the high scorers and less from the low ones, and blend a context tailored to that one word. He called the module attention.
Same data, same backbone, plus the module: BLEU went from 17.82 to 26.75, nearly nine points, and the length curve flattened — past fifty words it barely degraded at all. Three years later someone threw away every other component and kept only this one, in a paper titled Attention Is All You Need.
Meanwhile, at the other end of this hub, the neuroscience page on attention spends a whole subsection on the collision of names. The two are the same thing only at the most abstract level, it says: both weight information by relevance. Below that the mechanisms have nothing in common. The brain has no table of pairwise scores.
That is the strange part. An analogy that does not survive contact with the mechanism solved a specific problem nobody else could solve. So what passed?
Start with the side that got borrowed from.
The popular metaphor is a spotlight: attention is a torch you hold, and whatever you point it at gets brighter. Mechanistically that is backwards. The picture the neuroscience site gives is biased competition — neurons coding different things are already fighting each other over the small amount of downstream processing available, and what attention does is not illuminate one of them but quietly weight the scales so that one is likelier to win a contest that was running anyway.
Two forces do the weighting. One you deploy: hunting for your red car in a parking lot, every red thing in the lot leaps out. One barges in: motion at the edge of vision, a bang behind you, a red dot in the corner of a screen — your head has turned before you decided anything. They fight, and the barging one usually wins. That fact gets used twice later.
Why must anything be selected at all? Because downstream is a bottleneck; pitifully little of what arrives can actually be processed deeply and delivered into awareness. The famous demonstration is the pass-counting video: absorbed in counting, roughly half of viewers fail to see a person in a gorilla suit stroll through the center of the frame and thump their chest. The light landed cleanly on their retinas. It simply lost the competition. This is inattentional blindness.
The gate has an address, too. In the middle of the brain sits the thalamus, the relay through which nearly every sense except smell must change trains before reaching the cortex. Wrapped around it is a thin shell of inhibitory neurons called the thalamic reticular nucleus, which sends nothing to cortex at all and only decides which channels pass and which get cut. Francis Crick, the Nobel laureate, called it the brain's searchlight gate. Turn the same gate finely and you get attention; close it broadly and you get sleep.
Three words for the biological side: competition, bottleneck, cut. Hold on to the third — the sharpest test in this essay rests on it.
Now the side that borrowed. One design choice in Bahdanau's three steps carries the whole essay: it is soft. After scoring, a softmax turns the scores into a set of proportions summing to one, so every source word gets a little weight rather than one being hard-selected. Soft is differentiable everywhere, so the error signal flows back through it, and "where to look" and "how to translate" can be learned in the same loop.
The 2017 paper pushed that to its conclusion: if attention is what does the work, discard the rest. The result is the Transformer, the skeleton of nearly every large model today. Each token — a word or word-fragment as the model sees it — produces a query, a key and a value; the token's query is scored by dot product against every key in the sentence, softmax turns those into weights, and the values are summed in that proportion.
Attention(Q, K, V) = softmax(QKᵀ / √d) · VThat unremarkable √d cannot be dropped: at high dimension the dot products grow large and push the softmax into a near-all-or-nothing saturation zone where the gradient is roughly zero and nothing trains. What it bought was one matrix operation between any two words and a whole sentence computed in parallel — 28.4 BLEU on English to German, a new best at the time, trained on eight GPUs in three and a half days.
The cost is plain: every pair of positions has to be scored, so compute and memory grow with the square of the length. That bill has concrete numbers. Generating each word means re-reading the keys and values already computed out of GPU memory; for a seventy-billion-parameter model over a sequence of some eight thousand tokens, the most literal multi-head arrangement burns roughly 21.5 GB on that cache, and letting several heads share one copy brings it to about 2.7 GB.
So the last few years of engineering all do the same thing: stop it from attending to everything. The neuroscience page puts the punchline bluntly — pushed by the cost of compute, artificial systems rediscovered what biological brains were forced into by the cost of energy. Attending to everything is too expensive. You must select.
The word's third stop is where it acquired a price. The economist Herbert Simon took that step in 1971, in a passage the leadership site quotes:
What information consumes is rather obvious: it consumes the attention of its recipients. Hence a wealth of information creates a poverty of attention and a need to allocate that attention efficiently.
— Herbert A. Simon, Designing Organizations for an Information-Rich World (1971), pp. 40–41
That sentence converts attention from a mental state into a stock. A stock can be counted, and what can be counted can be mined.
The psychology site is specific about the mining. Platforms optimize not for your happiness but for your next click, and the strongest lever they pull is what the psychologist B. F. Skinner called variable-ratio reinforcement: when the reward is unpredictable, behavior runs at the highest rate and is hardest to extinguish. A fixed ratio — a reward every N times — is predictable, comes with a natural pause afterwards, and lets people stop. "The next one might be good" has no pause. Infinite scroll, pull-to-refresh and push notifications are all built on the second shape.
There is also an answer to why it stays compulsive without being pleasant: the neuroscientist Kent Berridge separated wanting from liking into two dissociable systems. And Shoshana Zuboff, in The Age of Surveillance Capitalism, lays the whole production line out: human experience becomes behavioral data, machines turn that into prediction products, and the products are sold to people betting on your next move. You are not even the product. The prediction is.
Notice what this stop took and what it left. It took scarcity — a quantity with a hard ceiling, countable in person-seconds. It took no selection mechanism whatsoever; there is no thalamus here and no softmax. But it aimed precisely at the barging force from the previous section. A red dot and a buzz trigger wanting directly, with no decision in between.
One more conclusion has to travel with this stop, because it overturns the explanation people reach for first. The source page is explicit: blaming compulsive use on weak self-control is wrong, since variable-ratio reinforcement is engineered to resist extinction — this is an environment-design problem. Nor does "ego depletion" explain it; that model has failed to replicate across laboratories.
The fourth stop sits somewhere else entirely. In 1942 the French philosopher Simone Weil wrote a line to a poet that has been quoted ever since: attention is the rarest and purest form of generosity. She put it harder elsewhere — absolutely unmixed attention is prayer.
She is not describing bandwidth. She is making a value judgment, and she knows she is. Her essay on school studies makes the claim concrete: a geometry problem you never solve has its value not in the answer but in the stretch of unrewarded attention you held for it, and that capacity transfers to any object, including a suffering person.
The psychologist Mihaly Csikszentmihalyi arrives nearby by a different road. In Flow he calls attention psychic energy — nothing happens without it, and things happen when it is spent — and from that draws a heavy conclusion: the self is nothing other than the sum of what your attention has been given to over the years.
This stop should not be read as a softened version of the first three. It is an honest borrowing: it takes the scarcity, then states plainly what it intends to do with it. It does not pretend to be a measurement, so it need not be tested as one.
Its own two soft spots travel with it. Flow is value-neutral, as Csikszentmihalyi himself concedes: a burglar picking a lock and a gambler placing a bet can both be in flow. The methodology is soft too — experience sampling rests entirely on subjects reporting how they feel right now, which is hard to falsify and hard to line up against any neural mechanism. As for Weil, the critic Susan Sontag put it directly in 1963: one reads an author like her for her seriousness, not because one actually agrees with her. She treated self-consumption as moral proof, refused in 1943 to eat more than the ration of occupied France, and stopped producing anything at thirty-four. The method is worth learning. The life plan is not.
Put the four stops side by side and go back to the opening question.
What they share is not a mechanism. There is no correspondence of parts between the thalamic reticular nucleus and a QK dot product, and still less between an ad auction and Weil's prayer. What they share is the shape of a constraint, assembled from three pieces, none of which can be missing:
Check them off, stop by stop. The brain: the ceiling is downstream processing bandwidth, forced by an energy budget; the candidate set is every sensory signal pouring in; allocation runs on biased competition plus the thalamic gate. The Transformer: the ceiling is compute and memory growing with the square of length, and 21.5 GB is what that looks like concretely; the candidate set is every token in context; allocation is a dot product through a softmax. The market: the ceiling is the waking hours in a person's day, counted in person-seconds; the candidate set is a nearly unlimited supply of content; allocation runs on bids, and ad auctions price attention daily. Weil and flow: the ceiling is the total attention a person can spend in a lifetime; the candidate set is everything worth attending to; and what allocation decides is not throughput but who you become.
Why does the shape cross when the mechanism cannot? Because a constraint does not ask what the constrained thing is made of. Sensory signals, tokens, ad slots and a suffering person share no material basis, but "the bridge is narrower than what wants across" applies to all four without modification.
Elegance does not count. The test of whether a cross-disciplinary connection is worth anything is whether it has solved a specific problem that was otherwise stuck. On this line, that point is Bahdanau's step.
Note what he actually borrowed. Not "how the brain does attention" — that was not available to borrow, and nobody could have supplied it. He borrowed the answer to "what does a person do when the channel won't hold it all": don't memorize, spread it out, look at what you're using as you use it. That is an answer at the level of the constraint.
And it held. The problem was stuck on the single-file bridge of a fixed-length vector, and the consensus response — build bigger networks — was pushing in the wrong direction. With look-back-on-demand, BLEU went from 17.82 to 26.75; on the subset without rare words it reached 36.15, passing the 35.63 of Moses, the statistical system that had ruled machine translation for twenty years. And nobody ever taught it to align. Alignment came out as a by-product of learning to translate.
The reverse case is more telling still. What the neuroscience page records is not a borrowing but a collision: the Transformer's cost explodes with the square of the sequence, expensive enough that you are forced to pick out a small subset — and going the long way around, machines driven by the cost of compute arrived at the conclusion brains had been driven to by the cost of energy. Neither side copied the other. One constraint forced the same solution twice, independently. That says more about the reality of the shape than any analogy could.
So the answer is that mechanisms never cross. Constraints do. The analogy being wrong about the mechanism is no obstacle at all, because the mechanism was never what carried the load.
With the shape identified, here is what each site was able to collect.
There is a cheaper win as well. In 2009 the organizational behavior researcher Sophie Leroy had subjects switch to evaluating résumés while a word puzzle was still unfinished; their judgments of the résumés were significantly worse than those of a control group that finished the puzzle first. The unfinished task keeps occupying working-memory bandwidth. She called it attention residue.
The remedy it suggests is counterintuitive: how you end a task matters more for output quality than how you begin the next one. Spend two minutes before switching writing down progress, open questions and the next concrete action; once the brain has the signal that the task is safely stored, it stops revisiting. The cost of an interruption was never those five minutes. It is that for the next half hour you are only seventy percent there.
Precisely because what you collected is real, you need to know where it stops travelling. These five questions are not for catching people out. They are for deciding whether a claim about attention can carry a conclusion.
The sharpest of the five, and the easiest to skip past.
On the biological side the gating is hard. The thalamic reticular nucleus genuinely cuts channels, and information that loses the competition never enters awareness at all. The gorilla walks through the center of the frame and half the room does not see it — not dimly remembered, never arrived. On the Transformer side the weighting is soft. Softmax gives every token in context some weight; nothing is discarded, it is only turned down.
That difference decides whether a conclusion from one side can move to the other. "Low attention weight means the model didn't see it" is false: it saw it, the weight was small, and residual weight accumulates across dozens of layers. Conversely, "try hard enough and you will notice" is false too; that gate is not yours to operate.
But the question to ask is which kind this system is using right now, not "machines are soft and people are hard." The moment you switch on a sliding window or sparse attention, the machine really does have hard gating: tokens outside the window take no part in the computation. Ask about the system, not the species.
Some tiers really do report numbers. The machine tier is hardest: gigabytes of cache, the share of compute spent on attention, all calculable to the decimal. The market tier reports too: person-seconds, daily active minutes, the number of times a day you unlock a phone without being able to say why — that last one reflects cue-driven use better than screen time does. But "deep work capacity" and "a three-slot attention budget" report no scale at all. Three slots is a useful convention, not a measured capacity.
What can be counted can be put in a controlled trial, and therefore can be overturned. When Allcott and colleagues had nearly three thousand people randomly deactivate a social platform for four weeks in 2020, subjective well-being did rise — by a small fraction of what psychotherapy achieves. The numbers are useful precisely because they can contradict you. Claims that report no number can inform a decision; they should not be offered as evidence.
It is not one thing, on either side.
Biologically there are at least three semi-independent networks — the psychologist Michael Posner's classic division into alerting, orienting and executive control — and they can be damaged separately and trained separately. So "my attention is bad" is a mush of a sentence. Easily hijacked? Slow to return? Unable to sustain? Only once it is taken apart is there anything to fix.
The parenting site puts this to the hardest use. On the modern understanding, ADHD is not an absence of attention but a developmental difficulty in regulating attention and inhibiting behavior; the psychologist Russell Barkley locates the core in impaired response inhibition. The classic misjudgment runs exactly against this test: "he can concentrate on video games, so he's faking." Intense interest-driven focus is itself one of the features of ADHD. The same site notes that the prefrontal executive network keeps developing into a person's twenties, so holding a seven-year-old to an adult standard produces conflict, not concentration.
The machine side is no more singular: different heads learn different kinds of relation — some track adjacent words, some link a pronoun back to its noun — and after training many heads turn out redundant and can be pruned with almost no loss.
Weil's line about the rarest and purest form of generosity is not a description of bandwidth. It is a value judgment, stated as one, never pretending to report a number. What to guard against is the inverse: "focus is the brain's highest function, therefore you should…" — the missing step there is a value premise, not data, and borrowed neuroscience vocabulary is exactly what covers the gap.
The same ruler has to be turned on this tier itself. "It produces flow" is nowhere near "it is worth doing," and flow theory cannot supply that step. Weil's failure is a different one: her intellectual weapon was suspicion of every collective power, and that weapon lost its calibration the moment it was pointed at her own origins. The sharpest framework crashes where it runs most smoothly.
This one grew on the machine side, and it is the only test here that remains formally contested.
Attention heatmaps are genuinely appealing, and they gave neural networks their first easily opened window. But both paper pages carry the same warning: the weights often diverge and do not always land on the "correct" word, so treating them directly as the model's explanation is risky. This is where the later "attention is not explanation" dispute began. As a general test: a module named "attention" is no guarantee that where its weights are high is where it is actually looking.
There is an exact human counterpart, sitting in Leroy's experiment: subjects were subjectively certain they had fully shifted to the new task while their cognitive resources were still locked on the old one. A person reporting that they are concentrating is likewise no guarantee that their resources are there. Both sides need external evidence. Self-report will not do.
Taking something apart obliges you to say how to use it, or the essay is just more rhetoric. Roughly three tiers.
When all three pieces can be named and the number reported, treat it as measurement.Given a candidate set, a ceiling with a unit and a rule for ordering, attention is a quantity you can compute: gigabytes of cache, the share of compute attention consumes, daily active minutes, the effect size from a randomized trial. Claims in this tier can be argued right or wrong, and should be.
When the three pieces are clear but no scale is available, treat it as a model.The test is whether it yields a prediction that could be overturned. Attention residue is the exemplar: it predicts that encapsulating a task before switching reduces the residue, and an experiment could refute that. A quick self-check for this tier: the claim can usually be rewritten in sentences containing no word for attention without losing content. If it survives the translation, there is real causal reasoning underneath.
When neither holds, it is rhetoric — and rhetoric is not an accusation.Look at what the word is doing in the sentence. If it explains — why this happens, what comes next, where to intervene — it is working. If it only raises the price of something, it is rhetoric. Weil's line is the finest thing quoted in this essay and it is of course rhetoric. What goes wrong is never the rhetoric itself. It is filing it under "and therefore science has shown."
Back to Montreal, finally. The metaphor Bahdanau borrowed was wrong about the mechanism; the neuroscience site is right that the two are separate things. His invention was right anyway, because the mechanism was never what he borrowed. He borrowed the shape of a constraint.
What genuinely crosses a disciplinary border tends to sit a layer more abstract than you expect. The mechanism failed the crossing. The constraint passed. Those nine BLEU are the receipt.