CS PAPERS DEEP-READ · PAPER 9
Bahdanau, Cho & Bengio · ICLR 2015
In late 2014, three researchers — Bahdanau, Cho, and Bengio — gave machine translation a new organ: attention. It lets the model, while translating each word, glance back at the most relevant words in the source sentence. The "attention" you hear about every day in ChatGPT and large-model news traces its bloodline straight back here — this paper is attention's birth certificate.
The previous paper, Seq2Seq, translated by "first understand, then re-say": read the whole English sentence, compress it onto one small sticky note, then write out the French from that note alone. Fine for short sentences; the moment sentences got long, quality fell off a cliff. No wonder — take a 60-word passage, allow yourself one sticky note, and try to reproduce every detail from it. Impossible. The strange part: human translators never work this way. Nobody memorizes a whole paragraph before writing; everyone reads as they translate.
This paper says: stop memorizing — let the model read as it translates too. While producing each word, the source text lies open on the desk, free to be consulted at any moment. The sticky note's single-plank bridge is torn down and replaced by the original text, always at hand.
Before writing each output word, the model takes its current thought — "where am I in the translation?" — and makes eye contact with every word of the source, one by one: each gets a relevance score. High scorers get a long look; low scorers just peripheral vision. Its gaze works like a spotlight — a bright spot plus dimmer surroundings — and in exactly those proportions it blends information from across the source into one custom reference, then writes. The best part: nobody taught it where to look. The model picked that up on its own, as a by-product of learning to translate — after training, its gaze lands on the right corresponding words, and where English and French swap word order, the gaze visibly criss-crosses back and forth.
The immediate win: long sentences stopped collapsing, and neural translation matched the old statistical systems that had ruled for twenty years, for the first time. The deeper win: it proved that "look things up as needed" beats "memorize once." Three years later, Google's Transformer paper declared "attention is all you need," threw away every other part and kept only this trick — and the large-model era began. One honest note: this version of attention still sat on the old word-by-word engine, and every output word required scanning the whole source again — slow and hard to parallelize. That bill was settled three years later, by the Transformer.
Don't compress the whole sentence onto one sticky note and recite from it — let the translator glance back at the source for every word it writes, allocating its gaze by relevance. That gaze is where the attention mechanism began, and it's what beats inside the heart of every large model today.
Want the three-step formula, the architecture diagram, and the BLEU numbers? → switch to the deep read
This paper adds a learnable soft alignment module — what we now call the attention mechanism — to the encoder–decoder neural translation model: for every word it generates, the decoder scores all source positions for relevance, turns the scores into weights via softmax, and takes a weighted sum to build a context vector custom-made for that word, tearing down the "squeeze the whole sentence into one fixed-length vector" bottleneck. Long-sentence translation stops degrading, English–French translation matches the statistical system Moses for the first time — and attention is born, growing into the Transformer three years later.
First author Dmitry Bahdanau was still a student interning in Yoshua Bengio's Montreal lab when he wrote this, together with Kyunghyun Cho and Bengio; posted September 2014, published at ICLR 2015. It directly follows the RNN Encoder–Decoder (Cho et al.) and Seq2Seq (Sutskever et al., Paper 8) from months earlier; downstream, Luong attention (2015) simplified it, Google's GNMT (2016) industrialized it, and three years later it grew into the Transformer (Paper 1).
Every 2014 neural translation model shared one soft spot: the encoder had to squeeze the entire source sentence — whether 5 words or 60 — into one fixed-length vector, and the decoder rebuilt the whole translation from that vector alone. The vector became a single-plank bridge: fixed capacity, so the longer the sentence, the more detail gets dropped. Cho et al. had measured it that same year: as sentences grow longer, BLEU plummets.
The authors' diagnosis: the problem wasn't network size but the design itself — demanding that arbitrarily long sentences fit losslessly into the same small container guarantees something gets sacrificed. Meanwhile, in old statistical translation, "alignment" (which output word corresponds to which source word) was a separate module with hand-crafted features, impossible to learn end-to-end alongside translation. Could a model learn, by itself, to look at the right place as it translates?
The paper's model is called RNNsearch. Instead of making the decoder clutch one vector the whole way through, it redoes three steps for every word it generates:
s₍ᵢ₋₁₎ ("where am I in the translation?") and compute a relevance score against each source position's representation hⱼ: eᵢⱼ = a(s₍ᵢ₋₁₎, hⱼ) — in plain terms, asking "how relevant is source word j to the word I'm about to write?" Here a is a small feed-forward neural network (later dubbed additive attention), and it learns for itself what "relevant" means.αᵢⱼ = softmax(eᵢⱼ) — turn the scores into weights summing to 1: the proportions in which the model's gaze is distributed over the source.cᵢ = Σⱼ αᵢⱼ·hⱼ — blend information from across the source in those proportions into a context vector cᵢ, handed to the decoder to generate word i.The burden thus shifts from "one vector must remember everything" to "fetch what you need at every step": writing the word for "bank," the gaze clusters near bank; writing the end of the sentence, the gaze moves there.
Why this design? One word: differentiable. Because softmax makes the gaze soft — every source word gets some weight rather than one being hard-selected — the whole module is smooth end to end, and the training signal can flow back through it. So "where to look" (alignment) and "how to translate" (translation) are learned jointly, end-to-end, in the same training loop — exactly what the title's jointly learning to align and translate means. No human alignment annotations needed: alignment falls out as a by-product of learning to translate well.
The hⱼ being looked up needed an upgrade too. If it came only from a left-to-right RNN, it would summarize just "everything before j." The paper uses a bidirectional RNN instead: one pass reads left to right, another right to left, and the two hidden states at position j are concatenated into that word's annotation hⱼ — centered on word j yet aware of both what precedes and what follows, so attention can pick up a single word with its context.
Two reasons. First, information no longer crosses a single-plank bridge: a 60-word sentence offers 60 annotations available at any moment — "memory capacity" grows naturally with sentence length, and the encoder is never forced to discard information. Second, the learning signal gets a shortcut: previously the correction signal had to travel from the end of the translation back through the entire RNN chain to reach the start of the source, decaying along the way (vanishing gradients); now attention lays a one-hop wire between the decoder and any source position — the same "build the signal a highway" intuition as ResNet's skip connections (Paper 2).
The benchmark is WMT'14 English→French. Control: RNNencdec (fixed-vector encoder–decoder); treatment: RNNsearch (with attention); each trained in two versions (sentences up to 30 / 50 words). BLEU on the full test set: RNNencdec-50 scores 17.82, RNNsearch-50 scores 26.75 — same data, same RNN backbone, one extra attention module, nearly 9 BLEU apart. The longer-trained RNNsearch-50★ reaches 28.45; on sentences without rare words it hits 36.15, edging past the reigning statistical system Moses (35.63 — and Moses had used a huge extra monolingual corpus). For the first time, neural translation stood level with the statistical approach that had ruled for twenty years, on a serious benchmark.
The sentence-length curve says even more: RNNencdec's BLEU slides steadily once sentences pass 20 words, while RNNsearch-50 barely drops even beyond 50 words (Fig 3). And plotting the attention weights αᵢⱼ as a heatmap shows the model learned English–French word alignment by itself — where word order flips (e.g., European Economic Area ↔ zone économique européenne), the gaze visibly criss-crosses — with nobody ever teaching it alignment.
It contributed "attention" — the most important architectural primitive of the following decade. Luong et al. (2015) simplified the scoring to a dot product; Xu et al. (2015) carried it into image captioning (visual attention over a picture); GNMT (2016) deployed attention-based translation inside Google Translate; and in 2017 the Transformer (Paper 1) took it to its logical conclusion — throw away the RNN entirely and keep only attention, swapping this paper's small scoring network for scaled dot-product softmax(QKᵀ/√d)V. The self-attention that every large model runs for every token it generates is, mechanically, the direct descendant of this paper's three steps: score → softmax → weighted sum. As a bonus, attention heatmaps gave neural networks their first readily openable window of interpretability.
n×m scorings — the direct ancestor of the Transformer's O(n²) attention cost.① The pain: the encoder–decoder squeezes the whole sentence into one fixed-length vector — fixed capacity, so the longer the sentence the more is lost, and long-sentence BLEU plummets.
② The core idea: for every word it generates, the decoder looks back at the source dynamically — score → softmax → weighted sum — into a context vector custom-made for that word.
③ Mechanism: eᵢⱼ = a(s₍ᵢ₋₁₎, hⱼ), α = softmax(e), cᵢ = Σ αᵢⱼ·hⱼ; the scorer is a small network trained along with everything else (additive attention).
④ "Soft" is the key: differentiable everywhere, so "where to look" and "how to translate" train jointly end-to-end, with no human alignment annotations — alignment falls out as a by-product.
⑤ The supporting piece: a bidirectional RNN encoder gives each source word a context-aware annotation hⱼ for attention to draw on.
⑥ Results: WMT'14 English–French, RNNsearch-50 at 26.75 vs the fixed-vector 17.82; long sentences barely degrade; 36.15 on the no-rare-word subset, past Moses at 35.63.
⑦ Impact: attention became a universal architectural primitive — via Luong and GNMT all the way to the Transformer and the self-attention in every large model today.
⑧ Limits: sequential RNN slowness; n×m scoring cost; soft alignment isn't true alignment and is debated as explanation; the UNK vocabulary problem was left for BPE.