Sequence to Sequence Learning (Seq2Seq) — CS Papers Deep-Read Paper 8

What did this paper do?

In 2014, three researchers at Google — one of them Ilya Sutskever, later OpenAI's chief scientist — proposed Seq2Seq (sequence to sequence): two neural networks working as a relay team, one that "reads" and one that "speaks." For the first time, machine translation of a whole sentence was done end to end by neural networks alone — and it beat translation systems that had been polished for years. The "you say something, it says something back" shape of ChatGPT today traces straight back to this design.

An analogy

Think of how a simultaneous interpreter works: listen to the whole sentence, form a "meaning" in your head, then say that meaning out loud in the other language — never word-by-word substitution. Machine translation before Seq2Seq was much more like the latter: chop the sentence into pieces, look each piece up in a phrase table, then try to glue the fragments into something fluent. Seq2Seq taught the machine the interpreter's way: first understand, then re-say.

What was new

The old world's translation systems were assembly lines: one component chopped phrases, one looked up tables, one reordered words, one scored fluency… dozens of parts, each hand-designed and tuned by experts, and when something went wrong it was hard to trace. Seq2Seq replaced all of it with one whole: feed it a huge pile of "source sentence – translation" pairs and it learns the entire translation process by itself, no hand-written rules anywhere in the middle.

How it works

The "reading" network takes in the source sentence one word at a time, updating its "mental state" as it goes; when the last word is read, that state is the whole sentence condensed into a single "thought." The "speaking" network takes over from that thought and says the translation one word at a time: for each new word it looks at the thought plus what it has already said, and it keeps going until it emits a special "full stop" marker — so the output can be as long or short as it needs to be, nothing fixed in advance.

The paper also had a famously clever trick: feed the source sentence in backwards. That way the beginning of the source ends up right next to the beginning of the translation, so the model first learns the easy "match the openings" step and then untangles the rest from there — like moving two relay runners closer together for a smoother handoff. Scores jumped noticeably.

What it brought

It proved for the first time that pure neural networks — no linguistic rules, no hand-built components — could beat traditional translation systems on official benchmark data. Two years later, Google Translate replaced its decade-old system with a descendant of this architecture; and "first understand, then generate" spread from translation to summarization, dialogue, speech, and image captioning, becoming AI's general recipe. One honest caveat: squeezing a whole sentence into one fixed-size "thought" means the longer the sentence, the more gets forgotten — the "attention" mechanism invented the very next year to fix this eventually grew into the Transformer.

Remember one thing

Two networks in relay: one reads the whole sentence into a single "thought," the other re-says it word by word from that thought — translation went from "assembly-line kit-bashing" to "first understand, then re-say," and today's ask-and-answer large language models grew out of exactly this shape.

Want the encoder–decoder diagram, the reversal trick, and the real scores? → switch to the deep read

In one sentence

Seq2Seq builds an encoder–decoder out of two multi-layer LSTMs: the encoder reads an input sequence of any length into one fixed-length vector, and the decoder generates an output sequence of any length from that vector, word by word. With the reversed-input trick, it became the first end-to-end neural system to beat a strong statistical machine translation baseline on WMT'14 English–French (BLEU 34.8 vs 33.3), establishing the "everything is sequence-to-sequence" paradigm.

Glossary

Neural network: a model built from stacked "layers" that learns patterns from large amounts of data; training means gradually adjusting its internal parameters to make predictions more accurate.
Recurrent neural network (RNN): a neural network built for sequences — it reads elements one at a time while maintaining a continuously updated "hidden state" as memory.
LSTM (Long Short-Term Memory): an improved RNN (see Paper 7) that uses "gates" to control what gets written into and erased from memory, letting it remember information from many steps back. Both networks in this paper are LSTMs.
Hidden state: the internal vector an RNN carries and updates at every step — its "memory of everything read so far."
Word embedding: representing each word as a list of numbers (a vector) so that similar words get similar vectors — the entry point for neural networks to handle text (see Paper 4).
Statistical machine translation (SMT): the dominant approach before Seq2Seq — build a "phrase table" from statistics over parallel text, then translate by chopping into phrases, looking them up, reordering, and scoring candidate outputs for fluency; a pipeline of many separate modules.
BLEU: the standard automatic score for machine translation (0–100), measuring overlap between machine output and human reference translations; higher is better, and even 1 point is a visible gap.
Beam search: when generating word by word, instead of committing to the single best word at each step, keep B candidate sentences alive and extend them all, then pick the best-scoring one overall — steadier than greedy "one path only."

Coordinates

The authors are Ilya Sutskever, Oriol Vinyals, and Quoc V. Le at Google; the paper appeared at NIPS 2014. It builds on the LSTM (1997, Paper 7) and on earlier encoder–decoder ideas from Kalchbrenner & Blunsom (2013) and Cho et al. (2014); it feeds directly into Bahdanau attention (2014) and the Transformer (2017) — the lineage of modern large language models starts taking shape right here.

Problem and motivation

By 2014, deep networks were thriving in speech and vision, but they had a fatal constraint: inputs and outputs had to be vectors of fixed, pre-specified dimensionality. Yet many of the most valuable problems — translation, speech recognition, question answering — are "sequence in, sequence out," with both lengths unknown in advance and no obvious alignment between them.

Translation at the time was ruled by SMT: a sprawling pipeline of word alignment, phrase tables, reordering models, and language models, trained separately and glued together with hand-tuned weights. It worked, but every module needed expert care, errors compounded down the pipeline, and the training objective was fragmented. The real question: could a single neural network learn "the probability of the output sequence given the input sequence" end to end, replacing the whole pipeline with one trainable system? What was missing was a general architecture for variable-length in, variable-length out.

Core ideas / method

Idea 1: Encoder–decoder — split "reading" and "speaking" into two networks

Seq2Seq uses two separate LSTMs. The encoder reads the source sentence word by word, updating its hidden state at each step; after the final word (marked by an end-of-sequence symbol <EOS>), that hidden state is the sentence's fixed-length vector v — informally the "thought vector," the whole sentence's meaning compressed into one vector.

The decoder is essentially a language model "set in key" by v: with v as its initial memory, it generates the translation word by word — as a formula, p(y₁…yₜ | x) = ∏ p(yₜ | v, y₁…yₜ₋₁), which in plain words says "each new word is chosen based on the thought v of the whole source, plus everything I've already said." At each step it outputs a probability distribution over the vocabulary; the chosen word becomes the next step's input, and generation stops when <EOS> is produced — the model decides its own output length, which is exactly how variable-length output is solved.

Why two networks? First, input and output are usually different languages, so each side gets its own parameters for its own job. Second, the interface is beautifully clean — the two sides exchange only the single vector v, so each side can be made deeper and wider independently. The paper uses 4-layer deep LSTMs on both sides (1000 units per layer, 1000-dimensional word embeddings, a 160k input vocabulary and an 80k output vocabulary), and the deep version clearly beats shallow ones.

Fig. 1 · Encoder–decoder: the encoder reads the input "A B C" into a fixed-length vector v; the decoder starts from v as its memory and generates "W X Y" word by word, feeding each output back as the next input, until it emits <EOS>.

Idea 2: Reversed input — an absurdly cheap big win

The paper's most famous trick: feed the source sentence backwards (train "A B C → W X Y" as "C B A → W X Y", target untouched). Why does it help? Fed forwards, the first words of the source aren't used until the whole sentence has been read and most of the translation generated — too many steps away. Reversed, the start of the source sits right next to the start of the translation, conjuring up a batch of "short-range dependencies" out of thin air. The optimizer first learns the easy "match the openings" alignment, and once that's anchored, the long-range correspondences fall into place — even though the average distance between corresponding words hasn't changed at all.

The effect was striking: perplexity (a measure of how "uncertain" the model is about the next word — lower is better) dropped from 5.8 to 4.7, and BLEU jumped from 25.9 to 30.6. Honestly said: this is a purely empirical trick, and the authors had no complete theoretical explanation for it at the time.

Fig. 2 · Reversed input: after reversing the source, the first word A sits 2 steps from its translation W instead of 4; BLEU rose from 25.9 to 30.6 because of this alone.

Idea 3: Training and decoding, the engineering

The training objective is plain: maximize the probability of the correct translation on a huge set of sentence pairs (word-level cross-entropy). Decoding uses beam search: keep the B most promising partial translations at each step and extend them all. Interestingly, B=2 already gets close to the best results — the model's own probability estimates are that good. To keep gradients from occasionally blowing up during training (exploding gradients), they are rescaled whenever their overall size crosses a threshold (gradient clipping). The whole system trained for about 10 days on 8 GPUs (4 running the LSTM layers, 4 computing the vocabulary probabilities).

Key results

The benchmark is WMT'14 English→French translation, trained on a 12-million-sentence-pair subset. Representative numbers:

An ensemble of 5 deep LSTMs with beam search: BLEU 34.8, beating a strong SMT baseline's 33.3 on the same data — the first time a pure neural system surpassed a statistical one on a large-scale translation task.
Using the model to rescore the SMT baseline's 1000-best candidate lists reached BLEU 36.5, close to the best published result on the task at the time (37.0).
The reversal trick alone: BLEU 25.9 → 30.6, perplexity 5.8 → 4.7.
Against everyone's expectations, it also translated long sentences well — the fixed vector was expected to choke on them, but reversal relieved much of the pressure.
Projecting the sentence vectors v to 2D (PCA): sentences with similar meanings cluster together, sensitive to word order but insensitive to active vs passive voice — evidence that v really encodes meaning, not surface form.

Why it matters / impact

It established the paradigm that ruled the following decade: "everything is sequence to sequence." Translation, summarization, dialogue, speech recognition, image captioning (a CNN as the encoder, an LSTM as the decoder), code generation — all fit the same frame. In 2016 Google Translate launched GNMT (the industrial version of this architecture plus attention), replacing its decade-old SMT pipeline overnight — the first time neural networks took over the core of a major internet product wholesale. Deeper still: today's large language models — "prompt in, answer out" — are the direct descendants of this conditional word-by-word generation framework, with the encoder and decoder merged into a single Transformer. The three authors all became central figures of the wave: Sutskever later served as OpenAI's chief scientist, and Vinyals leads deep-learning research at DeepMind.

Limitations and criticism

The fixed vector is a bottleneck: no matter how long the sentence, it all gets squeezed into the same vector v, so long sentences inevitably lose information. The next year, Bahdanau et al. proposed the attention mechanism — letting the decoder look back at the entire input sequence for every word it generates — which solved this head-on and made the reversal trick unnecessary; three years later that line grew into the Transformer.
Reversal is a hack, not a solution: it works for language pairs with similar word order, like English–French — which is precisely evidence that the model never truly learned long-range dependencies; the problem was just moved somewhere easier.
Fixed vocabulary, helpless on rare words: out-of-vocabulary words become UNK (an "unknown word" placeholder), so names and numbers were often mistranslated; subword segmentation (BPE) later fixed this.
Slow and hard to parallelize: an RNN must compute step by step, so training can't parallelize across a sentence the way a Transformer can, and word-by-word decoding is slow too. This is the root reason it was eventually replaced.
The idea wasn't conjured from nothing: Kalchbrenner & Blunsom (2013) and Cho et al. (2014, a few months earlier) had proposed similar encoder–decoder frameworks; this paper's distinct contribution was using deep-enough LSTMs + reversal + serious compute to make end-to-end translation actually beat SMT for the first time — proof the road was passable.

Takeaways

① One sentence: two deep LSTMs form an encoder–decoder that exchanges only a fixed-length vector v, learning "variable-length sequence → variable-length sequence" end to end.

② Pain point: deep networks only handled fixed-size inputs and outputs, while core problems like translation are variable-length both ways; the SMT pipeline had too many modules to optimize jointly.

③ Mechanism: the decoder is a language model "set in key" by v — p(y|x) = ∏ p(yₜ|v, y₁…yₜ₋₁) — generating word by word and stopping at <EOS>, so the model decides its own output length.

④ The big trick: feed the source reversed, turning start-of-sentence alignment into short-range dependencies; BLEU 25.9 → 30.6.

⑤ Results: BLEU 34.8 on WMT'14 English–French, beating the 33.3 SMT baseline — the first neural win; rescoring reached 36.5.

⑥ Evidence: sentence-vector visualization clusters by meaning and ignores voice — v really encodes "meaning."

⑦ Impact: established the seq2seq paradigm; spawned GNMT, attention, and the Transformer; today's LLM question-answering shape is its direct descendant.

⑧ Limits: the fixed vector loses information on long sentences (attention was born to fix it), reversal is an empirical hack, UNK vocabulary problems, and the hard-to-parallelize RNN was ultimately replaced by the Transformer.