Long Short-Term Memory (LSTM) — CS Papers Deep-Read Paper 7

What did this paper do?

Before ChatGPT, the things in your phone that could listen and translate — voice assistants, Google Translate, handwriting recognition — mostly had the same 1997 invention inside: LSTM (Long Short-Term Memory). This paper gave AI a memory device that can remember things from long ago, and it ruled "process things in order" AI for a full twenty years.

First, a puzzle

Sentences and speech must be read in order, one piece at a time. The "memory networks" of the day played a game of telephone: on each new word, they took everything remembered so far, kneaded it all over again, and mixed the new word in. After a dozen rounds of kneading, whatever was said at the start is unrecognizable. Worse, you couldn't even teach them: AI learns by "tracing blame backward after a mistake," and that blame signal also fades station by station as it travels back — gone after a few dozen steps. So the network could never learn long-distance accounting like "the mistake at the end was caused by something at the beginning."

The idea

LSTM's idea: stop letting memory get kneaded at every step. Give the network a protected memory conveyor belt: whatever is placed on it is carried forward exactly as is — never kneaded, never rewritten — no matter how far it travels.

How the gatekeepers work

A conveyor alone isn't enough — dump everything onto it and it becomes a junkyard. So each memory slot gets two gates, and the gatekeepers are learned by the network itself: the write gatekeeper decides "is this piece of information worth recording?", and the read gatekeeper decides "should the stored stuff be taken out and used right now?" Most of the time both gates stay shut and the contents ride along untouched; only at the right moments do the gates open to write or read. There's a second payoff: because the belt never kneads anything, the backward "blame" signal can also travel smoothly all the way back to long ago — long-distance accounting finally works.

What it brought

For the next twenty years, almost every best-in-class system for "one step at a time" work — understanding speech, translating text, reading handwriting — had LSTM inside: Google voice search, Google Translate, and Siri all used it. One honest note: it must compute strictly step by step, so it's slow, and its memory is far from unlimited — in 2017 the Transformer, where all words look at each other directly, took over.

Remember one thing

Give a neural network a memory conveyor belt that carries things forward untouched, plus a write gate and a read gate that learn when to open — distant information survives, the correction signal travels back, and AI could truly remember things from long ago for the first time — ruling sequence tasks for twenty years.

Want the memory-cell diagram, the formulas, and the 1000-step experiments? → switch to the deep read

In one sentence

LSTM builds a gated memory cell into recurrent networks: at its core is a self-connected loop with weight fixed at 1 — the "constant error carousel" — letting both memories and training error signals flow across a thousand-plus time steps without decay, while learned input and output gates decide when to write and when to read. It cured the RNN's vanishing gradient at the root, let networks learn dependencies spanning 1000+ steps for the first time, and ruled speech, translation, and other sequence tasks for the next twenty years.

Glossary

Sequence data: data with an order — words in a sentence, frames of audio, daily stock prices. It must be read in order, and "later" often depends on "much earlier."
RNN (recurrent neural network): a network built for sequences: at each step it updates its "memory so far," then carries that memory into the next step.
Hidden state: the RNN's carried "memory" — a set of numbers recomputed by the network at every time step.
Time step: one slot of the sequence — reading the k-th word means being at time step k.
Weight: an adjustable knob in a neural network; training means nudging these numbers.
Backpropagation: the training algorithm: compute how wrong the output is, then pass that "correction signal" backward through the network to tell each weight how to adjust. In an RNN it must travel back through time (BPTT).
Gradient / vanishing gradient: the gradient is the size of that correction signal; "vanishing" means it shrinks step by step going backward until it's nearly zero, so early steps never learn.
Sigmoid: a function that squashes any number into the range 0 to 1 — perfect as a "valve opening": 0 fully closed, 1 fully open.

Where it sits

The authors are Sepp Hochreiter and Jürgen Schmidhuber; the paper appeared in the journal Neural Computation (1997). The diagnosis of the problem comes from Hochreiter's 1991 thesis — among the first rigorous analyses of why RNN gradients vanish or explode. It inherits the RNNs and backpropagation of the 1980s, and leads to seq2seq (2014) and Bahdanau attention (2014) — which three years later grew into the Transformer (2017). Until then, the throne of sequence modeling was LSTM's.

The problem and the motivation

The RNN vision is beautiful: compress history into a hidden state and, in principle, remember as long as you like. In practice its memory is terrible. The reason lies in training: as the correction signal travels backward through time, each step back multiplies it once by "weight × activation slope." That multiplier is almost never exactly 1 — below 1 the signal shrinks exponentially (vanishing gradient); above 1 it blows up exponentially (exploding gradient, training diverges). After a few dozen multiplications, it's either zero or infinity.

The consequence: a plain RNN can only learn dependencies about ten-odd steps apart. Yet real tasks are full of long-distance accounting — the subject at the start of a sentence dictates the verb form at the end; a name in the first paragraph decides who "he" means in the last. The era's remedies (truncating the backward pass, hand-built time-delay structures, hierarchical sequence compression) were either band-aids or task-specific. This is a structural disease, and it needed a structural cure.

Fig 1 · The RNN's disease: the correction signal is rescaled at every step back through time, vanishing (or exploding) exponentially — long-range dependencies can't be learned.

The core idea

Idea one: the constant error carousel (CEC) — make the multiplier exactly 1

If the disease is "one multiplication per step back, and the multiplier isn't 1," then build a channel whose multiplier is always 1. The construction: take a linear unit (no squashing, no kneading) and give it a self-loop with the weight fixed at 1 — by default, this step's state is copied to the next step unchanged: c(t) = c(t−1) + new write. When the error signal travels backward along this loop, every step multiplies by exactly 1, so it crosses a thousand steps without shrinking or exploding. The paper calls it the constant error carousel (CEC) — the true heart of the whole work.

It may look familiar: ResNet's skip connection and the Transformer's residual connection walk the same line of thought — "keep it as is" shouldn't have to be learned; the structure should provide it for free. LSTM is that principle's forerunner, applied along the time axis.

Idea two: multiplicative gates — resolving the write/read conflict

A naked carousel is unusable. The same incoming weight would have to both "write the information in at the key moment" and "block irrelevant noise the rest of the time" — two goals fighting each other (the paper's input weight conflict); the outgoing side has the same problem: stored content shouldn't leak out at every step. The fix is to separate "what to store" from "when to write and read," handled by two multiplicative gates:

Input gate: computes an "opening" between 0 and 1, multiplied onto the amount written — 0 means "don't write this step," 1 means "take it all."
Output gate: the same kind of opening multiplied onto what is read out — deciding "should the stored content be handed to the rest of the network right now?"

Gates aren't hand-tuned: each gate is itself a small neuron that looks at the current input and the previous state and learns when to open and close, e.g. i(t) = σ(W·x(t) + U·h(t−1)) — σ is the sigmoid, whose output naturally lands between 0 and 1, perfect as a valve. So at irrelevant moments both gates stay shut and the state glides through unchanged; at key moments they open and reading or writing happens. The CEC keeps memories safe; the gates keep them managed — a memory cell = CEC + two gates.

Fig 2 · The LSTM memory cell: at the center a self-loop with weight fixed at 1 (the CEC); the input gate governs writing, the output gate governs reading; the dashed forget gate was added after 2000.

The formulas — and the missing gate

The original state update is c(t) = c(t−1) + i(t)·g(t) — the previous state kept as is, plus "input-gate opening × candidate write"; the output is h(t) = o(t)·h(c(t)), i.e. "output-gate opening × squashed state." Note: the 1997 original has no forget gate — the state only accumulates and never shrinks, so in a continuous input stream it grows without bound. Gers et al. added the forget gate f in 2000, turning the update into today's standard form c(t) = f(t)·c(t−1) + i(t)·g(t): the network can also learn to "wipe the slate when it's time to turn the page." What everyone calls LSTM today is this forget-gate version.

For training, the paper ships a truncated gradient algorithm that is local in both space and time — O(1) computation per weight per time step — affordable even on 1997 hardware.

Key results

The experiments are all carefully designed artificial tasks — the paper says so itself — but the comparisons are solid:

Embedded Reber grammar: a classic RNN test requiring a symbol from the start of the sequence to be remembered until the very end. The standard algorithms of the day (RTRL, BPTT, and others) often failed; LSTM solved it every time, and learned faster.
Long time-lag tasks: with the gap between "the clue" and "the moment the clue is needed" stretched to a minimal lag of 1000 steps (including noisy versions), LSTM still learned; the BPTT and RTRL baselines already failed completely at gaps of a few dozen steps.
The adding problem: remember two marked numbers inside a long stream of digits and output their sum at the end — testing "store exact values + hold them across long distances + resist distraction." LSTM solved this too.

To be honest: the paper contains no real-world benchmark — "ruling real applications" came a decade-plus later. But "learnable across 1000-step lags" was something no other general method could do at the time.

Why it mattered

This was a seed planted during a "neural network winter" that exploded ten years later. In 2009 Graves and colleagues won international handwriting-recognition competitions with LSTM; speech recognition then switched to it wholesale; in 2014 seq2seq built end-to-end machine translation on stacked LSTMs; in 2015 Google voice search shipped an LSTM acoustic model; in 2016 Google Translate's GNMT used deep LSTMs for both encoder and decoder — in the smartphone years when devices learned to listen and translate, LSTM was underneath almost all of it. It became one of the most-cited neural network papers in history.

Two pieces of its intellectual legacy are still alive. First, gating became a universal weapon — the GRU is its streamlined descendant, and highway networks carried gating into the depth dimension. Second, the principle of "give information a decay-free path whose multiplier is 1" lives on in the residual connections of ResNet and the Transformer. And Hochreiter's rigorous analysis of vanishing gradients is itself a foundational document of deep learning theory.

Limitations and criticism

The original was incomplete: with no forget gate, the state only accumulates and grows without bound on continual tasks; only after Gers et al. added the forget gate in 2000 did the design settle — today's "LSTM" is really the revised version.
Memory is not unlimited: artificial tasks span a thousand steps, but effective dependency lengths on real tasks are typically in the hundreds; tasks demanding precise recall of distant details remain hard.
Inherently sequential: each step must wait for the previous one, so there's no parallelism across time. In the GPU/TPU era that means slow, expensive training — one direct reason it was replaced by the fully parallelizable Transformer.
Relatively complex: three gates plus a state, and many parameters. Cho et al. proposed the two-gate GRU in 2014, which ties on many tasks; Greff et al.'s 2017 systematic comparison of eight variants ("LSTM: A Search Space Odyssey") found none clearly beats the vanilla forget-gate version, with the forget gate and output activation being the most critical parts.
Evidence lagged the claim: the original experiments are all synthetic; the paper's value was only fully stamped by large-scale real-world deployment years later.

The essentials

① The disease: an RNN's correction signal is multiplied by "weight × slope" at every step back through time, vanishing or exploding exponentially — dependencies more than ten-odd steps apart can't be learned (Hochreiter's 1991 diagnosis).

② Core idea: the constant error carousel (CEC) — a linear self-loop with weight fixed at 1; memories and error signals flow along it with a multiplier of exactly 1, crossing a thousand steps without decay.

③ Second idea: multiplicative gates. The input gate governs writing, the output gate governs reading, resolving the conflict of "one weight must both write and block noise"; gates are 0–1 openings learned by the network itself.

④ Formulas: original c(t) = c(t−1) + i·g; after Gers et al. added the forget gate in 2000, c(t) = f·c(t−1) + i·g — today's standard LSTM.

⑤ Results: learns across 1000-step lags on artificial tasks where BPTT/RTRL fail at a few dozen steps; but the paper has no real-world benchmark.

⑥ Impact: ruled sequence modeling for twenty years — handwriting recognition, speech recognition, seq2seq translation, Google voice search and GNMT all sat on top of it.

⑦ Legacy: gating (GRU, highway networks) and the "decay-free path with multiplier 1" (residual connections) live on today.

⑧ Limits: the original lacked the forget gate; effective memory is finite; inherently sequential and expensive to train — handed the throne to the fully parallel Transformer in 2017.