CS PAPERS DEEP-READ · PAPER 49

Adam: The Default Solver for Stochastic Optimization

Kingma & Ba · University of Amsterdam / University of Toronto · ICLR 2015

What did this paper do?

In late 2014, two researchers (Kingma and Ba) proposed an "optimizer" called Adam. Training a neural network (say the model behind ChatGPT) is really about turning billions of "knobs" to just the right settings, one step at a time — and the optimizer is the set of rules that decides which way to turn each knob and by how much on every step. Adam's contribution: it makes this work almost tuning-free — with default settings it just runs, steadily. It is simple, stable, and useful enough that today nearly every deep model is trained with it.

First, an annoyance

Picture training as walking downhill blindfolded: you can only feel the slope under your feet and take a step downhill. How big a step is the "learning rate." The catch — the whole mountain forces you to use one fixed stride. But some directions are cliffs where a slightly-too-big step overshoots and bounces you back and forth; others are long gentle slopes where a small step means crawling forever. One stride can't serve both cliffs and plains, so it's always wrong somewhere. Tuning that stride becomes a chore, a tightrope walk: a hair too big and you oscillate and diverge, a hair too small and it's hopelessly slow.

The idea

Adam fuses two plain ideas. First, "momentum": don't just look at this one step's slope (jittered left and right by random noise) — remember "which way have I mostly been heading lately" and follow that averaged direction, the way a ball rolling downhill has inertia and isn't stopped by every little bump. Second, "give every direction its own stride": instead of one stride for the whole mountain, let each direction set its own step size automatically — a direction that keeps swinging wildly takes small, careful steps; a direction that barely moves takes big, bold ones.

How does it pull that off?

Adam keeps two little ledgers for each knob. The first records "which way am I heading on average in this direction" (an inertia-smoothed average of the recent slopes, erasing the jitter). The second records "how big and how jittery has the slope been here" (an average of the recent slope magnitudes). To take a step, it heads in the direction of the first ledger, but scales the stride using the second: a direction that's always steep and jittery gets its step divided down to stay steady; a calm direction gets a bigger stride.

The beauty: whether the slopes are large or small overall — and in what units — cancels out in that one division. You no longer hand-tune a stride per direction or per stage; every step covers roughly "a sensible distance." One thoughtful extra: those two ledgers start out empty, which would bias you toward "afraid to move" early on, so Adam applies a tiny start-up correction that erases the bias and gets even the first few steps right.

What it brought

The result: training a new model went from "spend a few days tuning the learning rate" to "just run it with Adam's defaults." It isn't fussy about settings, works out of the box, and is both fast and stable — so it became the default of deep learning's last decade. Nearly every big model you've heard of was trained with Adam (or an improved variant).

One honest note: it's no cure-all. On some tasks, a simpler method tuned carefully actually reaches a model that generalizes a touch better; and Adam keeps two extra ledgers per knob, so it eats a good chunk of extra memory — which stings more the bigger the model gets.

Remember one thing

Training a model is walking downhill blindfolded, deciding each step's direction and size. Adam keeps two ledgers per direction — "which way on average" (momentum) and "how big and jittery is the slope here" (magnitude) — heads in the average direction and scales the stride by the magnitude, so it's fast and stable with almost no hand-tuning. It's the default optimizer behind nearly every modern deep model.

Want the update formula, the two "moments," bias correction, and the convergence flaw found later? → switch to the deep read

In one sentence

Adam (Adaptive Moment Estimation) fuses momentum with per-parameter adaptive learning rates, plus a bias-correction step: for each parameter it maintains the gradient's first moment (the mean — an inertia-smoothed direction) and second moment (the uncentered variance — how large and jittery that direction's gradient is), and updates via θ ← θ − α·m̂/(√v̂+ε) — heading in the smoothed direction while scaling the step by each dimension's own gradient scale. A single set of defaults α=0.001, β₁=0.9, β₂=0.999 works nearly out of the box, which made it the de facto default optimizer for over a decade of deep learning.

Glossary

Gradient descent / learning rate: training minimizes an "error function"; each step moves a little in the direction of steepest error decrease (the negative gradient), and the learning rate is how big that step is.
Gradient: the vector of error derivatives with respect to every parameter — it says "which way to turn each knob to reduce error fastest."
SGD / mini-batch: each step estimates the gradient from a small batch of samples — fast, but noisy, so the gradient jitters step to step.
Momentum: accumulate a fraction of past gradients, like inertia on a rolling ball — it smooths noise and speeds up across flat regions.
EMA (exponential moving average): a_t = β·a_{t-1} + (1-β)·x_t, a "memory-weighted average" that keeps recent values and forgets old ones at rate β. The closer β is to 1, the longer the memory.
1st / 2nd moment: here, the mean of the gradient (direction) and the mean of the gradient squared (magnitude, i.e. uncentered variance). A "moment" is a number summarizing a distribution.
AdaGrad / RMSProp: Adam's two ancestors, both scaling each parameter's step by "past squared gradients" — AdaGrad sums all history, RMSProp keeps only a moving average of the recent past.

Where it sits

The authors are Diederik P. Kingma (then at the University of Amsterdam, later OpenAI) and Jimmy Lei Ba (University of Toronto); the paper appeared at ICLR 2015 (released on arXiv in late 2014). It inherits momentum, AdaGrad (Duchi et al. 2011), RMSProp (Hinton's unpublished lecture note), and AdaDelta (Zeiler 2012); it launches refinements such as AdamW (decoupled weight decay), and nearly every modern large model — Transformer, BERT, GPT, diffusion models — is trained with Adam or a variant. It is one of the most-cited papers in machine learning.

The problem and the motivation

Plain SGD uses one global learning rate for all parameters and all stages. Three pains: too large and it oscillates or diverges, too small and it's absurdly slow; different parameters need wildly different steps — sparse features (rarely seen, with mostly-zero gradients) need big steps to keep up, while frequently updated parameters need small steps to stay stable; and the same parameter needs different steps early versus late in training. Hand-tuning that one number, plus a decay schedule, is training's most tedious chore.

AdaGrad took the first step: scale each parameter's step by the reciprocal square root of "the accumulated sum of past squared gradients" — rarely-updated parameters get a small denominator and thus big steps, exactly compensating for sparsity. But it accumulates the entire history, so the denominator only grows, the learning rate decays monotonically to zero, and it "stalls" prematurely in long training runs or non-convex deep nets — grinding to a halt before it reaches the bottom. RMSProp swaps the sum for an exponential moving average — keep the recent, forget the old — so the denominator stops ballooning and the stall is fixed. But RMSProp has no momentum and no correction for the moving average's start-up bias. What Adam does is combine RMSProp's adaptive scaling, momentum's directional smoothing, and one crucial bias correction into a single update rule that is fast, stable, and nearly tuning-free.

Fig 1 · The "ravine" intuition: a single-stride SGD ping-pongs across the steep direction and crawls down the gentle one; momentum smooths the jitter, and per-parameter adaptive steps take small steps on steep axes and big steps on flat ones — a straighter, faster path.

The core idea

Two "moments": one ledger for direction, one for magnitude

Adam keeps two exponential moving averages per parameter. Writing the parameter's gradient at step t as g_t:

First moment m (direction / momentum): m_t = β₁·m_{t-1} + (1−β₁)·g_t — a moving average of the gradient itself. It's "which way I've been heading on average," smoothing out the random jitter of mini-batches; this is momentum.

Second moment v (magnitude / jitter): v_t = β₂·v_{t-1} + (1−β₂)·g_t² — a moving average of the gradient squared. It measures "how large and jittery this direction's gradient has been," the ruler used to scale the step.

The update rule: head along m, scale by √v

The core is one step: θ_t = θ_{t-1} − α · m̂_t /(√v̂_t + ε). The intuition — step in the smoothed direction m, but divide the step by that direction's typical gradient magnitude √v: a dimension whose gradient is consistently large and jittery → large √v → the step is divided down and stays steady; a consistently gentle dimension → small √v → the step is enlarged and moves fast. So the overall scale of the gradients, and even differences in units across dimensions, cancel out in that one division — no more per-dimension hand-tuning.

Why divide by √v and not v? For dimensional consistency — √v shares the gradient's units, so the ratio m/√v is roughly a signal-to-noise ratio: when the direction is certain (strong signal, low noise) step boldly, when uncertain step small. A bonus is automatic annealing: near the optimum the average gradient m tends to 0 while the jitter v lingers, so the ratio shrinks and the steps rein themselves in. The authors show the effective step is roughly "capped" by α (|Δθ| ≲ α), a kind of per-step trust region — you know roughly how far a step can move, and it's invariant to rescaling the whole gradient by any constant. ε (default 10⁻⁸) is just a tiny term to avoid dividing by zero.

Fig 2 · The dataflow of one Adam step: the gradient feeds two moving averages — m records direction (momentum), v records magnitude (jitter); each is bias-corrected, then combined into the update that heads along m and scales the step per parameter by √v.

Bias correction: getting the first few steps right

One piece is still missing. Both m and v start at 0, so the moving averages are dragged toward that "0" in the first steps — especially when β is close to 1 (β₂=0.999 mixes in only 0.1% of each new gradient), leaving the early v estimate tiny and biased toward 0. Divide by it directly and the step is miscomputed. Adam applies an exact bias correction to remove this systematic bias: m̂_t = m_t/(1−β₁ᵗ), v̂_t = v_t/(1−β₂ᵗ) — as t grows, 1−βᵗ→1, so the correction fades out later and only matters at the start. This is exactly what lets Adam take correctly-sized steps from the very beginning, and it's the key patch over RMSProp. The authors also give a variant, AdaMax, which replaces the second moment with the gradient's L∞ norm (the running max magnitude), more stable in some settings. The defaults α=0.001, β₁=0.9, β₂=0.999, ε=10⁻⁸ need almost no change across a wide range of tasks.

Key results

The paper compares Adam against SGD-Nesterov, AdaGrad, RMSProp, and AdaDelta on several (by today's standards small) tasks: logistic regression on MNIST, a multilayer network on MNIST, a convolutional network on CIFAR-10, and a dropout-regularized bag-of-words classifier on IMDB. The verdict is consistent: Adam converges faster and more stably, with an especially clear edge under dropout, where the objective itself is very "noisy." The authors also give an O(√T) regret bound for online convex optimization as theoretical support (this proof was later found to be flawed — see below). Honestly: the experiments are small; Adam's real "proof" came from a decade of broad adoption — working out of the box on countless models far beyond the paper's own experiments.

Why it mattered

Adam turned "training a new model" from an experience-dependent craft into a stable default starting point. It's robust to hyperparameters, nearly tuning-free, and steady under heterogeneous and sparse gradients — so it quickly became deep learning's default optimizer: from CNNs to RNNs, to Transformers, BERT, GPT, and diffusion models, the overwhelming majority of modern large models are trained with Adam or its refinement AdamW. Your ability today to "grab a new architecture and just run it with Adam's defaults to see if it works" is, in large part, what this paper granted. As a piece of engineering infrastructure, its influence rivals any single model architecture.

Limitations and criticism

The convergence proof was wrong: Reddi et al. (On the Convergence of Adam and Beyond, 2018) built a counterexample showing the moving-average second moment can make the denominator v decrease (unlike AdaGrad's non-decreasing sum), breaking a key assumption of the original proof, so Adam fails to converge on some simple convex problems; they proposed AMSGrad (use the running max of past v) to patch it.
A generalization gap: Wilson et al. (2017) found that on some tasks (especially computer vision) adaptive methods reach worse test accuracy than a well-tuned SGD+momentum — Adam converges fast but may settle into a solution that generalizes slightly worse. Many vision benchmarks still favor SGD.
Weight-decay coupling: standard Adam folds L2 regularization into the gradient, where it gets distorted by the adaptive scaling; AdamW (Loshchilov & Hutter, 2017/2019) decouples weight decay from the gradient update and is now the de facto standard for training large models — a sign the original wasn't the last word.
It often needs learning-rate warmup: training Transformers without warmup can be unstable because the early second-moment estimate is unreliable (analyzed and fixed by RAdam and others).
Memory overhead: each parameter needs two extra state values, m and v — roughly 3× the parameter memory — a real burden at the hundred-billion-parameter scale, which spawned many memory-thrifty optimizer variants.

The essentials

① In one line: Adam = momentum (1st moment) + per-parameter adaptive step (2nd moment) + bias correction, updating θ ← θ − α·m̂/(√v̂+ε).

② The pain: SGD's single learning rate is one-size-fits-all — too big oscillates, too small crawls, and sparse/steep/flat directions each need a different step; hand-tuning hurts.

③ The ancestors: AdaGrad scales by past squared gradients but decays the rate to zero and "stalls"; RMSProp's moving average fixes the stall but lacks momentum and bias correction.

④ Mechanism: m records direction (smoothing noise), v records magnitude (the ruler); head along m, divide by √v — the gradient scale cancels out and each step covers "a sensible distance."

⑤ Signal-to-noise + auto-annealing: m/√v is roughly an SNR — bold when certain, small when not; near the optimum the ratio shrinks and steps rein themselves in.

⑥ Bias correction: m and v start biased toward 0, so ÷(1−βᵗ) undoes the start-up bias and gets the first steps right — the key patch over RMSProp.

⑦ Defaults α=.001, β₁=.9, β₂=.999 work nearly out of the box; the AdaMax variant uses the L∞ norm.

⑧ Impact: became deep learning's default optimizer; Transformer/BERT/GPT/diffusion models are largely trained with Adam or AdamW.

⑨ Limits: the original convergence proof was flawed (AMSGrad patches it); sometimes generalizes worse than SGD; weight decay must be decoupled (AdamW); often needs warmup; stores two extra states, costing memory.