CS PAPERS DEEP-READ · PAPER 30
Srivastava, Hinton, Krizhevsky, Sutskever & Salakhutdinov · University of Toronto · JMLR 2014
In 2014, Geoffrey Hinton's group in Toronto wrote up a trick that sounds like sabotage: while training a neural network, randomly send some of its neurons home at every single step — pretend, for that step, that they do not exist. The trick is called dropout, it takes a few lines of code, and it treats deep learning's most common disease: the model memorizes the practice questions and falls apart on the real exam. AlexNet, the network that won ImageNet in 2012 and set off this entire wave of AI, used it.
A neural network has millions of tunable knobs, and training data is always finite. With that many knobs, it can memorize the accidents that happen to be in this particular batch of data — "the cat photos all had grass in the bottom-right corner" — and then flounders on a photo it has never seen. This is overfitting, and back then it was the single biggest obstacle in the field.
The reliable cure was a committee: train several models and let them vote, so their individual flukes cancel out. The problem is the price — dozens of times the compute to train, and every incoming photo run through every model once deployed.
Dropout's move is almost crude in its simplicity: at each training step, pick roughly half the neurons at random and erase them, along with every wire going in and out. Let the crippled network that remains learn from that step as usual. Next step, draw again — a different half goes home.
First, it breaks up the cliques. If a neuron can always count on the one next door to cover for it, whatever it learns only works while that neighbour is present; groups of neurons lean on each other, patch each other's mistakes, and skills like that are brittle on new data. Once anyone might be absent, every neuron has to learn something that stands on its own. The paper's own analogy is lovely: rather than one conspiracy of fifty people, where a single slip sinks everything, run ten conspiracies of five.
Second, it makes the committee cheap. Every draw gives a different crippled network, so a few hundred thousand steps train an enormous family of them — and because they all share the same set of knobs, you paid for exactly one. That expensive voting effect comes free.
At prediction time you cannot keep drawing lots — the same photo would get different answers twice in a row. The fix is almost comically plain: let everyone show up, but turn each voice down in proportion to how often they used to attend. Only half the room was present on average during training; now that everyone is there, each speaks more softly so the total matches. Remarkably, that single ordinary forward pass lands very close to polling the whole enormous family of crippled networks.
One honest cost: because every step is fighting random noise, reaching the same quality takes roughly two to three times as long to train.
After dropout, building a bigger network no longer meant memorizing more. It became a standard part in 2012–2016 — nearly every vision, speech and text model of that era used it, and Transformers still keep a slot for it today. It also opened a whole line of thinking: deliberately inject noise during training and the model is forced to learn something sturdier.
Send half the neurons home at random on every training step, so no neuron can lean on a particular partner; the same trick trains an enormous family of crippled networks inside one shared set of weights, and at test time you bring everyone back and turn the volume down by the attendance rate — one forward pass, approximately the whole committee's vote.
Want the mask, why the weights get scaled by p, and the numbers on each dataset? → Switch to the deep read
Dropout randomly deletes units during training (setting their output to zero), so every step trains a different thinned network. This both breaks the co-adaptation between units and amounts to training exponentially many networks that share one set of weights. At test time nothing is random: the weights are multiplied by the retention probability p and a single ordinary forward pass approximates the ensemble average over all those thinned networks. It was the first general, cheap, near-universally applicable regularizer of the deep learning era.
The authors are Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever and Ruslan Salakhutdinov of the University of Toronto, published in JMLR in 2014; its predecessor is the same group's 2012 arXiv note, Improving neural networks by preventing co-adaptation of feature detectors. AlexNet, built by some of the same people that year, used dropout in its fully connected layers and made it the de facto standard. It picks up the classical idea of fighting overfitting with ensembles, and it launches a whole branch of "noise as regularization": DropConnect, stochastic depth, and Monte Carlo dropout (reading dropout as approximate Bayesian inference).
A deep network's expressive power comes from a huge number of parameters, and labelled data is never enough. With that many parameters, a network can fit the sampling noise specific to this training set, and generalization collapses. The tools of the day — weight decay, early stopping, data augmentation — helped, but only so far.
The theoretically beautiful answer is Bayesian model averaging: average over every possible parameter setting, weighted by its posterior probability. On a neural network it is completely intractable. The engineer's fallback is an ensemble: train a batch of networks with different architectures or initializations and vote. Ensembling almost always pays, but it multiplies training cost, tuning cost and inference cost by the number of models — which, at 2012 compute levels, put it out of reach for large networks.
The paper also names a subtler root cause: co-adaptation. Backpropagation only cares that the overall loss goes down; nothing stops a hidden unit from leaving its mistakes for other units to patch up. Groups of units then evolve into an entangled whole, where a feature is only meaningful when several other specific features also fire. Such committee-built features work on the training set and break on anything else. What you actually want is for each unit to be a useful feature detector on its own.
During training, for each training case (in practice, each mini-batch) the dice are rolled afresh: every unit is kept with probability p and dropped with probability 1−p. Dropped means its output is set to zero and every wire into and out of it is dead for that step. What remains is a thinned network, which does a normal forward and backward pass and updates its own share of the weights. Next step, roll again.
As a formula it is one line: multiply a layer's outputs y element-wise by a 0/1 mask r, giving ỹ = r ⊙ y, where each entry of r is an independent Bernoulli draw that is 1 with probability p; then carry on as usual with z = w·ỹ + b into the next layer. In plain words: for this step the unit does not exist — its message is not passed on, and no error is sent back to it.
Empirical values: p = 0.5 for hidden layers (keep half) is close to optimal across a wide range of tasks; input layers need to be far more conservative, typically keeping p ≈ 0.8 — the input is the information, and dropping much of it just deletes data.
Random deletion kills co-adaptation outright: a unit can no longer count on any particular partner being present, because that partner may be gone at any moment. It is forced to learn features that are useful across many random combinations of teammates — that is, features that stand on their own. The paper offers two fine analogies. One: rather than a single fifty-person conspiracy, where one failure sinks the plot, organize ten teams of five — redundant and robust. The other is sexual reproduction — a gene must work with whatever other half of the genome it is paired with, which selects for capabilities that mix rather than finely coupled complexes. The authors' visualizations of hidden units make the same point: without dropout the units learn unstructured, scattered patterns; with it, they learn clean edges, strokes and spots.
Seen from another angle, n units allow 2ⁿ keep/drop combinations, hence 2ⁿ possible thinned networks. Training amounts to sampling from that vast family and training the samples one at a time — and crucially, they all share one set of weights: the parameter on any given edge is used and updated by every thinned network that contains it. So for the cost of training one network you get a whole family trained at once.
That is the way around the "ensembles are too expensive" problem. If training dozens of networks separately is unaffordable, let exponentially many of them grow up inside one set of weights instead. They are of course highly correlated, and each is sampled only a handful of times, so this is only an approximate ensemble — but in practice a strikingly effective one.
The proper way to use an ensemble is to poll every member and average, but here there are 2ⁿ members, which is obviously impossible. The paper's weight scaling rule is startlingly simple: at test time keep every unit, but multiply each unit's outgoing weights by p, then run one ordinary forward pass.
Why p? Because during training a unit was present only a fraction p of the time, so the expected input reaching the next layer was "weight × output × p". At test time the unit is always there, so without scaling the next layer would receive signals roughly 1/p times too large, pushing the network into a numeric regime it was never trained in. Multiplying by p restores the training-time expectation.
How good is the approximation? For a single softmax layer the rule is exactly the (normalized) geometric mean of all the thinned networks' predictions. For a deep nonlinear network it is only an approximation, but the paper checks it with a Monte Carlo experiment: actually sample k thinned networks, predict with each and average, and the error falls as k grows — with the single weight-scaled forward pass performing about as well as averaging a few dozen sampled networks. One pass buying dozens of passes' worth of quality is what makes dropout practical.
Dropout does not work in isolation; the paper prescribes a recipe. A max-norm constraint — hard-capping the length of each unit's incoming weight vector at some c and rescaling it back if it exceeds that — combined with a large learning rate and high momentum. The reasoning: dropout's noise makes gradients loud, so you want big steps and momentum to average the noise away; but big steps let weights blow up, and max-norm supplies exactly the ceiling that prevents it. The authors report that the combination works best.
There is also a variant: Gaussian dropout, which multiplies activations by a Gaussian random number with mean 1 instead of a 0/1 mask. It works as well or better than the Bernoulli version, which says the essential ingredient is not the deletion as such but injecting mean-preserving multiplicative noise into the activations. Starting from the special case of linear regression, the authors also show that dropout there reduces to a form of L2 regularization scaled adaptively per feature — an analytic anchor for the question of what kind of regularizer it really is.
The paper's force comes from consistent gains across domains: the same trick helps on vision, speech, text and biological data alike.
Two control studies are especially informative. First, the effect of dataset size: with very little data (a few hundred MNIST examples) dropout does not help and can hurt — the model is underfitting already, and more noise only makes it worse; the gain is largest at moderate data sizes and shrinks again when data is abundant. Second, the effect of the retention probability: for hidden layers any p between 0.4 and 0.8 works well; too small and the network underfits, too close to 1 and there is not enough noise to matter, which degenerates into an ordinary network.
Dropout changed the arithmetic of "how big should the network be." Before it, going wider and deeper meant overfitting more easily, so capacity was throttled by generalization. After it, build the network large and hold overfitting down with dropout became routine, which directly enabled the 2012–2016 wave of ever-larger models. It also became a single line of API in every framework, used by essentially every vision, speech and text model of the period.
More lasting is the template it established: inject random noise during training and you get an approximate model average for free. A large body of later work descends from it — DropConnect (drop connections rather than units), stochastic depth (skip whole residual blocks at random), DropPath, and on the data side Cutout and Mixup. Gal and Ghahramani went further, interpreting dropout as approximate Bayesian inference in a deep Gaussian process, so that leaving dropout switched on at test time and running several passes (Monte Carlo dropout) gives a cheap estimate of model uncertainty — still one of the most widely used uncertainty estimators in practice. Today's Transformers still hang dropout on the attention weights, the feed-forward layers and the residual outputs.
p, to tune.① In one line: during training, delete units together with their connections at random so each step trains a different thinned network; at test time drop nothing and run one forward pass with weights scaled by the retention probability p.
② The pain: large networks plus finite data means inevitable overfitting; the reliable cure — ensembling — is prohibitively expensive in both training and inference, and Bayesian model averaging is intractable.
③ The root cause: co-adaptation — units cluster and patch each other's mistakes, so a feature is only meaningful when specific partners are present, and it breaks on new data.
④ The mechanism: multiply the layer's outputs by a Bernoulli 0/1 mask, ỹ = r ⊙ y; typically keep p=0.5 in hidden layers and p≈0.8 at the input.
⑤ The two intuitions: nobody can count on a particular partner being present, so every unit must learn a feature that stands on its own (ten teams of five > one conspiracy of fifty); and n units correspond to 2ⁿ thinned networks sharing one weight set, so one network's training cost buys an approximate ensemble.
⑥ Why scale by p at test time: it restores the next layer's expected input to training levels; for a single softmax layer it is exactly the geometric mean, and in deep networks one scaled pass performs about as well as averaging a few dozen sampled thinned networks.
⑦ Results: MNIST about 1.60% → 1.25% (best about 1.05%); CIFAR-10 14.98% → 12.61%; SVHN about 3.95% → 2.55%; TIMIT about 23.4% → 21.8% — consistent across vision, speech and text. The recipe also includes max-norm, a large learning rate and high momentum.
⑧ Impact and limits: it established "inject noise ≈ cheap ensemble" as a template, spawning DropConnect, stochastic depth and MC dropout, and Transformers still use it; but training is 2–3× slower, BN largely replaced it in convnets (and stacking the two causes a variance shift), it is not directly usable on recurrent connections, and it does nothing when data is very scarce or very abundant.