Day 61 · 2026.08.22

Statistical Learning Theory

The mathematics of moving from finite samples to an unseen world — and its crisis in the face of deep learning
"Learning is not the pursuit of truth. It is pushing 'too often wrong' and 'unlucky draw' down to a level you can live with."

The PAC Framework

Probably Approximately Correct · a provable definition of "learned it"
Framework
Intuition

Start by conceding two impossibilities: you see only finitely many samples, so you might always draw an atypical batch; and even with a typical batch you will not be exactly right. Valiant's 1984 compromise is brutally practical — define "learned" as Probably (with high chance) Approximately Correct (with small error). Two tolerances: $\varepsilon$ lets you be a little wrong, $\delta$ lets you be unlucky once.

Here is an example you can compute end to end. The target is an axis-aligned rectangle in the plane, and the algorithm is trivial: take the smallest rectangle enclosing all positive examples. It can only under-shoot, never over-shoot, so all the error hides in four thin strips just inside the true rectangle's edges. If one strip carries probability mass above $\varepsilon/4$ and yet none of the $m$ samples landed in it, that bad event has probability $(1-\varepsilon/4)^m$; a union bound over the four strips gives $4e^{-m\varepsilon/4}\le\delta$, hence $m\ge\frac{4}{\varepsilon}\ln\frac{4}{\delta}$. "I have learned it" has been translated into a piece of arithmetic about sample size.

solid = target rectangle dashed = tightest enclosing box the four strips hold all the error; each stays empty w.p. ≤ e^(−mε/4) m ≥ (4/ε)·ln(4/δ) → sample size buys ε and δ
Formal definition
$$\Pr_{S\sim D^m}\Big[\ \mathrm{err}_D(\hat h)\le\varepsilon\ \Big]\ \ge\ 1-\delta$$

$D$ is the unknown true distribution behind the data; $S$ is $m$ samples drawn independently from $D$; $\hat h$ is the hypothesis the algorithm returns after seeing $S$; $\mathrm{err}_D$ is its error rate on the real world. The outer $\Pr$ governs luck in sampling, the inner $\varepsilon$ governs precision — two layers of uncertainty, one dial each. Learnable means: there is an algorithm whose sample size $m(\varepsilon,\delta)$ is polynomial in $1/\varepsilon$ and $1/\delta$.

Why it is beautiful

It turns "learning," a philosopher's word, into a provable proposition — and exposes the structure of the price along the way: $\varepsilon$ appears as $1/\varepsilon$, but $\delta$ appears only as $\ln(1/\delta)$. Precision is expensive, confidence is cheap. Driving your failure rate from 5% to 0.001% costs a few times more data; halving your error costs double or worse. That asymmetry explains why "run it a few more times to confirm" is a bargain in engineering while "one more order of magnitude of accuracy" never is. And it is distribution-free.

Applications

PAC is the shared language of active learning, bandit sample complexity (Day 54), and differential-privacy budget analysis. Boosting was born from a question posed inside the framework: does weak learnability (barely better than a coin flip) imply strong learnability? Schapire proved in 1990 that it does; AdaBoost and XGBoost are constructive byproducts of that proof. Asking how many samples an A/B test needs is still asking for the same $m(\varepsilon,\delta)$.

Essence + a question
PAC does not chase truth. It promises that a controllable number of samples buys "too often wrong" and "unlucky" down to a price you accept.
Ask: the last time you drew a conclusion from a 0.5% gap between two benchmark scores — what was your $\delta$?

VC Dimension & Shattering

Vapnik–Chervonenkis Dimension · capacity is not parameter count
Capacity
Intuition

How do you measure how "powerful" a model family is? Don't count parameters — count how many points it can shatter. Given $d$ points, if for every one of the $2^d$ ways of labelling them $\pm$ some member of the family fits perfectly, the family shatters those points. The largest such $d$ is the VC dimension.

Lines in the plane: take any 3 non-collinear points and all 8 labellings are linearly separable. Four points? Place them as a square with equal signs on the diagonals — that is XOR, and no line can do it. So a planar linear classifier has VC dimension exactly 3; in $d$ dimensions it is $d+1$.

Meanwhile $\mathrm{sign}(\sin\theta x)$ has a single parameter and infinite VC dimension. There is no reliable exchange rate between parameter count and capacity.

3 points: all 8 labellings reachable 4 points, XOR: no line works VC(lines in the plane) = 3
Formal definition
$$\mathrm{VC}(\mathcal H)=\max\big\{d:\ \exists\,x_1..x_d,\ |\mathcal H_{|\{x_i\}}|=2^d\big\}$$

$\mathcal H_{|S}$ is the set of labelling patterns the class $\mathcal H$ can actually produce once restricted to the point set $S$ — two functions that label $S$ identically count as one pattern. So $|\mathcal H_{|S}|=2^d$ says "all $2^d$ patterns, none missing."

The real theorem is the Sauer–Shelah lemma: if $\mathrm{VC}(\mathcal H)=d$, the number of patterns on any $m$ points is $\le\sum_{i=0}^{d}\binom{m}{i}=O(m^d)$. A quantity that could have been $2^m$ collapses to a polynomial in $m$ the moment the VC dimension is finite — and the transition happens exactly at $m=d$.

Why it is beautiful

This is a purely combinatorial dichotomy. One expects "complexity" to be continuous, graded, hard to pin down; Sauer–Shelah says a function class has only two fates — shatter everywhere ($2^m$, infinite VC, unlearnable) or be pinned by a single integer $d$ to $O(m^d)$. There is no middle ground. More striking still, the lemma was discovered independently three times: in extremal combinatorics, in Shelah's stability theory (NIP) in model theory, and in the $\varepsilon$-net theorem of discrete geometry. Three groups who never spoke hit the same wall — usually a sign they were touching the same structure.

Applications

This is where the SVM gets its theoretical licence: kernels lift data to infinite dimension with infinitely many parameters, yet the max-margin constraint bounds the effective VC dimension by $(R/\gamma)^2$ ($R$ the data radius, $\gamma$ the margin), independent of the ambient dimension — the first elegant proof that the curse of dimensionality can be sidestepped geometrically.

Essence + a question
Capacity is not how many parameters you have; it is how many distinct labelling patterns you can produce — and finite VC dimension collapses an exponential into a polynomial.
Ask: a model with ten billion parameters, trained on data drawn from a single task distribution — how would you compute its effective capacity?

Generalization Bounds & Uniform Convergence

Occam's razor, made quantitative
Bounds
Intuition

Low training error is not by itself evidence — you picked this hypothesis after seeing the answers, which is shooting the arrow and then painting the target. To make training error trustworthy you must guarantee that every member of the hypothesis class has training error close to its true error (that is uniform convergence), so that whichever one you pick afterwards is safe.

And "every" costs money: the larger the class, the higher the insurance premium on your freedom to choose in hindsight. Every generalization bound therefore has the same shape — true error ≤ how well you fit + how big a pool you picked from.

training error ↓ capacity penalty ↑ bound = their sum optimum capacity d → error
Formal definition
$$\mathrm{err}_D(h)\ \le\ \widehat{\mathrm{err}}_S(h)\ +\ O\!\left(\sqrt{\frac{d\log(m/d)+\log(1/\delta)}{m}}\right)$$

With probability $1-\delta$, simultaneously for all $h$ in the class. $\widehat{\mathrm{err}}_S$ is training error, $d$ the VC dimension, $m$ the sample count. Three details worth reading: the outer square root comes from the central limit theorem — estimating one mean from $m$ samples has error of order $1/\sqrt m$; $d\log(m/d)$ is exactly the Sauer–Shelah polynomial after taking a logarithm, and stands for "how many essentially distinct hypotheses must be covered at once"; $\log(1/\delta)$ confirms once more that confidence is cheap. The ratio that matters is $d/m$.

Why it is beautiful

It gives Occam's razor a quantitative form. "The simpler explanation is more likely right" was an aesthetic preference for two thousand years; here it becomes a provable theorem — small $d$ converts directly into a tighter guarantee. Vapnik then turned it into an algorithm, structural risk minimization: do not minimize training error, minimize "training error + complexity penalty." Every regularization term you have ever written descends from this one inequality.

Applications

L2/L1 regularization, tree pruning, early stopping and dropout can all be read as suppressing effective capacity rather than fitting power. The more modern Rademacher complexity swaps in a different measuring stick — "how well can this class fit pure random labels?" — which varies with the actual data distribution and hugs reality far more closely than VC dimension.

Essence + a question
A generalization guarantee never says "I fit well." It says "I picked from a small enough pool."
Ask: you tuned 200 hyperparameter configurations for one model — how large was your real hypothesis class, and whose account is that capacity charged to?

The Generalization Puzzle of Deep Learning

Benign Overfitting & Double Descent · the old theory asked the wrong question
Modern
Intuition

Classical theory makes a clear prediction: far more parameters than samples → capacity explodes → catastrophic overfitting. Reality says otherwise. Zhang et al. ran a merciless experiment in 2017: randomize every CIFAR-10 label, and the network still trains to zero error — its capacity really is enough to memorize anything. Put the true labels back and the same network generalizes well. One model class can memorize pure noise and also learn structure, which means capacity is not what decides its fate. The algorithm's preference is.

More counterintuitive still is double descent: as model size grows, test error first falls then rises (the classical U), peaks right at the interpolation threshold (parameters ≈ samples), and then — keep growing — falls again, often below the first valley. Classical theory only ever saw the left half of this curve.

classical U second descent interpolation: params ≈ samples size → test error
Formal definition
$$\hat h=\arg\min_{h\,:\,\widehat{\mathrm{err}}_S(h)=0}\ R(h)$$

The key idea is implicit regularization: among all solutions that drive training error to zero, gradient descent does not pick one at random — it leans systematically toward a particular kind. $R$ is that preference: a complexity measure defined implicitly by the optimizer, not by the loss function. This is not hand-waving — on separable data, gradient descent on a linear classifier provably converges in direction to the max-margin solution (which lands us straight back on the VC card). Nobody knows what $R$ is for deep nonlinear networks; that ignorance is the open problem.

Why it is beautiful

This is a genuine theoretical crisis whose resolution is not to overthrow the old theory but to notice it asked the wrong question. Uniform convergence asks "is the entire hypothesis class safe?"; what actually happens is that the algorithm only ever visits one tiny corner of that class. Demand safety for the whole class and you get a vacuous bound. Benign overfitting supplies the other half: in high dimensions, noise can be absorbed into directions that barely affect prediction, so overfitting and generalization coexist. Mathematics did not fail — our ruler for capacity was simply too coarse, and knowing where it is coarse is itself progress.

Applications

This rewrites practical intuition: why "make it bigger" so often helps rather than hurts (the theoretical undertone of scaling laws); why model selection in the interpolating regime must rely on held-out data rather than a complexity penalty; why weight decay, early stopping and data augmentation are not interchangeable — each changes which solution the algorithm walks to, not which solutions exist in the class.

Essence + a question
The secret of generalization lies less in what a model can express than in where the optimizer is willing to go.
Ask: swap in a different optimizer that reaches the same zero training error — does generalization change? That the question can even be posed is already the answer.

Deeper

If VC dimension can't explain deep learning, should "capacity" be retired?
Not retired — re-denominated. VC dimension measures the capacity of a hypothesis class, while generalization depends on the algorithm–data–class triple. Rademacher complexity depends on the data distribution, PAC-Bayes on the posterior the algorithm outputs, and compression bounds simply measure how many bits the one trained model takes. All three keep the original skeleton — error ≤ fit + some complexity / $\sqrt m$. The skeleton lives; the skin is being replaced.
Is implicit regularization the same statement as the No Free Lunch theorem?
Same statement, two moods. NFL says that averaged over all possible target functions no algorithm beats random guessing — without bias, no learning. Implicit regularization says which particular bias gradient descent carries (small norm, large margin, low frequencies first). NFL tells you that you must take a side; implicit regularization tells you that you already took one without noticing. The real question then becomes: why is that bias so well aligned with the structure of real-world data?
Most generalization bounds evaluate to more than 1 — vacuous. Why do theorists still care?
Because the bound's shape is far more useful than its number. $\sqrt{d/m}$ tells you that quadrupling the data halves the error, and that scaling law guides budgets even when the numeric bound is vacuous. It also names which quantities sit in the numerator — and every quantity in the numerator is a knob you can turn. Regularization, pruning and distillation all grow out of that list.
If a model can memorize its training set, where is the line between memorization and generalization?
The recent answer is surprising: under long-tailed distributions, memorization is part of generalization — Feldman proved that when rare examples appear only once or twice, memorizing them is necessary to reduce total error. Privacy and utility then collide by construction: membership-inference attacks exploit exactly that memory, and differential privacy weakens it by limiting any single sample's influence, with the cost landing on tail accuracy. That is a provable trade-off, not a technical defect.