Day 34 · 2026.07.26

The Traps of Statistical Inference

Significance, the replication crisis, and two worldviews about probability
"Far better an approximate answer to the right question than an exact answer to the wrong question." — John Tukey

The p-value

Significance Testing · a ruler almost everyone reads backwards
Statistical Inference
Intuition

A jury starts by assuming the defendant is innocent, then asks: if he really were innocent, how strange would this evidence be? Strange enough, and you convict. The p-value is exactly that strangeness.

The fatal part is the direction of the conditioning. A p-value computes P(evidence | innocent); what people read off it is P(innocent | evidence). Everyone grasps this in the rare-disease screening example — yet transplanted into hypothesis testing, the same mistake has been repeated for a century.

Worse, a p-value surveys only the "innocent" world. It never asks how common this evidence would be if the effect were real (power), nor what fraction of hypotheses like this one are true to begin with (the base rate). Missing those two, it cannot answer the question anyone actually cares about.

1000 hypotheses tested (10% real, power 0.8) no effect: 900 100 5% slip through → 45 80% found → 80 125 "discoveries" with p < 0.05 (different scale) false: 45 true: 80 36% of the "findings" are false
The threshold governs only that 5% on the left. It says nothing about how many hypotheses were wrong to begin with.
Formal definition
$$p=P\big(T(X)\ \ge\ t_{\text{obs}}\ \big|\ H_0\big)$$

$T(X)$ is the test statistic — the whole dataset squeezed into one number, say a difference of group means divided by its standard error; $t_{\text{obs}}$ is the value you actually got; $H_0$ is the null hypothesis of no effect. Note the $\ge$: the p-value counts more extreme data that never occurred. Your conclusion therefore depends on what you would have seen had you repeated the experiment, not only on what you did see — the backbone of the frequentist framework, and the very thing Concept 4 argues about.

Why it is beautiful

The p-value has a near-miraculous property: whenever $H_0$ is true, no matter what distribution the raw data follow or which statistic you chose, the p-value is uniform on $[0,1]$. It is a universal change of coordinates — every experiment, however exotic, pressed onto one shared ruler, so that evidence from different disciplines becomes comparable for the first time. And the trap grows out of precisely that universality: the scale is so uniform that people forget what it measures.

Applications

The shared gatekeeper of clinical trials, A/B tests and the social sciences. Particle physics insists on $5\sigma$ ($p\approx3\times10^{-7}$) before claiming a new particle, precisely because it knows it is scanning an enormous range of energies at once. In 2016 the American Statistical Association took the rare step of issuing a formal statement refuting the common misreadings one by one — needing a policy document to teach people how to read a single number tells you how deep the trap runs.

In one line: a p-value answers "how weird would the data be if there were no effect" — never "how likely is it that an effect exists".
Question: if nine out of ten hypotheses in some field are false to begin with, what fraction of its "significant findings" are doomed to be false — even with everyone doing everything correctly and with ample power?

Multiple Comparisons & the Garden of Forking Paths

The paths you did not take still count
Meta-Science
Intuition

Run 20 unrelated tests, each impeccably at $\alpha=0.05$, and the chance at least one comes out "significant" is $1-0.95^{20}\approx64\%$. You did nothing wrong; noise hands you a discovery all by itself.

Gelman's version is subtler: you genuinely ran only one test. But had the data looked different, you would have split the groups differently, adjusted for a different covariate, used a different outlier rule — and those paths you did not walk are multiple comparisons all the same. With no intent to cheat, the mere fact that your analysis depends on the data you saw has already inflated $\alpha$. You walk one path; the whole garden counts.

number of tests m P(at least one false positive) 100% 64% already at m=20 20 40 0 α=0.05
A 5% error rate per test swells to 64% after twenty — and the steepest part of the curve is exactly where everyday research lives.
Formal definition

Bonferroni pushes the threshold down to $\alpha/m$ (with $m$ tests), controlling the family-wise error rate $\mathrm{FWER}=P(\text{at least one false positive})$. Benjamini–Hochberg changes the target instead: sort the $m$ p-values ascending, find the largest $k$ with $p_{(k)}\le k\alpha/m$, reject the first $k$, and you control the false discovery rate $\mathrm{FDR}=\mathbb{E}[V/R]$ — where $V$ counts false positives and $R$ counts total rejections, so the ratio is "what share of my claimed discoveries are wrong".

Why it is beautiful

BH's breakthrough is not a stronger technique but a redefinition of the problem: from "never be wrong once" to "keep the wrong fraction bounded". That single relaxation restores the power Bonferroni strangles — lowering the bar in the right place beats redoubling effort in the wrong one. Its proof leans on exactly one fact: p-values are uniform under $H_0$, the miracle from Concept 1. And it adapts: the more real signal in the data, the looser the threshold becomes on its own.

Applications

Genome-wide association studies test millions of loci at once, which is why their significance line sits at $5\times10^{-8}$. Neuroimaging's famous "dead salmon" study found significant brain activation in a dead fish's fMRI — the canonical demonstration of skipping correction. Machine learning has the isomorphic disease, test-set reuse: repeatedly selecting models on one held-out set is multiple comparison against it, which is why leaderboard standings are systematically inflated.

In one line: multiple comparison is not about how many tests you ran, but about how many degrees of freedom your analysis path had.
Question: preregistration forces you to fix the analysis before seeing the data. Which degrees of freedom does that actually kill — and which ones does it leave untouched?

Effect Size, Power & the Winner's Curse

What significance cannot buy you
Estimation
Intuition

With a large enough sample, any non-zero difference turns significant. Significance says "almost certainly not zero"; it never says "worth caring about".

The trap in the other direction is nastier. When the true effect is small and the sample tiny (low power), only samples that noise happened to push to an extreme clear the significance line. So once it is significant, the estimate is necessarily exaggerated: the distribution of effect sizes visible in the literature has been truncated. Gelman calls this a Type M error (magnitude inflated); drop the power further and you get Type S errors, where even the sign is wrong.

true effect (small) significance line only this sliver gets published mean published estimate 0 sampling distribution
The threshold slices away most of the distribution; what remains has a mean far to the right of the truth. That gap is the Type M error.
Formal definition

Effect size divides the difference by the noise: Cohen's $d=(\mu_1-\mu_2)/\sigma$, numerator the true gap between group means, denominator the between-individual standard deviation. Dimensionless, it measures a signal-to-noise ratio rather than an absolute gap. Power $1-\beta$ is the probability of detecting an effect that genuinely exists. The Type M error is $\mathbb{E}\big[\,|\hat\theta|\ \big|\ \text{significant}\,\big]\big/|\theta|$ — how many times larger the published estimate runs than the truth. At power 0.1 that ratio is routinely above 3.

Why it is beautiful

It exposes a quiet, universal structure: the act of filtering by significance is itself what turns an unbiased estimator into a biased one. Mathematically it is just a conditioning — truncate a distribution to the right of a threshold and its mean rises; no one has to behave badly. The beauty is that the same operator surfaces everywhere: survivorship bias in investing, selection effects in evolution, the performance drop after a hiring screen, picking models on a validation set. Wherever there is selection, there is a curse.

Applications

In 2015 the Open Science Collaboration repeated 100 psychology studies: only about a third replicated, and the effect sizes averaged half of what was originally reported — exactly what Type M predicts. Early small-sample drug trials routinely shrink in phase III; "the variant that won the experiment underdelivers after launch" is the everyday industrial version. SOTA gains on ML leaderboards are inflated for the same reason: what gets reported is always the best of many attempts.

In one line: significance only answers "is it zero"; effect size and power answer "how big, and how much should you believe it".
Question: a low-powered study reports "significant". Should a rational reader treat that as evidence in favour — or shrink its estimate toward zero? By how much?

Bayesian vs Frequentist Inference

Two views of probability, two different questions
Foundations
Intuition

The two camps do not give different answers; they answer different questions. The frequentist treats the parameter as a fixed unknown constant and the data as random: a 95% confidence interval promises that "if this procedure were repeated indefinitely, 95% of the intervals would cover the truth" — a long-run property of the procedure, saying nothing about the one interval in your hand.

The Bayesian treats the data as settled fact and the parameter as random: a 95% credible interval says "given the data I have seen, the truth lies here with probability 95%" — a belief right now. Intuition has always wanted the second; the standard toolkit delivers the first. That mismatch is the root of a century of misread p-values.

prior p(θ) likelihood p(D|θ) posterior p(θ|D) weaker data, harder pull θ
The posterior peaks between prior and likelihood — that pull back is shrinkage.
Formal definition
$$p(\theta\mid D)\ \propto\ p(D\mid\theta)\,p(\theta)$$

$\theta$ is the parameter you want to infer, $D$ the observed data. $p(\theta)$ is the prior — how large you believed $\theta$ might be before looking; $p(D\mid\theta)$ is the likelihood — how probable this data would be if $\theta$ took that value; multiply and normalise to get the posterior. The prior is often dismissed as subjective contamination, yet what it carries is exactly the information Concept 1 was missing: how large effects of this kind usually are, and how many hypotheses of this kind are true.

Why it is beautiful

The Bayesian view collapses the first three traps into one hole — the absence of a prior over base rates and effect sizes. The hierarchical-model repair is startlingly elegant: give the group-level parameters a shared prior and the posterior produces shrinkage automatically, pulling each group's estimate toward the overall mean, pulling hardest on the noisiest. Multiple comparison needs no patch; the winner's curse is discounted on its own. As Gelman puts it: what multiple-comparison corrections approximate, multilevel models simply do. One mechanism, three diseases cured — because it is not a patch but an honest statement of how the data were generated. The frequentist beauty is just as real: no prior required, and an auditable worst-case long-run error rate.

Applications

A/B platforms increasingly report "probability B beats A" and expected loss directly, far closer to the language of decisions than a p-value; Thompson sampling allocates traffic by posterior probability. The most concrete counterpart in machine learning is regularisation as a prior: an $L_2$ penalty is a Gaussian prior on the weights, $L_1$ a Laplace prior, and the optimum you train to is the MAP estimate. Shrinkage for cold-start items in recommenders and credibility estimates in actuarial work are hierarchical priors doing the same job.

In one line: the frequentist guarantees a procedure is reliable in the long run, the Bayesian answers what to believe now — the misreading happens when you read the former's number in the latter's voice.
Question: if the prior can only be guessed at, the strength of shrinkage becomes a new degree of freedom. Has subjectivity been eliminated, or merely relocated?

Going Deeper

If p-values are so easily misread, why not retire them outright?
Because every substitute has its own price. A p-value needs no agreed prior, is cheap to compute, and can be independently recomputed by a third party — a low-cost, auditable gatekeeper. The real disease is not the number but the 0.05 threshold that slices continuous evidence into "significant / not", plus an incentive structure that publishes only the significant half. That is why the ASA's 2019 recommendation was to abandon the phrase "statistically significant" rather than the p-value itself; preregistration and registered reports are the treatment that targets the cause.
Deep learning barely does significance testing. Has it escaped these traps?
No — it only changed their appearance. Reusing one test set over and over is multiple comparison; a leaderboard that records only the best attempt is the winner's curse; tuning seeds, sweeping hyperparameters and revising cleaning rules is the garden of forking paths. Recent papers demanding multiple seeds and interval reporting are statistics' old answer arriving a round late. The theoretical response is adaptive data analysis: differential privacy yields a quantitative bound on how many times a held-out set can still be queried — treating information leakage as a measurable budget.
Can the Bayesian–frequentist dispute be settled mathematically?
Partly. Cox's theorem shows any measure of uncertainty satisfying consistency axioms must be isomorphic to probability theory, which favours the Bayesian side; de Finetti's exchangeability theorem answers that frequencies are themselves the limit of exchangeable beliefs. The Bernstein–von Mises theorem goes further, guaranteeing that in large samples the posterior is asymptotically the frequentist sampling distribution — so the disagreement only becomes visible with small samples and influential priors. Which is precisely where the hardest and most consequential judgements have to be made.