A jury starts by assuming the defendant is innocent, then asks: if he really were innocent, how strange would this evidence be? Strange enough, and you convict. The p-value is exactly that strangeness.
The fatal part is the direction of the conditioning. A p-value computes P(evidence | innocent); what people read off it is P(innocent | evidence). Everyone grasps this in the rare-disease screening example — yet transplanted into hypothesis testing, the same mistake has been repeated for a century.
Worse, a p-value surveys only the "innocent" world. It never asks how common this evidence would be if the effect were real (power), nor what fraction of hypotheses like this one are true to begin with (the base rate). Missing those two, it cannot answer the question anyone actually cares about.
$T(X)$ is the test statistic — the whole dataset squeezed into one number, say a difference of group means divided by its standard error; $t_{\text{obs}}$ is the value you actually got; $H_0$ is the null hypothesis of no effect. Note the $\ge$: the p-value counts more extreme data that never occurred. Your conclusion therefore depends on what you would have seen had you repeated the experiment, not only on what you did see — the backbone of the frequentist framework, and the very thing Concept 4 argues about.
The p-value has a near-miraculous property: whenever $H_0$ is true, no matter what distribution the raw data follow or which statistic you chose, the p-value is uniform on $[0,1]$. It is a universal change of coordinates — every experiment, however exotic, pressed onto one shared ruler, so that evidence from different disciplines becomes comparable for the first time. And the trap grows out of precisely that universality: the scale is so uniform that people forget what it measures.
The shared gatekeeper of clinical trials, A/B tests and the social sciences. Particle physics insists on $5\sigma$ ($p\approx3\times10^{-7}$) before claiming a new particle, precisely because it knows it is scanning an enormous range of energies at once. In 2016 the American Statistical Association took the rare step of issuing a formal statement refuting the common misreadings one by one — needing a policy document to teach people how to read a single number tells you how deep the trap runs.
Run 20 unrelated tests, each impeccably at $\alpha=0.05$, and the chance at least one comes out "significant" is $1-0.95^{20}\approx64\%$. You did nothing wrong; noise hands you a discovery all by itself.
Gelman's version is subtler: you genuinely ran only one test. But had the data looked different, you would have split the groups differently, adjusted for a different covariate, used a different outlier rule — and those paths you did not walk are multiple comparisons all the same. With no intent to cheat, the mere fact that your analysis depends on the data you saw has already inflated $\alpha$. You walk one path; the whole garden counts.
Bonferroni pushes the threshold down to $\alpha/m$ (with $m$ tests), controlling the family-wise error rate $\mathrm{FWER}=P(\text{at least one false positive})$. Benjamini–Hochberg changes the target instead: sort the $m$ p-values ascending, find the largest $k$ with $p_{(k)}\le k\alpha/m$, reject the first $k$, and you control the false discovery rate $\mathrm{FDR}=\mathbb{E}[V/R]$ — where $V$ counts false positives and $R$ counts total rejections, so the ratio is "what share of my claimed discoveries are wrong".
BH's breakthrough is not a stronger technique but a redefinition of the problem: from "never be wrong once" to "keep the wrong fraction bounded". That single relaxation restores the power Bonferroni strangles — lowering the bar in the right place beats redoubling effort in the wrong one. Its proof leans on exactly one fact: p-values are uniform under $H_0$, the miracle from Concept 1. And it adapts: the more real signal in the data, the looser the threshold becomes on its own.
Genome-wide association studies test millions of loci at once, which is why their significance line sits at $5\times10^{-8}$. Neuroimaging's famous "dead salmon" study found significant brain activation in a dead fish's fMRI — the canonical demonstration of skipping correction. Machine learning has the isomorphic disease, test-set reuse: repeatedly selecting models on one held-out set is multiple comparison against it, which is why leaderboard standings are systematically inflated.
With a large enough sample, any non-zero difference turns significant. Significance says "almost certainly not zero"; it never says "worth caring about".
The trap in the other direction is nastier. When the true effect is small and the sample tiny (low power), only samples that noise happened to push to an extreme clear the significance line. So once it is significant, the estimate is necessarily exaggerated: the distribution of effect sizes visible in the literature has been truncated. Gelman calls this a Type M error (magnitude inflated); drop the power further and you get Type S errors, where even the sign is wrong.
Effect size divides the difference by the noise: Cohen's $d=(\mu_1-\mu_2)/\sigma$, numerator the true gap between group means, denominator the between-individual standard deviation. Dimensionless, it measures a signal-to-noise ratio rather than an absolute gap. Power $1-\beta$ is the probability of detecting an effect that genuinely exists. The Type M error is $\mathbb{E}\big[\,|\hat\theta|\ \big|\ \text{significant}\,\big]\big/|\theta|$ — how many times larger the published estimate runs than the truth. At power 0.1 that ratio is routinely above 3.
It exposes a quiet, universal structure: the act of filtering by significance is itself what turns an unbiased estimator into a biased one. Mathematically it is just a conditioning — truncate a distribution to the right of a threshold and its mean rises; no one has to behave badly. The beauty is that the same operator surfaces everywhere: survivorship bias in investing, selection effects in evolution, the performance drop after a hiring screen, picking models on a validation set. Wherever there is selection, there is a curse.
In 2015 the Open Science Collaboration repeated 100 psychology studies: only about a third replicated, and the effect sizes averaged half of what was originally reported — exactly what Type M predicts. Early small-sample drug trials routinely shrink in phase III; "the variant that won the experiment underdelivers after launch" is the everyday industrial version. SOTA gains on ML leaderboards are inflated for the same reason: what gets reported is always the best of many attempts.
The two camps do not give different answers; they answer different questions. The frequentist treats the parameter as a fixed unknown constant and the data as random: a 95% confidence interval promises that "if this procedure were repeated indefinitely, 95% of the intervals would cover the truth" — a long-run property of the procedure, saying nothing about the one interval in your hand.
The Bayesian treats the data as settled fact and the parameter as random: a 95% credible interval says "given the data I have seen, the truth lies here with probability 95%" — a belief right now. Intuition has always wanted the second; the standard toolkit delivers the first. That mismatch is the root of a century of misread p-values.
$\theta$ is the parameter you want to infer, $D$ the observed data. $p(\theta)$ is the prior — how large you believed $\theta$ might be before looking; $p(D\mid\theta)$ is the likelihood — how probable this data would be if $\theta$ took that value; multiply and normalise to get the posterior. The prior is often dismissed as subjective contamination, yet what it carries is exactly the information Concept 1 was missing: how large effects of this kind usually are, and how many hypotheses of this kind are true.
The Bayesian view collapses the first three traps into one hole — the absence of a prior over base rates and effect sizes. The hierarchical-model repair is startlingly elegant: give the group-level parameters a shared prior and the posterior produces shrinkage automatically, pulling each group's estimate toward the overall mean, pulling hardest on the noisiest. Multiple comparison needs no patch; the winner's curse is discounted on its own. As Gelman puts it: what multiple-comparison corrections approximate, multilevel models simply do. One mechanism, three diseases cured — because it is not a patch but an honest statement of how the data were generated. The frequentist beauty is just as real: no prior required, and an auditable worst-case long-run error rate.
A/B platforms increasingly report "probability B beats A" and expected loss directly, far closer to the language of decisions than a p-value; Thompson sampling allocates traffic by posterior probability. The most concrete counterpart in machine learning is regularisation as a prior: an $L_2$ penalty is a Gaussian prior on the weights, $L_1$ a Laplace prior, and the optimum you train to is the MAP estimate. Shrinkage for cold-start items in recommenders and credibility estimates in actuarial work are hierarchical priors doing the same job.