Day 37 · 2026.07.29

The Mathematics of Randomness

At what scale does disorder generate order?
"Chance is a more fundamental conception than causality." — Max Born, Natural Philosophy of Cause and Chance

Random Walk

Random Walk · randomness produces diffusion, not cancellation
Probability
Intuition

A drunkard stands on a long street and flips a coin at every step to go left or right. Where is he after a thousand steps? The instinct that says the pluses and minuses cancel, so he is still near the start, is half right and half badly wrong — his expected position really is 0, but what cancels is direction, not distance.

The real answer: after $n$ steps his typical distance from the origin is about $\sqrt n$ steps — roughly 10 after 100 steps, roughly 100 after 10000. Not $n$ (that would be walking straight), not 0 (that would be cancellation), but the square root. Randomness does not pin you in place; it pushes you away at a rate slower than linear. That is diffusion.

Dimension changes the ending too: a walk on a line or a plane returns to its starting point with probability 1, while in three dimensions the chance is only about 34% (Pólya's theorem). The drunkard always finds his way home; the drunken bird never finds its nest.

±√n envelope steps n →
Formal definition
$$S_n=\sum_{i=1}^{n}X_i,\qquad \mathbb{E}[S_n]=0,\qquad \mathbb{E}[S_n^2]=n$$

$X_i$ are i.i.d. single steps, equal to $+1$ or $-1$ with equal probability; $S_n$ is the position after $n$ steps. Expand the square: $\mathbb{E}[S_n^2]=\sum_i\mathbb{E}[X_i^2]+\sum_{i\ne j}\mathbb{E}[X_iX_j]$. Each term of the first sum equals 1 and there are $n$ of them; every term of the second vanishes, because the steps are independent with mean zero. Hence the typical displacement is $\sqrt{\mathbb{E}[S_n^2]}=\sqrt n$.

Why it's beautiful

Why do the cross terms vanish? Because in the space whose inner product is the expectation of a product, independence (with mean zero) simply is orthogonality. Variances add, and so $\sqrt n$ turns out to be the Pythagorean theorem: lay $n$ mutually perpendicular unit vectors end to end and the resultant has length $\sqrt n$. Independent and perpendicular are the same statement — $\sqrt n$ is Pythagoras echoing inside probability space. Pólya's dimensional threshold is likewise mere arithmetic: the return probability hinges on whether $\sum_n n^{-d/2}$ diverges, and $d=2$ is exactly the dividing line.

Applications

In 1905 Einstein used $\langle x^2\rangle=2Dt$ to explain the motion of pollen grains, and from it computed Avogadro's number for the first time — the $\sqrt n$ law became experimental evidence that atoms exist. PageRank is the stationary distribution of a random walk on the web graph; MCMC uses one to explore a posterior; node2vec builds node embeddings from walks on graphs. SGD is a random walk with drift: the gradient is the drift, while minibatch noise accumulates as $\sqrt{\text{steps}}$ — which is where the rule "scale the learning rate with the square root of batch size" comes from.

One line · a question
Randomness does not return you to zero; it carries you away at speed $\sqrt n$.
If single steps follow a heavy-tailed law with occasional enormous jumps (a Lévy flight), does the $\sqrt n$ rule survive? And why might the long jumps seen in animal foraging — and in attention mechanisms — beat steady diffusion?

Martingale

Martingale · the exact definition of a fair game
Stochastic Process
Intuition

Let $X_n$ be your bankroll after round $n$ of some game. A martingale has exactly one clause: lay the entire past on the table, and the best prediction of the next bankroll is still whatever it is right now.

It does not say the value stays put — the swings can be violent enough to ruin you. It says the trend is zero: the history contains no extractable directional information. That is the precise content of the word fair.

This settles the doubling system (double after a loss; one win nets you a dollar). It does almost surely win that dollar — at the price of an astronomical loss on the vanishingly unlikely branch of consecutive losses, and the two balance exactly. The optional stopping theorem promotes that remark into a theorem: as long as you cannot foresee the future and your stakes are bounded, no stopping rule can change the expectation.

Formal definition
$$\mathbb{E}\!\left[X_{n+1}\mid\mathcal{F}_n\right]=X_n$$

$\mathcal{F}_n$ formalises everything known up to step $n$ — a $\sigma$-algebra, which is where the measure theory of Day 21 pays off; $\mathbb{E}[\,\cdot\mid\mathcal{F}_n]$ is the best prediction given that information. Replace the equality by $\le$ and you get a supermartingale: once the house takes its cut, your bankroll is exactly that.

Why it's beautiful

A martingale compresses the value judgement "fair" into a single equation, which is what makes it possible to prove theorems about impossibility. One layer deeper: any reasonable process decomposes uniquely into a predictable drift plus a martingale (the Doob decomposition) — the martingale is the pure, unpredictable component of randomness. It is also probability's conservation law: on Day 18 symmetry produced conserved quantities; here the martingale property conserves expectation, and optional stopping is the statement of that law.

Applications

The foundational theorem of finance: no arbitrage ⟺ there exists a measure under which discounted prices form a martingale (the risk-neutral measure) — that is the skeleton of Black-Scholes. Azuma–Hoeffding gives concentration bounds for randomised algorithms and online learning. Anytime-valid inference in A/B testing (test martingales, e-values) lets you look at the data whenever you like without inflating false positives, curing exactly the peeking problem from Day 34. In reinforcement learning the TD error is a martingale difference sequence, which is the heart of the convergence proofs.

One line · a question
A martingale is the mathematical definition of "no strategy can exploit this".
If your information set is larger than the market's (inside knowledge), the same price process is no longer a martingale for you — the property belongs not to the process but to the relation between the process and whose information we mean. So is randomness objective, or relative to an observer?

Brownian Motion

Brownian Motion · randomness's fixed point under magnification
Stochastic Analysis
Intuition

Shrink the steps of a random walk and pack them ever more densely, approaching continuous time. The decisive question is: as the time step $\Delta t\to0$, how small should each spatial step be?

Choose $\Delta t$ and the variance collapses to zero, leaving a motionless straight line; choose a constant and the variance explodes. The only surviving scale is $\sqrt{\Delta t}$ — the mirror image of the $\sqrt n$ law.

The consequence is startling: the path is continuous everywhere and differentiable nowhere. The reason is self-similarity — magnify any small piece (time ×4, space ×2) and you see the same jaggedness, so it never straightens into a tangent the way a smooth curve does (a direct relative of the fractals of Day 17). This is also the source of "$dW$ squared is of the same order as $dt$": displacement squared scales like time, so displacement is of higher order in smallness than time itself.

magnified: the same jaggedness time t →
Formal definition
$$W_0=0,\qquad W_t-W_s\sim\mathcal{N}(0,\,t-s),\qquad (dW)^2=dt$$

The axioms: it starts at 0; increments over non-overlapping intervals are independent; each increment is normal with mean 0 and variance equal to the length of the interval; paths are continuous. Variance $=t-s$ is just the formal way of writing the $\sqrt{\Delta t}$ scale. And $(dW)^2=dt$ is the core of Itô calculus: second-order terms are no longer negligible, so the chain rule acquires an extra term $\tfrac12 f''\,dt$.

Why it's beautiful

Here the normal distribution is not an assumption but a forced conclusion: whether the single step is a coin, a die, or any law with finite variance, the rescaled limit converges to the same Brownian motion (Donsker's invariance principle). Microscopic detail is forgotten entirely and one object remains — this is mathematics' deepest species of beauty: universality. The second beauty is its equivalence with the heat equation (Feynman–Kac): probability and analysis translate into each other here.

Applications

Financial assets are modelled by geometric Brownian motion, and Black-Scholes follows from Itô's lemma; in physics it is the noise term of the Langevin equation. The forward process of a diffusion model is precisely the stepwise injection of Brownian noise, and the reverse denoiser learns the score — today's image and video generation rests on the mathematics of Einstein's 1905 paper. The continuous-time approximation of SGD (a stochastic differential equation) is used to explain flat minima and generalisation.

One line · a question
Brownian motion is randomness's fixed point under rescaling: the more you magnify it, the more it is itself.
A real pollen grain has mass and velocity, so its path is in fact differentiable. Is "nowhere differentiable" therefore wrong, or does it simply hold above some scale? Which other models are likewise true only after coarse-graining?

Random Matrices

Random Matrix Theory · in high dimension randomness turns predictable
Linear Algebra
Intuition

Take a $1000\times1000$ matrix, fill every entry independently with a random number (keeping it symmetric), and compute its thousand eigenvalues. You would expect a mess. The opposite happens: plotted as a histogram they fall almost perfectly under a semicircular curve; draw fresh random numbers and the shape is the same semicircle again.

More counterintuitive still is the repulsion between eigenvalues: independently scattered random points clump together, whereas eigenvalues spread out almost evenly, as though they carried like charges.

At the level of a single entry there is total disorder; at the level of the whole spectrum there is rigid order. Dimension does not pile randomness into a bigger mess — it irons it flat.

−2 0 +2 semicircle law ρ(x) = √(4−x²) / 2π normalized eigenvalues
Formal definition
$$\rho(x)=\frac{1}{2\pi}\sqrt{4-x^2},\qquad x\in[-2,2]$$

Wigner's semicircle law: for an $N\times N$ real symmetric matrix with independent entries of mean 0 and variance $\sigma^2$, divide the eigenvalues by $\sigma\sqrt N$ and their empirical distribution converges to the formula above as $N\to\infty$. That $\sqrt N$ scaling is the $\sqrt n$ law once more — multiplying by a vector adds $N$ random terms within a row, whose magnitude is exactly $\sqrt N$. The counterpart in data analysis is the Marchenko–Pastur law.

Why it's beautiful

This is the prettiest display of concentration in high dimension: the fluctuation of a macroscopic function of $N^2$ random numbers — the spectral distribution — dies away like $1/N$, and the random becomes determinate. The term for it is self-averaging. Then in 1972, over tea at Princeton, Montgomery and Dyson noticed that the spacings between zeros of the Riemann $\zeta$ function match the eigenvalue spacings of random matrices (the GUE) exactly — the deepest open problem in number theory shares a statistical law with quantum chaos, and half a century later nobody knows why.

Applications

PCA and covariance denoising: Marchenko–Pastur supplies the baseline for what the spectrum looks like if it is all noise, so only eigenvalues past the upper edge can be real signal (used for portfolios and gene expression). Initialising a neural network is in essence controlling the spectral radius of the weight matrices — the Xavier/He variance formulas exist so that signal neither explodes nor dies across depth; the spectrum of the Hessian explains the wildly unequal curvature along different directions of the training landscape. In engineering there are also MIMO channel capacity and the phase-transition thresholds of compressed sensing.

One line · a question
In high enough dimension, random no longer means unpredictable.
If the spectrum of a thousand-dimensional random matrix is essentially determined, how much of a huge network's behaviour is fixed by architecture and data, and how much by that particular draw of initial weights? "Would retraining with a new seed give a different model?" is really a question about how strong the self-averaging is.

Going deeper

Is randomness a property of the world, or a name for our ignorance?
Probability theory is only a branch of measure theory and presumes nothing about where randomness comes from, yet it offers several answers. Kolmogorov defines randomness as incompressibility — a sequence is random exactly when the shortest program describing it is no shorter than the sequence itself, a definition about structure rather than causation. Cryptography is more pragmatic: a pseudorandom generator is fully deterministic but indistinguishable to any polynomial-time observer, and that suffices. Quantum mechanics, meanwhile, ruled out local hidden variables through experimental violations of Bell's inequality. All three routes point the same way: randomness is always relative to some power of observation.
Why does higher dimension make things more predictable, not less?
Because the law of large numbers has a geometric form: concentration of measure. Almost all the area of a high-dimensional sphere is squeezed into a thin shell near its equator — equivalently, a well-behaved function in high dimension takes nearly the same value almost everywhere. The semicircle law, Marchenko–Pastur, and the Johnson–Lindenstrauss lemma are all faces of it. Hence an apparent paradox: the curse of dimensionality makes filling space with data hopeless while making estimates of macroscopic quantities remarkably stable. Deep learning sits on exactly that tension.
Why is $\sqrt n$ everywhere, and can it be broken?
The root is one sentence: independence is orthogonality, variances add, and the standard deviation grows like $\sqrt n$. Monte Carlo error $\propto 1/\sqrt N$ is the other side of the same coin — ten times the accuracy costs a hundred times the work, the tax every sampling method pays. Breaking it means giving up independence: quasi-Monte Carlo deliberately manufactures negative correlation with low-discrepancy sequences and pushes the error towards $1/N$. The other end is broken by heavy tails — with infinite variance the limit is not Brownian motion but a Lévy process, and $\sqrt n$ becomes $n^{1/\alpha}$. $\sqrt n$ is the joint fingerprint of independence plus finite variance.
Is one idea hiding behind all four concepts?
Yes, and the idea is scale. The random walk asks: add up many small random quantities — what survives? The answer is $\sqrt n$ and the normal shape, everything else forgotten. Brownian motion is that question pushed to its limit, a fixed point: magnify it as you like and the form does not change. The martingale cuts from the other side, stripping off predictable drift and leaving the purely random component. Random matrices swap the axis for dimension: as $N$ grows, individual randomness is ironed into a determinate collective curve. All four share the same motif — under aggregation and magnification, what disappears and what remains — which physics calls a universality class. The intuition carries straight over for anyone who builds distributed systems: a single machine's latency is unpredictable, yet tail-latency quantiles are stably measurable.