In 1738 Daniel Bernoulli posed a gamble: flip a coin until heads appears; if it first lands on flip $n$, you are paid $2^n$. The expected payoff is $\tfrac12{\cdot}2+\tfrac14{\cdot}4+\cdots=\infty$, yet nobody will pay a hundred for it. Nothing was miscalculated — the mistake is treating money as the goal itself. The first million moves you from renting to owning; the tenth adds a digit to a statement. Bernoulli's fix: first convert money into what it means to you — an internal scale called utility — and only then take the expectation. The curve flattens as it rises. It is concave.
von Neumann and Morgenstern (1944) went further than Bernoulli. Let $\succcurlyeq$ be your preference ordering over random outcomes (lotteries). If it satisfies four axioms — completeness, transitivity, continuity (between a better and a worse option there is always a probability mix exactly equivalent to the middle one), and independence (mixing the same third option into both sides leaves the ranking unchanged) — then a function $u$ must exist such that
Here $x_i$ are outcomes, $p_i$ their probabilities, and $u$ maps outcomes onto a real-valued scale.
Note the direction of the logic: nobody assumed you have a utility function. The premise is only that your choices are mutually consistent; the conclusion is that your behaviour must then look like maximizing some expectation. Such results are called representation theorems — the numbers are not inserted into the model, they are forced out of the order. And "risk aversion", a psychological word, collapses here into a single geometric statement: $u$ concave $\iff$ $E[u(X)]\le u(E[X])$. The chord always lies below the curve, and the gap is the insurance premium you are willing to pay.
An RLHF reward model does exactly this: humans supply only pairwise comparisons (A beats B), Bradley–Terry fits them into a single scalar score, and the policy maximizes its expectation — the vNM theorem is what licenses the step "a scalar score exists". Conversely, wherever the axioms break is where behavioural economics enters: in the Allais paradox people systematically violate independence, which is why prospect theory switched to reference points and probability weighting.
Variance is the standard measure of risk, but it has an embarrassment: it penalizes gains just as heavily, and nobody is afraid of earning too much. Hence VaR, which ruled risk management in the 1990s — the line such that on 95% of days your loss stays below it.
The trouble is that it reports only where the threshold sits and says nothing about what lies beyond: two portfolios can share a VaR while one loses slightly more past that point and the other goes to zero. Worse, bundling two assets each with a small VaR can produce a combined VaR larger than their sum — diversification apparently raising the reading. The yardstick is broken.
Artzner et al. (1999) asked the question backwards: what must a legitimate yardstick satisfy? Read $\rho(X)$ as "how much cash must be added before position $X$ is safe". The answer is four axioms: monotonicity, translation invariance $\rho(X+c)=\rho(X)-c$ (cash offsets exactly its own amount of risk), positive homogeneity, and subadditivity $\rho(X+Y)\le\rho(X)+\rho(Y)$. A measure satisfying all four is called coherent. VaR fails precisely on the last one, whereas conditional value at risk
— the mean loss inside the tail rather than the entrance to it — passes all four.
Subadditivity looks like a technical condition, but it is the exact encoding of the common-sense demand that diversification never increase risk. The lovely part is its consequence: every coherent risk measure can be written as $\rho(X)=\sup_{Q\in\mathcal Q}E_Q[-X]$ — the worst-case expected loss over a family of candidate probabilities $\mathcal Q$. That proves two things to be one: "I am risk averse" and "I am unsure which probability applies, so I plan for the worst one" are equivalent. Four plain axioms, one pass through convex duality, and robustness and pessimism become the same object.
In 2016 the Basel framework replaced VaR with ES (i.e. CVaR) as the regulatory market-risk metric, citing subadditivity; Rockafellar and Uryasev showed CVaR optimization reduces to a linear program, making it computable. The same $\sup_Q$ appears in machine learning as distributionally robust optimization: instead of minimizing average loss on the training distribution, minimize the worst loss over a family of distributions near it, which buys resistance to distribution shift.
You face a gamble with an edge, repeatable, and you choose what fraction of your bankroll to stake each round. Bet everything? One loss removes you permanently — ruin is an absorbing state, and no number of prior wins undoes it. Bet one percent? Safe, but too slow to matter. Somewhere in between lies an optimum.
The key is that wealth is multiplied, not added. Long-run performance of a multiplicative process is governed by the geometric mean, which is the arithmetic mean of $\log$. So what should be maximized is never the per-round expected return but the expected logarithmic growth rate.
Let the odds be $b{:}1$ (stake 1, win $b$), the win probability $p$, $q=1-p$, and $f$ the fraction of current wealth staked each round. After $n$ rounds $W_n=W_0(1+fb)^{k}(1-f)^{n-k}$ with $k$ wins. Take logs, divide by $n$, and the law of large numbers gives the long-run growth rate
Note that $\log$ is not an assumed taste in utility; it is what you get from turning multiplication into addition so the law of large numbers can apply.
Kelly derived this at Bell Labs in 1956, with Shannon down the hall, under the title A New Interpretation of Information Rate. For an even-money binary bet the optimal growth rate is $g^*=\log 2-H(p)$, where $H$ is Shannon entropy. The right-hand side is how much more you know than an ignorant bettor; the left-hand side is how fast your money can grow exponentially. What you can earn equals what you know beyond everyone else. A quantity invented to measure code length becomes, unmodified, the ceiling on wealth growth — information theory and gambling are one theorem read twice.
Thorp used it for blackjack and convertible-bond arbitrage, and carried Kelly into hedge-fund position sizing. In practice people stake only half-Kelly: as the figure shows, $g(f)$ is extremely flat near $f^*$ but plunges to the right, crossing zero at $2f^*$ — and $p$ is always an estimate. On a curve that is gentle to the left and steep to the right, estimation error should be allowed to fall leftwards.
A row of slot machines with different, unknown payoff distributions. Every pull does two things at once: it earns this round's money, and it buys a little information about which machine is better. Information cannot be bought separately — it arrives only as a by-product of betting, and that is what separates this from ordinary optimization. Always pull the currently best arm and you may never find the truly good machine; try them all in turn and most pulls are wasted on arms already known to be bad. The tension cannot be removed, only priced.
With $K$ arms, arm $a$ has expected payoff $\mu_a$ and the best is $\mu^*$. What measures an algorithm is not total reward but regret — the gap against knowing the answer from the start: $R_T=T\mu^*-E\big[\sum_{t=1}^{T}\mu_{a_t}\big]$. At each step UCB selects
where $\hat\mu_a$ is the current estimate (exploitation) and the square root is a confidence radius: the fewer the pulls ($n_a$ small) the wider it is, and it creeps upward with $t$. This puts a price tag on ignorance and lets long-unverified options resurface on their own. In the figure the arm selected is the one whose mean is not highest but whose upper bound is.
Lai and Robbins (1985) proved that no reasonable algorithm escapes $R_T=\Omega(\log T)$, and UCB attains $O(\log T)$ — the bounds meet in order. This means "optimism in the face of uncertainty" is not an engineering heuristic but a provably optimal principle.
More surprising still is Gittins (1979): under infinite-horizon discounting the optimal policy decomposes — compute for each machine an index depending only on its own history, and pull the largest. A $K$-dimensional coupled dynamic program collapses into $K$ one-dimensional problems. And Thompson had already proposed, in 1933, sampling an arm according to its posterior probability of being best; that too is optimal in order. The frequentist confidence bound and the Bayesian posterior sample arrive at the same frontier from opposite directions.
A/B testing uses it to keep tilting traffic toward the winner instead of waiting for the experiment to end; recommender cold start, adaptive allocation in clinical trials, and hyperparameter search (Hyperband) are bandits outright. The most famous instance places UCB at the nodes of a search tree: UCT and Monte Carlo tree search, the skeleton of AlphaGo. Preference-data collection for LLMs is the same shape — candidates are arms, and the labelling budget is the number of pulls.