What exactly is information? Shannon's 1948 answer is startlingly simple: information = uncertainty removed. "The sun will rise tomorrow" carries no information—you were already certain. "It will rain tomorrow" carries information, because it was genuinely uncertain.
A fair coin has two equally likely outcomes before the flip; the result removes that uncertainty—Shannon defines it as worth 1 bit. A die has six possibilities, is more uncertain, and its result is worth about 2.58 bits. Entropy measures exactly this: on average, how much "surprise" one observation brings you. So entropy is both "uncertainty" and "information"—uncertainty is the anticipation before the reveal, information the payoff after.
$p(x)$ is the probability of an event; $-\log_2 p(x)$ is its "surprise"—the smaller the probability, the larger it grows, the more unexpected the event. The $\sum p(x)\cdot$ in front is a probability-weighted average, so $H(X)$ is the "average surprise" (bits with $\log_2$, nats with $\ln$). Why a logarithm? Because information must be additive: two independent coin flips should carry $1+1=2$ bits, while probabilities multiply, $\frac12\cdot\frac12$—and the logarithm turns "multiply" into "add."
The entropy formula is nearly identical in form to thermodynamic entropy (Boltzmann's $S=k\ln W$)—no coincidence. Physical entropy measures "the disorder of microstates," information entropy "the uncertainty of a distribution"; deep down they are the same object. Von Neumann reportedly told Shannon to call it "entropy," "because nobody really knows what entropy is, so you'll always win the argument." Even more beautiful: entropy sets the theoretical limit of lossless compression—no code can have an average length shorter than $H(X)$ bits. "Information" thus went from a fuzzy philosophical notion to a measurable quantity with a hard lower bound.
The core of lossless compression (ZIP, PNG, FLAC) is Huffman coding: short codes for frequent symbols, long codes for rare ones, approaching the entropy limit. Decision trees (random forests, XGBoost) use information gain—the drop in entropy—to choose which feature to split on. Even linguistics: each English letter carries about 1.1 bits of entropy, far below $\log_2 26\approx4.7$—and it is exactly this redundancy that lets us read sentences with missing letters.
You want to know whether there'll be traffic tomorrow ($Y$). Someone tells you whether today is Friday ($X$). Once you know $X$, how much does your uncertainty about $Y$ drop? That drop is the mutual information between $X$ and $Y$.
It measures how much information two variables share. If $X$ and $Y$ are fully independent, knowing one tells you nothing about the other, and mutual information is 0. If $Y$ is fully determined by $X$, knowing $X$ is knowing $Y$, and mutual information equals the entire entropy of $Y$. It is the information-theoretic measure of "association"—far more powerful than the correlation coefficient of statistics.
$H(Y)$ is $Y$'s own uncertainty; $H(Y\mid X)$ is "$Y$'s remaining uncertainty once you know $X$" (conditional entropy). Their difference is exactly "the part of $Y$'s uncertainty that $X$ removes for you." The second equality is more symmetric: $H(X)+H(Y)$ minus the joint entropy $H(X,Y)$ leaves the overlap—like the intersection of two circles. This immediately yields a lovely property: $I(X;Y)=I(Y;X)$, mutual information is symmetric—how much X tells you about Y equals how much Y tells you about X.
The Pearson correlation captures only linear relationships. If $Y=X^2$ with $X\in[-1,1]$, the correlation is 0—as if there were no relation—even though $Y$ is completely determined by $X$. Mutual information has no such limit: linear, nonlinear, periodic dependence—it catches them all; it is positive whenever any statistical association exists, and is 0 if and only if the two are strictly independent. It is a "complete detector of independence"—indifferent to the shape of the relation, caring only "how much does knowing one reduce the other's uncertainty."
Feature selection picks features with the highest mutual information with the label. Neuroscience uses it to measure how much a neuron's firing encodes about an external stimulus (a core tool of neural coding). Representation learning: self-supervised methods (InfoNCE, contrastive learning like SimCLR) are essentially maximizing the mutual information between different views of the same image, forcing the model to learn invariant semantics. Medical image registration maximizes the mutual information between CT/MRI modalities to align them.
Shouting across a noisy bar, you always risk being misheard. A communication channel is the same: wires have thermal noise, fiber has attenuation. The naïve thought is—since there's noise, transmission must err, faster means more errors, reliable communication is impossible.
Shannon overturned this in 1948. He proved: every channel has a definite "capacity" $C$, and as long as your transmission rate is below $C$, there exists a code that drives the error rate arbitrarily close to 0—even in a channel full of noise. Just as in the bar, repeating keywords and using predictable phrasing lets you convey a message almost error-free through the noise. Noise doesn't make communication "unreliable"; it sets a speed limit on reliable communication: above it errors are unavoidable, below it near-perfect transmission is always possible.
The left side is the definition: capacity = the maximum mutual information between input $X$ and output $Y$ over all input distributions—i.e. the most bits one use of the channel can reliably carry. The right side is the most famous special case (the Shannon–Hartley theorem): $B$ is bandwidth, $S/N$ the signal-to-noise ratio. Capacity grows linearly with bandwidth but only logarithmically with SNR—to double the rate, adding bandwidth beats adding power. This one formula sets the speed ceiling for everything from WiFi to deep-space probes.
Shannon's noisy-channel coding theorem is an "existence miracle": he never told you how to build the near-perfect code, only proved it must exist. The method is exquisite—he "picks a random codebook" and proves that on average a random code is already good enough, so at least one good code must exist. This is a classic triumph of the probabilistic method: inferring "an individual exists" from "the average is good enough." Humanity took nearly half a century (Turbo codes, LDPC, Polar codes) to build practical codes that truly approach the Shannon limit—chasing a target proven to exist back in 1948.
5G / WiFi 6 use adaptive modulation: when the channel is good, switch to higher-order constellations for more speed; when poor, fall back to robust modes—running the Shannon formula directly. Voyager, billions of kilometers away with an extremely weak signal, still sends back images thanks to capacity-approaching error-correcting codes. Hard drives and QR codes embed codes like Reed–Solomon, so a scratched disc or a smudged QR code still reads. Information theory unifies "transmission" and "storage" into one problem: reliably moving bits over a noisy channel.
Training a classifier ("is this a cat or a dog?") is essentially making the model's output probability distribution $q$ approximate the true distribution $p$. How to measure how well it approximates? Information theory's answer: if you use the optimal code designed for $q$ to encode data that truly follows $p$, how many bits do you waste? Less waste means $q$ is closer to $p$.
That "waste" has two names: the full encoding cost is cross-entropy $H(p,q)$; the part "overpaid for using the wrong distribution" is the KL divergence $D_{KL}(p\Vert q)$, which is always $\ge 0$ and equals 0 iff $q=p$—a "distance" between distributions (though asymmetric). The "cross-entropy loss" literally means minimizing this waste.
Cross-entropy $H(p,q)=-\sum_x p(x)\log q(x)$ is the average code length when the model distribution $q$ encodes real data $p$. It splits in two: $H(p)$ is the data's intrinsic entropy (independent of the model, a constant), and $D_{KL}(p\Vert q)$ is the model's "error tax." Since $H(p)$ is fixed, minimizing cross-entropy = minimizing KL divergence = making the model distribution hug the true one. One layer deeper: "minimizing cross-entropy" can be shown to be exactly equivalent to statistics' "maximum likelihood estimation"—two roads from different disciplines pointing at the same loss function.
Why do classifiers and language models worldwide all use cross-entropy? Because it simultaneously satisfies three identities—"information-theoretic optimal coding," "statistical maximum likelihood," and "Bayesian posterior"; it is not chosen arbitrarily. Training GPT to predict the next word is minimizing cross-entropy; a language model's key metric, perplexity, is exactly $2^{H}$, read as "how many words the model hesitates among per step on average." Information theory gives deep learning something it otherwise lacked: a physical meaning for the loss number.
Nearly all classification tasks use cross-entropy loss; large-language-model pretraining = minimizing the cross-entropy of next-word prediction. VAEs and diffusion models include a KL term in their ELBO training objective, forcing the latent distribution toward a prior; knowledge distillation makes a small model's output distribution approach a large model's via KL—information theory becomes the language of "compressing a neural network." Cross-disciplinarily, this echoes neuroscience's predictive coding theory—the brain, too, may be minimizing some divergence between "prediction" and "sensory input."