Day 33 · 2026.07.25

Coding & Error Correction

How a message arrives intact through a world that is guaranteed to corrupt it
"Can we communicate with arbitrary reliability over a channel that makes mistakes?" In 1948, Shannon's answer was: yes — and there is an exact price.

Channel Capacity & the Shannon Limit

A cliff edge in a noisy world
Information Theory
Intuition

Channels make errors, and the crudest defence is repetition: send each bit three times and take a majority vote. The error rate falls, but the rate collapses to 1/3; want more reliability and you send it five or seven times — so "almost never wrong" seems to demand that the rate go to zero.

Shannon said no. Every channel has a definite number $C$ (its capacity): so long as your rate $R<C$, there exists a code driving the error rate arbitrarily close to 0 with no loss of rate; once $R>C$, no code however clever can succeed. Fast and accurate are not a seesaw — they are separated by a cliff. The trick is not to armour bits one at a time but to encode in blocks: map a long message as a whole to a single point in a high-dimensional space. If the points sit far enough apart, noise can shove one around and it still lies closest to where it started.

rate R error R = C arbitrarily reliable doomed 0
Capacity is a phase-transition threshold: left of the cliff error can be pushed to zero, right of it no code works.
$$C=\max_{p(x)} I(X;Y)\qquad\text{binary symmetric channel:}\;C=1-H(p)$$
Formal definition

$I(X;Y)$ is the mutual information between input $X$ and output $Y$ — how much of your uncertainty about $X$ is removed by seeing $Y$ (in bits). Capacity $C$ is its maximum over all input distributions $p(x)$: with the channel used to its utmost, how many bits can one use carry. For a channel that flips each bit with probability $p$, $C=1-H(p)$, where $H(p)=-p\log_2 p-(1-p)\log_2(1-p)$ is the entropy of the noise itself — the share the noise eats is subtracted exactly from the full bandwidth of 1.

Why it is beautiful

The beauty is in the strangeness of the proof. Shannon constructed no good code at all; what he proved was that a code drawn at random from all codes already performs well on average, so good codes must exist — using randomness to prove that a deterministic object exists, the probabilistic method's most startling debut. The deeper beauty is the shape of the threshold: intuition expects a smooth trade-off curve, and mathematics hands back a sharp phase transition. Reliable communication turns out to be a phase transition, like water coming to a boil.

Applications

This limit is the origin of coordinates for the whole communications industry: every scheme is scored by "how many dB from Shannon". Turbo codes first came within 0.5 dB; LDPC codes now sit inside 5G data channels and SSD controllers. On the AI side, rate–distortion theory is the mathematical parent of variational autoencoders and the information-bottleneck view: training a representation is deciding which bits to keep under a capacity budget.

In one line: reliability need not be bought with rate — every channel has a fixed allowance, near-perfect inside it and doomed outside.
To ponder: Shannon proved good codes exist almost everywhere, yours for the grabbing; humans still needed forty-five years to build one that could actually be decoded. What other gaps between "exists" and "constructible" are that glaring?

Hamming Codes & Sphere Packing

Spreading points out on a hypercube
Algebraic Coding
Intuition

See each 7-bit string as a vertex of a 7-dimensional cube (128 of them); noise flipping one bit is a single step along an edge. Error correction becomes geometry: choose a set of vertices as "legal words", pairwise at least 3 steps apart — then the vertices within one step of each legal word form a small ball, the balls are disjoint, and any once-flipped string must land inside exactly one of them.

Hamming's construction is slicker still: of the 7 bits, 3 are parity checks, each covering a particular subset of positions. Write the pass/fail of the three checks as a three-bit binary number and that number is the index of the corrupted bitthe checks announce the street address of the error themselves.

distance ≥ 3 ⟹ radius-1 balls disjoint ⟹ single errors always corrected
An error-correcting code is sphere packing in a discrete space: codewords are centres, noise is the radius.
$$s=H\mathbf{r}\pmod 2,\qquad 2^{k}\cdot\!\!\sum_{i=0}^{t}\binom{n}{i}\le 2^{n}$$
Formal definition

An $[n,k,d]$ linear code encodes $k$ bits into $n$, any two codewords differing in at least $d$ places (the minimum Hamming distance), so it corrects $t=\lfloor (d-1)/2\rfloor$ errors; Hamming(7,4) is $[7,4,3]$. The columns of the parity-check matrix $H$ are exactly the 7 non-zero three-bit vectors; given a received $\mathbf r$, compute the syndrome $s=H\mathbf r$: $s=0$ means clean, otherwise $s$ read as a binary number is the index of the faulty column. The Hamming bound then says: $2^k$ balls of $\sum_{i\le t}\binom{n}{i}$ points each cannot exceed the whole space.

Why it is beautiful

Put the numbers in: $2^4\times(1+7)=128=2^7$. The inequality is an equality — 128 vertices tiled seamlessly by 16 balls, none missed, none overlapping. This is a perfect code: the discrete counterpart of Kepler's packing problem, where the answer is not "about 74%" but exactly 100%. Such gems are vanishingly rare: in the binary case only the Hamming family and two Golay codes exist, proved exhaustive in 1973.

Applications

In 1947 Hamming's weekend batch jobs kept being discarded by the Bell Labs machine over a single wrong bit, and he demanded in exasperation: "If the machine can find the error, why can it not fix it?" — the whole discipline grew from that question. Today's server ECC memory runs SECDED (correct one bit, detect two), and every bit flip a cosmic ray induces in DRAM is repaired silently right there.

In one line: correction means spreading points far apart in a discrete space — and Hamming codes pack the balls so they tile it exactly, without a single gap.
To ponder: perfect codes are rare because the ball volume $\sum\binom{n}{i}$ seldom divides $2^n$; when exact tiling is impossible, waste must be accepted. Is "approaching the optimum but never reaching it" peculiar to coding, or the normal condition of optimisation?

Reed–Solomon Codes

Two points fix a line; send a few more and nothing is lost
Finite Fields
Intuition

Two points determine a line. If the message I want to send is the two coefficients of a line, I do not send the coefficients — I sample 5 points on the line and send those. Lose three and it hardly matters: any surviving two reconstruct the line. Redundancy is not repeating the message; it is oversampling it.

In general: treat $k$ data symbols as the coefficients of a degree-$(k-1)$ polynomial, evaluate it at $n$ points, and send the values; any $k$ of them interpolate the original back. If a value is altered rather than lost, it strays off the curve and stands out among its collinear neighbours — which is why correcting an error costs twice what filling a gap costs: you must spend redundancy to find who is lying, then more to set it right.

altered → off the curve k data points n−k redundant points
The message is a polynomial and what travels are its values at chosen points: lost points interpolate back, corrupted ones betray themselves by leaving the curve.
$$m(x)=m_0+m_1x+\cdots+m_{k-1}x^{k-1}\;\longmapsto\;\big(m(\alpha^0),\dots,m(\alpha^{n-1})\big)$$
Formal definition

Work in the finite field $\mathrm{GF}(2^8)$ — a "world of bytes" with exactly 256 elements and full arithmetic, generated by $\alpha$. The $k$ message bytes are the coefficients of $m(x)$, evaluated at $n\le 255$ points to give the codeword. The parameters are $[n,k,n-k+1]$: the minimum distance is exactly $n-k+1$, so it corrects $t=\lfloor (n-k)/2\rfloor$ corrupted symbols, or fills $n-k$ losses whose positions are known.

Why it is beautiful

That $n-k+1$ was not cleverly engineered; it is the direct consequence of an elementary algebraic fact: two distinct degree-$(k-1)$ polynomials agree at no more than $k-1$ points, so two codewords differ in at least $n-k+1$ positions — and the Singleton bound says no code can exceed that, so Reed–Solomon sits flush against the ceiling. Lovelier still is the change of viewpoint: Hamming preserves distance by placing points geometrically, Reed–Solomon by algebra. Change the lens and optimality stops being the reward of a careful search and becomes an incidental corollary of a theorem.

Applications

A scratch on a CD ruins thousands of consecutive bits, but RS works in bytes, so a wholly destroyed byte counts as a single error — add interleaving to scatter the burst across code blocks and a 2.4 mm scratch still plays through. Closer to home is erasure coding in distributed storage: HDFS, Ceph and S3 replace three-way replication with RS(10,4), tolerating any 4 simultaneous disk failures while storage overhead drops from 200% to 40%. The cost curve of cloud storage was pushed down by a polynomial.

In one line: view the message as a polynomial and redundancy as extra sample points, and optimal distance no longer has to be designed — it follows from a theorem.
To ponder: erasure coding buys four-disk tolerance for 40% extra space where replication demands 200%. Why then does industry still replicate hot data so heavily? (Hint: when one disk dies, how much must be read to repair it?)

QR Codes: Theory Landing on Paper

A public performance of abstract algebra
Applied Coding
Intuition

A QR code is the three previous ideas fused into engineering. The large squares at three corners are finder patterns — the phone locates them first and thereby learns orientation and perspective, so tilted or upside-down scans still read. Data fills the grid along a zigzag path under a layer of Reed–Solomon parity; four correction levels exist, the highest (H) carrying about 30% redundancy — which is precisely why a logo stamped over the centre still scans: the covered region is, mathematically, a scratch.

One further step is easily overlooked and rather elegant: masking. The data may by chance produce broad black or white regions, or accidentally mimic a finder pattern. So the encoder XORs the data against each of 8 fixed patterns and keeps whichever balances black and white best — a reversible layer of pseudo-randomness to break up unlucky regularity.

◼ finder → orientation & perspective ▨ data + RS parity over GF(256) ⊕ mask → break up flat regions ↯ interleave → spread bursts
One QR code houses finite-field arithmetic, the geometry of correction, and pseudo-randomisation at once.
$$g(x)=\prod_{i=0}^{2t-1}\!\left(x-\alpha^{i}\right),\qquad c(x)=x^{2t}m(x)-\big[x^{2t}m(x)\bmod g(x)\big]$$
Formal definition

Practical RS encoding uses this systematic form: shift the message polynomial $m(x)$ up by $2t$ and divide by the generator polynomial $g(x)$, appending the remainder as parity symbols. The codeword $c(x)$ is then divisible by $g(x)$ — equivalently, it vanishes at $\alpha^0,\dots,\alpha^{2t-1}$. To decode, evaluate the received polynomial at those points: a non-zero result means an error, and the system of equations woven from those non-zero values solves for where the errors are and by how much.

Why it is beautiful

The beauty is the distance the abstraction travelled. Finite fields were dreamt up by Galois before he turned twenty, purely to answer why the quintic has no radical solution; two centuries later they run billions of times a day in front of your phone's camera — a tool built to understand impossibility, finally deployed against coffee stains.

There is structural beauty too: the three defences are orthogonal — finder patterns, RS, and masking each guard a distinct failure dimension without interfering. Good engineering is isomorphic to good mathematics: split a coupled problem into independent ones.

Applications

Mobile payments, boarding passes and supply-chain tracking all ride on this; industry's Data Matrix codes use the same RS core, laser-etched into metal parts and still readable after cutting fluid and heat. Masahiro Hara invented the format in 1994 to track car components, and Denso's decision to waive patent royalties is what made it universal.

In one line: a QR code prints Galois's abstract algebra onto paper — an invisible layer of polynomial bought in exchange for an image that survives being defaced.
To ponder: the three layers each guard a different failure mode. Look at a system you know well — is its redundancy layered orthogonally, or are several layers guarding the same thing while the real failure mode goes unwatched?

Going Deeper

Is the genetic code an error-correcting code?
Not strictly, but it was certainly shaped by noise. The 64 codons map onto 20 amino acids with the redundancy concentrated in the third position — a mutation there usually leaves the amino acid unchanged; and similar codons tend to encode chemically similar amino acids, so even a missense mutation does limited damage. This is not detect-and-correct but "make errors cost less". Evolution has no decoder; all it can do is align semantic distance between codewords with mutational distance — exactly the property one wants from an embedding space.
If erasure coding halves the storage bill, why is hot data still replicated?
Because storage is not the only cost — repair bandwidth is the silent killer. When a replicated disk dies, you copy an equal amount from another replica; when an RS(10,4) disk dies, you must read all ten surviving blocks to recompute the lost one. Hence locally repairable codes (LRC): a few groups of local parity alongside the global parity, so the common single-disk failure only touches a handful of blocks. Storage overhead, fault tolerance and repair cost form a three-way trade-off that cannot be jointly optimised — much like CAP.
Can a neural network's representation be read as a code?
The analogy is sharper than it looks. Distributed representations are natively codes: meaning is spread across the directions of a high-dimensional vector, and perturbing a few dimensions does not change the class — that is exactly large minimum distance. And adversarial examples say precisely that the distance is not large enough: in some directions an imperceptible nudge crosses the decision boundary. "Adversarial robustness" can thus be restated as "enlarge the minimum distance of the representation space", and defences such as randomised smoothing really do argue by sphere packing. The reversal is the interesting part: Shannon says higher dimension pushes random points further apart, yet deep learning finds more adversarial directions in high dimension. The same high-dimensional geometry is a blessing for reliable communication and a curse for robust classification — the difference being only whether the noise is random or chosen by an adversary.