Day 40 · 2026.08.01

Fourier Analysis

Change the basis, and a chaotic signal sorts itself into line
"Fourier's theorem is not only one of the most beautiful results of modern analysis, but it is said to furnish an indispensable instrument in the treatment of nearly every recondite question in modern physics." — Lord Kelvin

The Frequency Domain

One signal, seen through a different basis
Harmonic Analysis
Intuition First

The waveform of a recording looks like nothing at all. But wind it around a circle — travelling $\xi$ turns per second, with the height of the wave becoming distance from the centre — and something remarkable happens. When the winding rate matches a frequency genuinely present in the signal, the peaks keep landing on the same side and the figure's centre of mass drifts far off the origin. When it doesn't match, the peaks scatter evenly and the centre of mass stays put.

So "measuring a frequency" becomes pure geometry: how far the centre of mass travels is the amplitude, which way it points is the phase. The signal hasn't changed — only the basis describing it has.

Time domain: three sines summed, no visible structure Frequency domain: three lines, structure at a glance ξ₁ ξ₂ ξ₃ frequency amplitude
Formal Definition

Write "wind it up and take the centre of mass" as an integral and you have the Fourier transform:

$$\hat f(\xi)=\int_{-\infty}^{\infty} f(t)\,e^{-2\pi i\xi t}\,\mathrm{d}t$$

$e^{-2\pi i\xi t}$ is a point circling the unit circle $\xi$ times per second — the winding machine itself; multiplying by $f(t)$ treats the signal's value at time $t$ as a distance from the centre; the integral adds up every position to find the centre of mass. The result is complex: modulus is amplitude, argument is phase. This basis is privileged because it consists of eigenfunctions of the translation operator: shift by $\tau$ and the shape is untouched, merely multiplied by the constant $e^{2\pi i\xi\tau}$ — so every linear time-invariant system is diagonalized here.

Why It Is Beautiful

Diagonalization turns the hardest operation into the easiest. Convolution — that irritating sliding sum — degenerates into pointwise multiplication:

$$\widehat{f*g}=\hat f\cdot\hat g$$

Differentiation degenerates too: $\widehat{f'}(\xi)=2\pi i\xi\,\hat f(\xi)$. Taking a derivative becomes multiplying by a number, and a differential equation becomes an algebraic one. This is exactly how Fourier cracked the heat equation in 1807: each frequency decays independently as $e^{-4\pi^2\xi^2 kt}$, the high ones fastest — the mathematical reason heat smooths everything out. And the fight over convergence that his work provoked forced analysis to make "limit" rigorous: a trick for solving a physics problem rewrote the foundations of a field.

Applications

An MRI scanner does not photograph and then transform — gradient fields let it acquire Fourier coefficients directly (k-space), and the image is the inverse transform taken at the very end. In quantum mechanics the position and momentum representations are Fourier transforms of each other, and Heisenberg's uncertainty is the corollary that a function and its transform cannot both be narrow. In deep learning, implicit neural fields use Fourier feature encoding to break an MLP's native reluctance to fit high-frequency detail.

The Essence · A Question
The Fourier transform is not a computational trick but a move into the eigenbasis of linear time-invariant systems; hard problems get easy because the matrix becomes diagonal.
If a system is not time-invariant — attention weights that vary with position, say — what counts as its eigenbasis? Is there still a way to find it?

The Fast Fourier Transform

Symmetry cuts N² down to N log N
Algorithms
Intuition First

The discrete version needs $N$ frequencies, each sweeping $N$ samples — $N^2$ multiplications. At $N=10^6$ that is $10^{12}$ operations, past the edge of practicality.

The key observation: among those $N^2$ twiddle factors there are only $N$ distinct values, since $N$ steps around the circle return you to the start. Enormous redundancy is being recomputed for nothing. So the FFT divides and conquers: split the samples by even and odd index, run an $N/2$-point transform on each, and spend $O(N)$ stitching them back — $\log N$ times down to the bottom.

Divide and conquer: one rotation serves two outputs N-point DFT even terms E odd terms O Xₖ = E + ωO Xₖ₊ɴ⁄₂ = E − ωO the same ωO, added once and subtracted once — the butterfly
Formal Definition

Let $\omega_N=e^{-2\pi i/N}$ (the single rotation that cuts the circle into $N$ parts), and let $E_k$, $O_k$ be the $N/2$-point transforms of the even- and odd-indexed subsequences. Then

$$X_k=E_k+\omega_N^{\,k}O_k,\qquad X_{k+N/2}=E_k-\omega_N^{\,k}O_k$$

The whole secret lives in that minus sign: $\omega_N^{\,k+N/2}=-\omega_N^{\,k}$, half a turn more is a change of sign. So computing $\omega_N^{\,k}O_k$ once serves two outputs at once, halving the work on the spot. The recurrence $T(N)=2T(N/2)+O(N)$ gives $O(N\log N)$: at $N=10^6$ we drop to about $2\times 10^7$ operations, fifty thousand times faster.

Why It Is Beautiful

The speed-up is not an engineering optimization but a dividend of group theory. An $N$-point transform is harmonic analysis on the cyclic group $\mathbb{Z}/N\mathbb{Z}$; when $N=2^m$ that group has a full chain of subgroups, and each level down halves the problem. The fall in complexity corresponds directly to the decomposability of an algebraic structure — which is why an FFT of prime length needs an entirely different trick. Worth noting in passing: Gauss wrote this algorithm down in 1805, two years before Fourier published, and it slept until Cooley and Tukey rediscovered it in 1965.

Applications

The Wi-Fi and 5G you are connected to right now run OFDM: data is spread across thousands of orthogonal subcarriers, each end runs one FFT, and frequency-selective fading collapses into a single complex division per subcarrier. Large-integer multiplication uses "multiplication is convolution" to reach near-linear time via the FFT — infrastructure for computational number theory; the period-finding step of Shor's algorithm is likewise a quantum Fourier transform.

The Essence · A Question
The FFT's ten-thousand-fold speed-up comes not from faster machines but from noticing that a computation is mostly repetition — once seen, symmetry can be cashed in for time.
Backpropagation wins the same way, by sharing repeated intermediate gradients. Is that the same idea twice? What other $O(N^2)$ algorithms are waiting for their Cooley–Tukey?

Wavelets

Fourier knows which frequencies are there, not when
Time-Frequency
Intuition First

Fourier's basis functions are infinitely long sines. To represent one short drum hit you need a delicate cancellation among vast numbers of them — everything outside that instant must cancel away. The spectrum gets smeared across the board, yet says nothing about which second the hit occurred. A musical score, by contrast, has time on one axis and pitch on the other: what we actually want is local frequency information.

Wavelets change the building block to a short packet that oscillates a few times and decays to zero. Translating it asks "where"; dilating it asks "how fast". The point is that the dilation is not uniform — high frequencies get the narrow, fast version (sharp in time), low frequencies the wide, slow one (sharp in frequency). This is exactly the cochlea's strategy: ask fast events when, ask slow events what.

Short-time Fourier: uniform tiles Wavelets: wide tiles high up, tall tiles low down x: time y: frequency x: time y: frequency every tile has a minimum area — you can only redistribute it
Formal Definition

Take a mother wavelet $\psi$ with zero mean and fast decay; the whole family is generated by translation and dilation:

$$\psi_{a,b}(t)=\frac{1}{\sqrt{a}}\,\psi\!\left(\frac{t-b}{a}\right)$$

$b$ is the shift (which instant we are looking at), $a$ the scale (small = narrow and high-frequency, large = wide and low-frequency), and $1/\sqrt{a}$ keeps energies comparable across scales. Why can't we be narrow and frequency-precise at once? Because of a hard boundary:

$$\Delta t\cdot\Delta\xi\ \ge\ \frac{1}{4\pi}$$

$\Delta t$ and $\Delta\xi$ are the spreads in time and in frequency. This is the mathematical prototype of Heisenberg's uncertainty principle — literally the same theorem as in quantum mechanics, where $t$ and $\xi$ are called position and momentum instead. Every tile in the time-frequency plane has a minimum area: you cannot shrink it, only choose whether it is wide or tall.

Why It Is Beautiful

Wavelets deliver multiresolution analysis: in a nest of spaces $V_0\subset V_1\subset V_2\subset\cdots$, $V_j$ holds the approximation down to level $j$, and the difference between consecutive levels is spanned precisely by the wavelets — which is to say, by the detail newly added. A signal thus splits into "coarse outline plus successive detail", the order in which people look at things anyway.

Prettier still is the self-reference: a Daubechies wavelet has no closed form in elementary functions and can only be defined recursively by the scaling equation $\varphi(t)=\sqrt2\sum_k h_k\varphi(2t-k)$ — a function assembled from shrunken copies of itself, self-similar in the way fractals are, while in implementation nothing survives but convolutions with filter coefficients and downsampling.

Applications

When LIGO detected gravitational waves, it was a time-frequency plot that showed the chirp sweeping from 35 Hz to 250 Hz — a plain Fourier spectrum would have smeared it into broadband noise. In imaging, wavelet coefficients are naturally sparse (near zero over smooth regions, large only at edges), so thresholding alone denoises; JPEG2000 and digital cinema masters are built on this.

The Essence · A Question
The time-frequency budget is conserved; wavelets add no information, they only allocate resolution where it is worth spending — asking high frequencies when, and low frequencies what.
Shallow layers of a network see texture and deep layers see meaning: is that a learned multiresolution analysis?

Transform Coding & Sparsity

Compression is finding the basis in which a signal goes sparse
Information
Intuition First

In the pixel basis every pixel of a photograph carries information and there is almost nothing to spare. Move to a cosine basis and the energy collapses dramatically inward: the vast majority of coefficients are nearly zero — natural images are mostly gentle regions, with abrupt jumps only at edges.

Hence the three-step logic of compression: change basis so the signal is sparse → quantize the unimportant coefficients brutally or throw them away → entropy-code what remains losslessly. JPEG does exactly this: an 8×8 block DCT, division by a quantization table designed around human visual sensitivity, then entropy coding. What is discarded is not random information but the part the eye was never going to see.

Decay of sorted coefficients: the basis decides compressibility pixel basis: all of it matters cosine basis: a few coefficients carry the energy coefficient rank (largest first) magnitude
Formal Definition

The DCT is the real-valued cousin of the Fourier transform: extend the signal evenly before transforming and the block boundary no longer has an artificial jump, so no spurious high frequencies are manufactured. The only step that loses information is quantization:

$$\hat c_{uv}=\mathrm{round}\!\left(\frac{c_{uv}}{q_{uv}}\right)$$

$c_{uv}$ is the transform coefficient at position $(u,v)$ and $q_{uv}$ the matching step size in the quantization table — $q$ is large at high frequencies, so dividing and rounding sends a great many coefficients to zero; the "quality" slider in your software is scaling that whole table. As for how small a file can ever get, Shannon's rate-distortion function $R(D)$ sets the floor: the more distortion you allow, the lower the rate you need, and no encoder gets under that curve.

Why It Is Beautiful

"Which basis is best" has a definite answer: the KLT (that is, PCA), which decorrelates the coefficients completely, is optimal for any class of signal — but it depends on the signal's statistics, must transmit the basis itself, and has no fast algorithm. For a first-order Markov model of highly correlated neighbouring pixels, the DCT is an excellent asymptotic approximation to the KLT while also enjoying an FFT-class fast algorithm. Theoretical optimum and engineering feasibility meet exactly here, which is the real reason JPEG has ruled for thirty years.

Deeper still: sparsity can also cut back the sampling itself. Compressed sensing says that if a signal is $k$-sparse in some basis, roughly $O(k\log n)$ random linear measurements suffice to reconstruct it — far below the Nyquist rate. If nine tenths of what you sample is going to be thrown away, why sample it?

Applications

MP3 and AAC derive their quantization tables from psychoacoustic models — a loud tone masks nearby quiet ones, and whatever is masked is simply discarded. H.264/H.265 run an integer DCT over the residual after motion compensation, cutting video bandwidth by two orders of magnitude. Compressed sensing shortens MRI scans several-fold. Weight quantization in neural networks (INT8/INT4) is the same logic: find a representation in which the weights cluster, then shave bits where the metrics cannot notice.

The Essence · A Question
Compression is not "deleting some data" but "changing basis until the unimportant turns into zeros"; finding that basis is where all the intellectual work lives.
Is pretraining a large model a search for such a basis over the whole human corpus? And who wrote its quantization table — the one that decides what gets thrown away?

Going Deeper

Why are sines the "natural" basis? What if we used square waves?
Square waves also form an orthogonal basis (Walsh–Hadamard). The privilege of sines lies not in their shape but in their being eigenfunctions of the translation operator: physical law does not care where you set the origin of time, so the linear operators that describe the world are all diagonal in the sine basis, whereas a shifted square wave becomes a combination of other square waves. And if a system's symmetry were scaling rather than translation, its "sines" would be the power functions $t^s$ and the corresponding transform the Mellin transform. Symmetry chooses the basis, not us.
Why is the uncertainty principle the same theorem in mathematics and in quantum mechanics?
Mathematically it says only that a function and its Fourier transform cannot both be concentrated, and the proof uses nothing but Cauchy–Schwarz and integration by parts — no physics at all. Quantum mechanics adds exactly one physical assumption: the momentum representation is the Fourier transform of the position representation. Accept that, and Heisenberg's inequality becomes the physical translation of a mathematical theorem automatically. It is not the technical limitation "measurement disturbs the system" but an intrinsic feature of describing anything by waves.
Are compression, prediction and understanding the same thing?
Information theory gives an affirmative formalization: a good probabilistic model converts directly into a short code, so compression ratio is a measure of model quality; Kolmogorov complexity goes further, making the shortest generating program the best explanation of the data. Training a large model to predict the next token is mathematically equivalent to compressing the corpus losslessly. But compression only wants reproduction, whereas understanding also demands intervenability: knowing $P(\text{next word})$ is not knowing what change would cause what.