A deep network is hundreds of nested functions: $f_n(\cdots f_2(f_1(x)))$. Training must answer one question — "if I nudge one of these billions of knobs a hair, by how much does the final loss change?" — i.e. the partial derivative of the loss with respect to each weight.
The naive way is to test each knob separately: billions of forward passes, astronomical. Backprop's insight: run forward once, cache each layer's intermediate result, then propagate the "blame" backward layer by layer, starting from the loss. Each layer needs only two things — its own local derivative, and the gradient handed down from above — and their product is that layer's contribution, while the gradient passes on to the next layer. Like assigning blame down a chain of command: ask each link "how much did you amplify the error?", one pass, everyone accounted for, nothing recomputed.
This is the chain rule: $L$ is the loss, $a_\ell$ the activation of layer $\ell$, $w_\ell$ its weights. The product is really a string of Jacobian matrices multiplied together. Everything hinges on the order: right to left (reverse mode) — compute the "loss-over-output" vector at the left end first, then left-multiply step by step — every step is just "vector × matrix," never building a big matrix explicitly. So the gradient of all parameters costs about one forward pass.
The same product of terms, run in the opposite direction, collapses the cost from "number of parameters × forward" to "one forward." This is the heart of reverse-mode automatic differentiation — mathematically, the "adjoint." It's no black magic but an older duality: to get one output's sensitivity to many inputs, walk backward. Exactly this $O(n)$-instead-of-$O(n^2)$ gap turns training billions of parameters from "impossible" into "an overnight job."
Every trained neural network relies on it — PyTorch's and JAX's autograd automatically differentiates any computation graph in reverse. The same idea is just as sharp outside nets: the adjoint method in weather and fluid inversion, Pontryagin's principle in optimal control, differentiable rendering pushing gradients through a graphics pipeline, even reasoning backward from a gradient to a molecular structure. Wherever you have "many inputs, one scalar objective," backprop is the optimal solver.
Read "the kitten chased its tail because it was bored." To know what "it" refers to, you scan back over the text, find the most relevant word, and pull its meaning in. Attention makes this mathematical: each word sends out a query (Q), every other word holds up a key (K) answering "am I relevant to you?", and the more relevant ones hand over more of their value (V).
So attention is a soft dictionary: an ordinary dictionary retrieves by an exact key, attention gives every key a continuous weight by similarity and takes a weighted average. And it is differentiable — how well it retrieves can be optimized by the gradient — so "where to look" goes from hand-designed to something the network learns itself.
$Q,K,V$ are three linear projections of the same batch of word vectors (query, key, value). $QK^{\top}$ uses the dot product to measure each query's similarity to each key, giving a table of scores; dividing by $\sqrt{d}$ keeps large dimensions from blowing up the dot products and saturating the softmax; softmax squashes each row of scores into weights summing to 1; multiplying by $V$ is "mix all the words' contents, weighted by relevance."
It reduces the fuzzy cognitive act of "retrieving memory by content" to two matrix multiplications and a softmax — clean, differentiable, and naturally suited to GPU parallelism. Where an RNN must go word by word and lets information decay down a long chain, attention lets any two words reach each other in one step, a constant distance no matter how far apart. This operator, writable in a few lines, holds up the entire era of Transformers and large language models.
GPT and BERT model language with it, the Vision Transformer (ViT) treats image patches as "words," AlphaFold uses it to capture long-range pairings between amino acids and infer protein folding, and diffusion models use cross-attention to let text steer image generation. "Which elements should influence each other" is answered, across nearly every modality, by the same formula.
A single neuron does something simple: weight and sum the inputs, then pass through a nonlinearity — the output approximates a soft step or a little bump. One bump is dumb, but stack many bumps of varying height, width, and position and you can carve out any undulating curve — like Lego bricks approximating a mountain ridge, or Fourier stacking sine waves into a square wave.
The universal approximation theorem says exactly this: with enough hidden neurons, a network can approximate any continuous function to any precision you want. A neural network learning "anything" is no mysticism — it is essentially a tunable function-Lego machine, and learning is adjusting each brick's height and position.
Cybenko (1989) and Hornik's theorem: finite sums of the form $\sum_{i} c_i\,\sigma(w_i\!\cdot\! x+b_i)$ ($\sigma$ a nonlinearity like a sigmoid) are dense in the space of continuous functions on a compact set — they can approximate any continuous $f$ arbitrarily well. But the crux: the theorem only guarantees that such parameters exist; it says nothing about how many neurons (possibly exponentially many), and nothing about whether gradient descent can find them.
It reconnects the fashionable neural network to a three-century classical lineage of approximation — Weierstrass, Fourier — the network is just another set of basis functions. Better still, it separates three things often conflated: can it represent (the theorem — yes), can it learn (optimization), can it generalize (statistics). Deep learning's real mystery is not the first but this: why does an over-parameterized network with far more parameters than data, trained by plain SGD, nonetheless find solutions that generalize well?
It is the license to use a network as a "universal function fitter": if a problem can be written as an "input→output" function, there's reason to try. The same logic explains why deep often beats wide — some functions need exponentially many neurons in a shallow net but only polynomially many when made deeper; depth buys efficiency through composition.
A $100\times100$ grayscale image lives in a $10000$-dimensional pixel space. But scatter a handful of random pixels and you will never land on a human face — real images occupy only a thin, curved sheet within that vast space, of nearly zero measure. The manifold hypothesis says: real high-dimensional data (images, speech, text) does not fill the whole space but concentrates on a much lower-dimensional curved surface (a manifold).
An analogy: a ribbon crumpled into a room is intrinsically one-dimensional (walking along it there's only "forward/back"), yet embedded in three dimensions. Learning is flattening that crumpled sheet and finding its intrinsic coordinates — grasp the sheet's shape, and a ten-thousand-dimensional problem collapses into a few-dozen-dimensional one.
Assume high-dimensional data lies approximately near a low-dimensional manifold $\mathcal{M}\subset\mathbb{R}^D$ whose intrinsic dimension $\dim\mathcal{M}=d\ll D$. Representation learning is finding a set of coordinates (a chart / embedding) for that manifold, compressing the $D$-dimensional observation into a $d$-dimensional latent vector with almost no loss.
It resolves the paradox of the curse of dimensionality: in theory learning in $10000$ dimensions needs an astronomical number of samples, yet in practice a few million images suffice — because the effective dimension is low. It ushers differential geometry and topology into deep learning: smoothly interpolating between two points in latent space corresponds to "walking along the sheet" of the manifold, so a face can shift continuously from a smile to a frown without passing through a field of snow. The shape of the data is itself a form of prior knowledge.
Autoencoders compress data to low dimensions and reconstruct it; diffusion models can be read as learning to "push noise that has strayed off the manifold back onto it," step by step generating realistic images from pure noise; t-SNE and UMAP spread a high-dimensional manifold onto two dimensions for the eye; latent-space interpolation and image editing all rest on the geometric intuition of "moving on the manifold."