AI/ML Explained: Continual Learning & Catastrophic Forgetting

A neural net's weights are a piece of globally shared mutable state. Learning a new task B means an in-place update—directly overwriting that state. No version isolation, no transactions: the weights task A relies on get silently rewritten by B's gradients, and A is catastrophically forgotten. This isn't "memory is full"—it's a write-write conflict with no isolation mechanism.

Catastrophic forgetting means: when a network learns several tasks in sequence, learning the later ones sharply erases what it learned earlier. McCloskey & Cohen documented it back in 1989. The root cause: task A's knowledge is distributed across one specific setting of all the weights, while SGD on task B optimizes only B's loss—it neither knows nor cares which weights are A's "load-bearing walls"; it just pushes wherever lowers B's loss.

This gives the field's central tension—the stability-plasticity dilemma: weights too stable (can't change) → can't learn new tasks (no plasticity); weights too plastic (freely changed) → learning the new forgets the old. It's isomorphic to system trade-offs you know: a read-only replica is forever consistent but can't be written; a write-anywhere cache has high throughput but no consistency. Continual learning is about finding a compromise on this spectrum:

The next 3 concepts are three mainstream remedies on this spectrum: constrain weights (regularization), rehearse old data (replay), give each task dedicated parameters (isolation).

# "See" catastrophic forgetting: learn A, then B, then re-test A
import torch, torch.nn as nn
net = nn.Sequential(nn.Linear(20, 64), nn.ReLU(), nn.Linear(64, 2))
opt = torch.optim.Adam(net.parameters(), lr=1e-3)
loss_fn = nn.CrossEntropyLoss()

def make_task(seed):                       # two different binary tasks
    g = torch.Generator().manual_seed(seed)
    X = torch.randn(400, 20, generator=g)
    y = (X @ torch.randn(20, generator=g) > 0).long()
    return X, y

def train(X, y):
    for _ in range(200):
        opt.zero_grad(); loss_fn(net(X), y).backward(); opt.step()
def acc(X, y): return (net(X).argmax(1) == y).float().mean().item()

Xa, ya = make_task(1); Xb, yb = make_task(2)
train(Xa, ya); print("after A, acc on A:", acc(Xa, ya))   # ~0.95
train(Xb, yb); print("after B, acc on A:", acc(Xa, ya))   # drops to ~0.55 → forgot

EWC puts a soft write-lock on each weight, with lock strength set by "how important this weight is to the old task." Like giving hot rows / load-bearing indexes in a database high-priority protection: not "no writes allowed," but "want to change these critical weights? Sure—but it'll cost you dearly." Changing irrelevant weights is nearly free; changing load-bearing ones incurs a steep penalty—learning gets elastically pulled back.

Replay needs old data; isolation needs extra parameters. EWC (Kirkpatrick et al. 2017) aims for zero old data, zero new parameters—fighting forgetting purely by constraining weights. The key question: how do you know which weights are task A's "load-bearing walls"? The answer is Fisher information—it measures "if I nudge weight i slightly, how much does task A's output change." Big change = load-bearing = lock it tight; near-zero change = irrelevant = let it go.

When learning task B, EWC adds a quadratic penalty on top of B's loss:

L(θ) = L_B(θ) + Σ_i (λ/2) · F_i · (θ_i − θ*_A,i)²

Symbol by symbol: L_B(θ) is the new task B's loss (to be minimized); θ*_A,i is weight i's value after learning A (the "old-memory anchor"); F_i is weight i's Fisher information (importance); λ tunes "how much you care about not forgetting A." Intuition: hang a spring on each weight pulling it toward "its value under task A," θ*_A,i; the spring's stiffness is F_i·λ—load-bearing weights get very stiff springs (barely movable), irrelevant weights get soft springs (free for B to use). So B can only optimize in "directions that don't disturb A." This is an explicit knob on the stability-plasticity spectrum: bigger λ = more stable, smaller = more plastic.

# EWC core: after A, estimate Fisher; when learning B, add quadratic penalty
def estimate_fisher(net, X, y, loss_fn):
    fisher = {n: torch.zeros_like(p) for n, p in net.named_parameters()}
    for i in range(len(X)):
        net.zero_grad()
        loss_fn(net(X[i:i+1]), y[i:i+1]).backward()
        for n, p in net.named_parameters():
            fisher[n] += p.grad.pow(2) / len(X)   # squared grad ≈ Fisher diagonal
    return fisher

# after A: star = weight snapshot (θ*_A), fisher = importance
star = {n: p.clone().detach() for n, p in net.named_parameters()}
fisher = estimate_fisher(net, Xa, ya, loss_fn)

def ewc_penalty(net, lam=1000.):     # add to total loss when learning B
    return lam * sum((fisher[n] * (p - star[n]).pow(2)).sum()
                     for n, p in net.named_parameters())
# total = loss_fn(net(Xb), yb) + ewc_penalty(net)  → A's accuracy preserved

Replay is like a database's WAL (write-ahead log) replay: to keep old and new consistent, you re-run old transactions alongside. Experience replay stores a small set of old samples and feeds old+new mixed. Generative replay is cleverer—instead of storing raw old data, it trains a generator to synthesize "fake old data," effectively replacing a full backup with a compressed snapshot generator, balancing privacy and storage.

Forgetting's root cause is "not seeing A's data while learning B." The most direct fix: rehearse A while learning B. Experience replay keeps a small memory buffer of A's samples and mixes a few old ones into each batch, so the gradient lowers both old and new losses at once and the weights don't drift. Crude but extremely effective—it's nearly the strongest baseline in continual learning.

But storing old data has two hard constraints: storage blowup (tasks grow without bound) and privacy/compliance (medical or user data can't be retained). Generative replay (Shin et al. 2017, Deep Generative Replay) offers an elegant fix: train a pair of cooperating models—a generator that learns to produce "data that looks like old tasks," and a solver that does the real classify/predict work. When learning a new task, the generator first emits a batch of synthetic old samples (labeled by the old solver), then trains on a mix with the new data. The inspiration is the hippocampus "replaying" experiences during sleep to consolidate memory. Flow:

# Experience replay: the minimal, strongest baseline. Keep a buffer, mix in
import random
buffer = []                                  # stores (x, y) old samples

def reservoir_add(buf, sample, cap=200):    # reservoir sampling: uniform retention
    if len(buf) < cap: buf.append(sample)
    else:
        j = random.randint(0, len(buf))      # random replace, stay representative
        if j < cap: buf[j] = sample

def train_with_replay(net, opt, loss_fn, Xb, yb, k=32):
    for i in range(200):
        xb, yb_ = Xb, yb                     # current task batch
        if buffer:                           # mix in old samples, joint loss
            old = random.sample(buffer, min(k, len(buffer)))
            ox = torch.stack([o[0] for o in old])
            oy = torch.tensor([o[1] for o in old])
            xb = torch.cat([xb, ox]); yb_ = torch.cat([yb_, oy])
        opt.zero_grad(); loss_fn(net(xb), yb_).backward(); opt.step()
# while learning A, reservoir_add into buffer; learning B auto-rehearses A → forgetting eased

Parameter isolation is a partitioning / sharding idea: since shared mutable state causes write conflicts, give each task its own dedicated block of parameters and set it read-only once learned. Other tasks use other shards—physically no write conflict, and old tasks are structurally protected—just like microservices deployed independently, never overwriting each other.

Regularization and replay are "soft" protections—old knowledge can still be slowly eroded. Parameter isolation gives a "hard" guarantee: any weight allocated to an old task is frozen and never updated again, killing forgetting at the root. Two representative approaches:

① PackNet (Mallya & Lazebnik 2018)—"pack" multiple tasks into fixed capacity. Large nets are highly redundant; many weights can be pruned with almost no accuracy loss. Recipe: learn task A → prune away a batch of unimportant weights → freeze the remaining "A weights" → use the just-freed spare weights to learn task B → prune, freeze again... Like carving new partitions on a fixed disk for new tasks until space runs out. Zero storage growth, but the task count is capped by a hard capacity limit.

② Progressive Networks (Rusu et al. 2016)—add a new column per task. Each new task adds a fresh column of parameters, with all old columns frozen; the new column reads features learned by old columns via lateral connections (enabling forward transfer), but can't modify the old ones. Truly zero forgetting and it reuses old knowledge—at the cost of parameters growing linearly with the number of tasks, like spinning up a new replica per task that may only read the old replicas.

# PackNet-style: after A, prune+freeze, hand "off" weights to B
import torch

def prune_and_freeze(weight, keep=0.5):
    # keep top-`keep` by magnitude as task A's "locked partition"
    flat = weight.abs().flatten()
    thresh = flat.kthvalue(int(len(flat) * (1 - keep))).values
    taskA_mask = weight.abs() >= thresh    # True = belongs to A, frozen
    weight.data *= taskA_mask                     # prune the unimportant (zero, leave for B)
    return taskA_mask

def masked_grad_hook(mask):
    # zero the "A partition" gradient on backward → A weights never updated by B
    def hook(grad): return grad * (~mask)
    return hook

W = net[0].weight
maskA = prune_and_freeze(W, keep=0.5)   # call after learning A
W.register_hook(masked_grad_hook(maskA))  # from now, learning B hard-protects A's partition
# training B: only ~50% spare weights update, A's accuracy stays exactly the same

• Kirkpatrick et al. 2017 — Overcoming Catastrophic Forgetting (EWC) — the founding regularization work; Fisher info + quadratic constraint, a must-read
• Shin et al. 2017 — Deep Generative Replay — generator+solver dual model, a hippocampus-inspired replay framework
• Mallya & Lazebnik 2018 — PackNet — prune+freeze to "pack" multiple tasks into a fixed-capacity net
• Rusu et al. 2016 — Progressive Neural Networks — a column per task, lateral connections transfer old features
• Lopez-Paz & Ranzato 2017 — Gradient Episodic Memory (GEM) — constrain gradient direction with old samples, balancing anti-forgetting and forward transfer
• van de Ven & Tolias 2019 — Three Scenarios for Continual Learning — clarifies task/domain/class-incremental settings; a framework for fair comparison