A better fit is not a better model
2026-08-29 · Method & Limits
The more beautifully a model fits the data you already have, the less its statements about tomorrow tend to be worth. That is not a figure of speech — it is a curve, and it has been drawn.
Every December somebody walks through the year just past — the markets, the company, the state of the world — and it all hangs together: because of that, this; because of this structure, that outcome. You listen and think, yes, that's how it went. The same people wrote predictions in January, and when you dig those out, they are wrong with remarkable consistency.
Both things are true at once, and that is the phenomenon this issue is about: explaining well and guessing right are not the same skill, and they frequently work against each other. The tighter a model — or a story — fits what already happened, the less constraint it has left over for what hasn't.
The first forty-one issues built tools: power laws, networks, criticality, resilience, measures of complexity. This one goes back and checks how those tools get misused. It introduces no new mechanism. It covers four gates: overfitting, after-the-fact narrative, the statistical mistake behind "that looks like a power law", and what it actually takes for a model to count as tested. Issue 43 will ask the bigger question — what the field as a whole stands accused of.
Start with the smallest possible example. You have eight points and you want the pattern behind them. You can draw a straight line through the middle of them, or you can draw a wandering curve that passes exactly through every single one.
The second curve has zero error on those eight points. By the standard of "fits well", it wins outright. But go measure a ninth point and it will probably land far from that wandering curve — further than it would from the clumsy straight line.
The reason is not subtle. Those eight points contain a real pattern, and they also contain measurement error, chance fluctuation, and whatever mood the instrument was in that day. The model cannot tell which is which. Give it enough freedom — enough knobs to turn — and it will memorise the errors too, as though they were the pattern. That is overfitting.
John von Neumann has a line, relayed by Freeman Dyson, that gets quoted constantly: "With four parameters I can fit an elephant, and with five I can make him wiggle his trunk." Somebody eventually took it as a challenge — a 2010 paper drew a recognisable elephant with four complex parameters. Parameters are not knowledge. Parameters are freedom.
So you cannot judge a model by how it does on the data you tuned it on. That data is in-sample, and the error on it only tells you how much the model memorised. What carries information is out-of-sample data — a batch the model has never seen and that played no part in any tuning decision. Plot both errors on the same axes and the situation becomes obvious.
The worst part: if you only watch the blue line, you will feel like you are making progress the whole time. Overfitting does not hurt. While it is happening, every metric is improving.
Before any model ships, hold out a slice of data that took no part in any tuning, and report its error exactly once. Cut that slice by time — take the most recent stretch — rather than sampling at random; random splitting scatters same-day, same-batch, same-source records across both sides, which lets the model peek at the answers. If you have no such slice at all, then do not quote an "accuracy" figure; the number does not mean what it appears to mean.
The parameters in the last section were numbers. But when people forecast and post-mortem, they don't use numbers — they use words: "resilient team", "decisive leadership", "the timing was right", "riding the wave". Words have a property numbers don't: they are free, and there is no limit on how many you may use.
The consequence follows immediately. With unlimited words available, you can assemble a smooth account of any outcome whatsoever. And usually out of the very same facts.
Lay this figure over the previous one and you get the most useful sentence in this issue: after-the-fact narrative is the limiting case of overfitting. Unlimited parameters, zero cost, error identically zero — it sits at the far right end of that orange curve, where out-of-sample error is highest. This is not an analogy. It is the same position on the same curve.
Psychology reinforces it. Hindsight bias means that once something has happened, people sincerely misremember their earlier judgement and feel they "knew it all along". So you don't even notice yourself fitting after the fact — memory has already tuned the parameters for you.
How do you tell a real explanation from a smooth account? There is exactly one operational test: an explanation must rule some outcomes out in advance. If a story would hold up just as well had the outcome been reversed, its information content is zero, however sensible it sounds. That is all "falsifiable" means in daily life — not the philosophical posture of "could be refuted", but the concrete question of which possibilities it crossed off beforehand.
Keep separate books for explanatory power and predictive power, and don't let the first pass itself off as the second. Run any account — of the company, the market, someone's behaviour — through one reversal test: "if the outcome had gone the other way, could this story be told just as smoothly?" If yes, strike it from your decision inputs; it is comfort, not constraint. What survives the test is what you actually learned.
One straight line has kept recurring on this site: take the logarithm of both event size and frequency, plot them, and a lot of phenomena line up. That line is a power law → ref · Power-law distributions, and it means there is no typical size — big events are far more common than intuition allows.
The trouble lives in the phrase "looks like". On log-log axes, human judgement of straightness is terrible. Logs flatten differences, so a gently bending curve is nearly indistinguishable from a line. And several completely different distributions produce that same picture — the usual rivals being the log-normal (bell-shaped once you take logs) and the exponentially truncated power law (a tail that drops off a cliff somewhere).
Worse, the standard way of drawing the picture manufactures straight lines. Tail data is inherently sparse, so if you count events in equal-width bins, the tail bins hold zeros and ones, the plot turns to confetti, and the eye starts "seeing a trend". (The professional fix is the complementary cumulative distribution — the fraction of events at or above x — which needs no binning and behaves far better in the tail.)
In 2009 Aaron Clauset, Cosma Shalizi and Mark Newman systematised all of this into one testing procedure (the CSN procedure below) and re-examined two dozen widely cited "power-law" datasets with it. The verdict was deflating: only a handful passed cleanly, and a substantial share could not be distinguished from a log-normal at all.
Two terms to unpack. Maximum likelihood means: of all possible parameter values, pick the one that makes this data the least surprising. It is far more reliable than drawing a regression line through a log-log plot, because that line assumes every point carries the same amount of error — and tail points plainly don't. A p-value here answers: if the data really had come from a power law, how often would we see a discrepancy at least as large as the one measured? If that probability is small, the power-law hypothesis does not hold up.
But the statistics are not the interesting part. The interesting question is: why does everybody want that straight line so badly? Because a power law is not just a shape — throughout this site it has come attached to mechanism stories: self-organised criticality, preferential attachment, fractals. Claiming a power law quietly claims the mechanism too. Meanwhile, for decisions, power law and log-normal differ far less than people assume: both are heavy-tailed, both mean the average is unreliable, both mean tail risk is underpriced and you must not size leverage off historical mean drawdowns. So the insistence on the words "power law" is usually not the decision talking. It is the story talking.
Next time you are about to say "this is a power law", run a downgrade test first: replace it with "this is heavy-tailed" — does any decision of yours change? Almost always it doesn't, so say heavy-tailed, which is the weaker claim you can actually support. Only when you genuinely need the numerical value of α — to infer a mechanism, or to extrapolate tail probabilities — is the full CSN procedure worth running, and then you must report x_min alongside it, because α can be tuned simply by sliding x_min.
The first three sections took things apart. This one is the constructive half: what would it actually take for a model of a complex system to count as tested?
Forecasts about complex systems are rarely "tomorrow's value will be X". They are "how likely". And a single "how likely" cannot be graded — you say 70% chance of rain, it stays dry, that does not make you wrong. A hundred of them can be graded. If, across the hundred days you said 70%, it rained on thirty, then what you produced was not a probability. It was rhetoric.
The property is called calibration: the probabilities you state must pay out as long-run frequencies. It draws as a picture — your stated probability on one axis, the fraction of those occasions that actually happened on the other. Perfect calibration is the diagonal.
And where does the "how likely" come from in the first place? The dominant method is ensemble simulation: since the initial conditions are imprecise and the model is imperfect, don't run it once. Perturb the starting state a hundred times, run each, and look at how far the trajectories spread → ref · Monte Carlo & ensemble simulation. A narrow spread licenses a confident statement; a wide one demands an uncertain one. The width is part of the forecast, not a defect in it.
Essentially one industry has made calibration routine: weather forecasting. The reason is unglamorous — it is scored every single day. Forecast published, truth revealed tomorrow, tens of thousands of repetitions, and the reliability diagram draws itself. The field even has a dedicated score, the Brier score (Glenn Brier, 1950): take the gap between your stated probability and what happened (1 or 0), square it, average. Lower is better. It has an elegant property — it decomposes exactly into "how well calibrated" and "how sharp". Being right and saying something are separate virtues: someone who says 50% every day may be well calibrated and completely useless.
This also explains why the mechanism models earlier on this site so rarely quote an accuracy figure: they mostly never get scored. But wherever the opportunity exists, it should be demanded.
When you present a model, present its failure conditions in the same breath as its conclusion — "this rests on three premises, A, B and C; if B fails, the conclusion reverses." Without that sentence your audience has no way to calibrate how much to believe you. Second: log every probabilistic judgement, yours and your team's and your vendors', and once you have a few dozen, draw the reliability diagram. The cost is a spreadsheet. The return is finding out, for the first time, what your "very likely" is actually worth.
Four sections demanding that models submit to testing. Now the other side: this demand itself gets misapplied in several ways, and they are common ones.
First, not every model exists to predict, and grading those on predictive accuracy uses the wrong ruler. Schelling's segregation model → ref · Schelling's segregation model never set out to predict the segregation level of any actual city. It establishes a sufficiency claim: very mild neighbour preferences are enough to produce a sharply segregated pattern, with nobody harbouring hostility. That claim does not stand on fitting data; it stands on demolishing the default inference that visible segregation implies strong prejudice. What such models should be tested on is different: whether the premises were quietly strengthened, whether the conclusion really follows from the rules, and whether a different but equally reasonable parameter set flips it (robustness).
Second, "all models are wrong, but some are useful" has been degraded into a liability waiver. The line is George Box's, and his point was to direct your effort at whether the wrongness matters — not to hand every model a pass. Used correctly it must be followed by two specifics: wrong in which direction (systematically over- or under-stating?) and by how much. If you cannot supply those two, the model has not been tested, and the quotation is just deflecting the question in advance.
Third, calibration is unavailable for one-off events. A century of climate, one financial crisis, a country's transition — these happen once; there is no "seventy of the hundred times". The standard from section 4 simply does not apply. What remains are substitutes, and they must be labelled as weaker: can the model reproduce the key qualitative behaviour over a known historical stretch (not the numbers — whether the turning point appears at all)? Does a different reasonable parameter set flip the conclusion? And, written down in advance: "if we observe X, this model is finished." All three are weaker than a reliability diagram. All three beat doing nothing.
Fourth, overfitting has an equally lethal mirror image: oversimplification. This issue has spent five sections attacking excess parameters, which reads easily as "simpler is safer". It isn't. The left half of that error curve climbs just as steeply: a model missing a decisive variable is not made safe by its elegance — it has only buried the error deeper. The problem with mainstream macroeconomic models before 2008 was not too many parameters; it was the reverse. The class of frameworks known as dynamic stochastic general equilibrium (DSGE) models generally contained no financial intermediation and no leverage. The real choice is never "simple or complex". It is what to throw away.
Fifth, held-out data is a consumable. It works only while it has never been seen. You evaluate version one on it, dislike the result, go back and revise, and evaluate again — from the second evaluation onward that data has passed through your hands and participated in tuning. It is no longer out-of-sample. This is the standing disease of public leaderboards and shared benchmark sets: an entire community iterating against one test set, collectively and unwittingly overfitting to it.
Put a label on every model you own, and allow only three: point forecast (report out-of-sample error), probabilistic forecast (report a reliability diagram, or say plainly "not yet tested" if you lack the repetitions), and mechanism demonstration (report premises and robustness; never report accuracy). Mislabelling is the cheapest and commonest of all the traps here — talking like a mechanism model while making a point forecast, and mocking a mechanism model for its point accuracy, are two faces of the identical error.
What is left is precisely the part that least resembles narrative: establishing what happened, when, who was present, and what information they had in hand at the time. Those claims can be overturned by new evidence, which is exactly why they carry information. The part that carries none is the connective layer of "and therefore he was bound to act that way". A usable test: tell the episode to someone who does not know the ending — can they rule any endings out? If not, that stretch was narrative rather than record.
Because new samples are extraordinarily expensive: there is only so much history, and it is non-stationary — institutions, participants and market mechanics all change, so the "out-of-sample" period from twenty years ago may not share a distribution with today. Worse is the social overfitting of strategy research: the whole industry has tried countless strategies against the same stretch of history, and the ones that get published and circulated are naturally those that happened to fit it. So in finance, "this strategy backtests well" carries even less prior information than it would elsewhere.
Not necessarily, but it must take a risk somewhere. Evolutionary theory predicts the specific path of any given lineage very poorly, yet it makes many refutable claims — the stratigraphic order in which fossils appear, the ranking of genome similarity across species. So the criterion is not "can it produce a number" but "is there a class of observations that would put it out of the game". A model that cannot name such a class, however elegant, has not entered.
A common account is asymmetric cost: where information is scarce and action is required immediately, a wrong causal story often beats a blank — at least it yields something to do. But that explains its origin, it does not endorse its present use. Worth noticing: the modern decision environment has inverted that cost structure. Data is no longer scarce, and most actions need not be taken this second.
None of the current options is clean: rotate in fresh test sets (expensive, and old results become incomparable), seal the test set and permit a single submission (which kills iteration), or use a hidden set that reports only aggregates. The more fundamental move is to record how many times the dataset has been queried — the degree of overfitting tracks that count directly, and almost nobody counts it.