Whether a forecast is good can't be judged by whether that single call turned out right. The real gold standard is this: of all the things you said were "70% likely," did roughly 70% actually happen? Calibration means aligning the confidence you voice with the true frequency of the world. One "90% sure" call that blows up proves nothing; but if only six in ten of your "90%" claims come true, that's not bad luck — you are systematically overconfident.
Bin all your forecasts by confidence: put every "60%" in one pile, every "90%" in another, then look at how often each pile actually came true. Perfect calibration is a diagonal line — whatever probability you state, that fraction happens. Most people's curve isn't on the diagonal; it sags below it: the high-confidence band under-delivers, the low-confidence band happens more than expected. That sag is the fingerprint of overconfidence, and it correlates almost not at all with how smart you are — but strongly with whether feedback has ever ground your judgment down.
The best-calibrated profession isn't scientists — it's weather forecasters. When the forecast says "70% chance of rain today," pull out every day they said 70% and the fraction that rained lands remarkably close to seven in ten — because they get undeniable feedback every single day. Flip it around: the more telegenic the political pundit, the worse the forecasts. A twenty-year study of tens of thousands of predictions found that famous experts scored barely better than chance, and often worse the more famous they were — because the stage rewards flat certainty and punishes hedging, while calibration demands you dare to say "sixty percent."
The same disease lives in machine learning models. Modern deep neural networks are notoriously overconfident: they report 99% confidence while their actual accuracy may be only eighty percent, which is why engineers now run explicit "confidence calibration" (e.g. temperature scaling) to pull probabilities back toward real frequencies. Medicine shows it too: a doctor's certainty about a diagnosis routinely runs far ahead of the rate later proven correct. Anywhere a system — a brain, a model, an organization — uses "probability" to make decisions, calibration is the easily-ignored dimension that quietly decides success.
In architecture reviews, capacity planning, and hiring calls, you already report probabilities in your head every day — you just never write them down, so they can never be calibrated. Try translating "this design should be fine" into a number — "I'm 80% sure it'll survive the traffic peak" — and record it. Accumulate a few dozen, then reconcile: you'll see for the first time what your "80%" is actually worth.
Think of the last time you thumped the table and said "this will definitely be fine" about a technical call. If you'd been forced to give a number then, would it have been 95% or 70%? And in hindsight, was that number too high — or were you simply unwilling to let it land on paper at all?
Calibration alone isn't enough. Someone who reports "50%" for everything can be perfectly calibrated yet completely useless — they never risk giving you information. A good forecast needs both calibration and resolution: the nerve to push toward 0 or 100 when the situation warrants it. The Brier score quantifies both at once into a single comparable, trainable number — the first time "forecasting ability" turns from mystique into a craft with a feedback signal.
The rule is almost brutally simple: take the probability you reported, subtract the outcome (1 if it happened, 0 if not), square it, and average over all your forecasts. Lower is better. Say 90% and it happens, you're docked 0.01; say 90% and it doesn't, you're docked 0.81 — it comes down hard on being "confidently wrong." The score also decomposes into three parts: calibration (does your confidence deserve your hit rate), resolution (do you dare push different events toward different probabilities), and the inherent uncertainty of events themselves. It punishes two kinds of cowardice at once: overconfidence, and forever hiding at 50%.
Two forecasters: A always says 50%, never wrong and never blown up; B dares to say 85% and 15%, and occasionally takes a big fall. Intuition calls A, who "never errs," the safer bet — yet the Brier score usually declares B the winner, because A, while safe, supplies no information you can act on, whereas B gives you a genuinely actionable signal most of the time. The score forces you to confront a counterintuitive truth: a forecaster who never risks anything and stays perpetually vague is in fact the least valuable one.
Anywhere uncertain judgments must be scored, the Brier score casts a shadow. Credit-scoring models, sports-betting odds, and the performance review of weather agencies all run on the same idea. Its close cousin in machine learning is log-loss — the loss function that, when training a classifier, forces the model to output honest probabilities and savagely penalizes being confidently wrong, essentially a continuous version of the Brier spirit. It also echoes the iron law of deliberate practice: without clear, timely, quantified feedback no skill can truly sharpen — and most people's judgment stalls precisely because no one ever kept score.
You can build your team a minimal "forecast ledger": for every key technical bet — will this middleware hold, will this migration slip — have each person report a probability, then reconcile with the Brier score at quarter's end. After a few rounds, who is genuinely reliable and who is merely loud emerges in a way no one can argue with — and everyone finally gets a feedback loop to sharpen their own judgment.
Is there someone on your team who "never commits to anything and is somehow always right afterward"? If everyone's forecasts were kept on a Brier ledger, where would they rank — would the score reward their prudence, or expose that they never actually placed a bet?
The best starting point for forecasting something is often not to study that thing's own details, but to first ask: how does a whole class of similar things usually end up? This is the "outside view." The counterintuitive part — the more deeply you know a project and the more invested you are, the more systematically optimistic and off your estimates become. Because the inside view sees only this one unique path in front of you, and stays blind to the graveyard of failures that similar events have already dug.
Two steps. First, find the base rate: sort the thing in front of you into a "reference class" — similar projects, similar decisions, similar cases — and see what that class's average outcome was. Second, adjust gently from that base rate using the specific evidence at hand. The order is everything: anchor first on the world's historical statistics, then correct — rather than immersing yourself in your own story first and hunting up a number to back it. In Bayesian terms, the reference class is your prior — set the starting point wrong and no amount of later detail can rescue it.
There's a classic story a psychologist lived through: a team was writing a new textbook and, judging by their progress, estimated "about two years to finish." But when someone pressed, "how long did similar teams writing a textbook from scratch actually take?" the answer was seven years — and about four in ten gave up midway. The team learned the base rate, and still chose to believe they were different — and ended up working a full eight years. This is the "planning fallacy": we almost always underestimate time, cost, and risk, because in imagination we walk the one path where everything goes right, while reality is the average line of similar projects.
Cost overruns on large infrastructure aren't accidents but a predictable norm — precisely because decision-makers use the inside view (this tunnel will be different) instead of the outside view (this class of tunnel overruns by how much on average). Startup success rates, the odds a new drug clears clinical trials, the fraction of mergers that create value — all are base rates repeatedly proven and repeatedly ignored. It's continuous with yesterday's causal inference too: the reference class gives you the prior probability, the evidence at hand gives you the likelihood, and rational updating multiplies the two — while the human default is to discard the prior and stare only at the vivid story in front of you.
Next time you schedule a technical project, don't rush to stack up hours from a requirements breakdown. First ask: "for the similar projects we've done, from kickoff to actually shipping, what was the median? And how many of those finished on their original schedule?" Write that base rate at the top of your estimate, then argue why this time might be faster or slower. You'll find that honest historical number lands closer to reality than any bottom-up Gantt chart.
For the project you're scheduling right now, if you were only allowed to estimate using "the actual duration of past similar projects" — and forbidden to look at any detail of this one — how much longer would your number be than your current schedule? That gap is the "inside-view tax" you're currently paying.
Accurate forecasting is neither talent nor insider access, but a set of learnable thinking habits. The most subversive evidence: a group of ordinary people with no classified channels whatsoever, given training and scorekeeping, can consistently and reliably beat professional analysts armed with intelligence. This pulls forecasting down off the altar of "the master's intuition" into a craft anyone willing to seriously practice can improve at — provided you actually put the first three cards (calibration, scoring, reference classes) to work.
The shared habits of top forecasters are almost all counter-instinctive. They break a big, vague question into several checkable sub-questions; they actively hunt for evidence against their own view rather than gathering support for it; they anchor the starting point with a reference class; and then — the crucial step — they update frequently in small increments, nudging the probability up or down a touch with each new piece of information rather than clinging to a position and waiting to be blindsided. They treat beliefs like an investment portfolio, not an identity: changing your mind isn't shameful, it's settling the account.
In a large forecasting tournament, a team of amateurs — retired programmers, homemakers, engineers — forecast with accuracy roughly thirty percent higher than intelligence analysts who could read classified briefings. What they won on wasn't information but method: decompose, seek counter-evidence, update in small steps, keep score and review. This punctures a popular superstition — that forecasting well comes from "knowing what others don't"; the truth is it comes from "handling the information everyone can get more honestly than everyone else."
This habit set is nearly a universal operating system for rationality. In investing, it's probabilistic thinking resisting the seduction of "telling a good story" — great investors treat every stake as a probabilistic bet updated with evidence, rather than falling in love with a holding. In AI, it maps to model ensembling and Bayesian updating: weighting many weak signals and continuously correcting with new data usually beats any single strong assertion. In any field of repeated decisions under uncertainty, "decompose — seek counter-evidence — update in small steps — keep score" is the same underlying engine.
Turn it into a light team ritual: for every major technical bet — betting on a framework, deciding build vs buy, wagering on an architecture evolution — write down "my probability right now + three reasons + what evidence would change my mind." Update the probability as new information arrives, then review with the Brier score afterward. Over a few quarters you cultivate not just sharper judgment but a culture that "dares to change its mind, and does so on evidence" — precisely the scarcest core of the "AI super-individual."
When was the last time you deliberately lowered your probability on some technical judgment because of new evidence? If you can't recall one, is that because you've been consistently right — or because you've been mistaking "holding your position" for "having judgment"?