The single question "how much faster does AI make developers" yields +55%, +20%, and −19% — and none of those answers need be fraudulent. The difference isn't in the data; it's in the definition. Are you measuring what people expected beforehand, what they felt afterward, completion time under controlled conditions, or a randomized causal effect on real work? Those four form a ladder, and the higher you climb the more conservative — and less flattering — the number gets. The percentages that circulate most widely nearly all sit on the bottom two rungs, while being quoted as if they came from the top.
Each rung up removes one class of contamination. Self-estimates run on memory and self-image. Felt experience absorbs novelty and the rationalization of sunk effort. A controlled experiment removes recall bias but not task selection — winning on a toy problem is not winning on a real system. Only random assignment plus objective timing separates "using the tool" from "who uses it, and on what kind of work." Information rises with each rung; cost rises exponentially. So the evidence in circulation gravitates to the bottom: cheap, plentiful, and flattering. It is a systematic optimistic drift that requires no one to lie.
In a 2023 controlled experiment, nearly a hundred programmers used an AI assistant to write an HTTP server and finished about 55% faster than the control group — a genuine randomized trial, but on a small program built from scratch. A 2025 randomized trial changed the setting: a dozen-plus experienced open-source developers, working real issues in the large mature codebases they had maintained for years. Tasks where AI was allowed took 19% longer on average. The subjective side is the part worth sitting with: they had predicted AI would make them 24% faster, and after finishing still believed it had made them 20% faster. They were in the group that slowed down, and their felt experience pointed steadily the other way.
Medicine's evidence hierarchy is the same ladder: cell assays, animal models, observational studies, randomized trials, systematic reviews — effect sizes shrink as you climb, and a great many therapies that dazzle low on the ladder go to zero near the top. Ed-tech's "learning gains" mostly stop at the satisfaction-survey rung — and as Day 65 argued, fluency during training runs inverse to long-term retention. In hiring, "interview score" and "performance one year in" are two rungs apart, and the measured correlation is far weaker than intuition suggests. Any field where low-rung evidence is cheap and high-rung evidence is expensive grows this same optimistic drift on its own.
When your team evaluates AI coding tools, the easiest thing to collect is "how much faster do people feel." Label that explicitly as rung one, then add at least one rung of measurement: take a batch of real tickets, record cycle time from pickup to merge, and compare stratified by AI use; then go further and randomize which tickets allow it. You will most likely find the gain is real but concentrated in particular task types — and "which task types" is exactly the information a self-report survey can never give you, and exactly what determines how to roll the tool out.
The productivity number in your last quarterly review — which rung was it on? If you were made to re-measure it one rung higher, how much do you expect it to shrink? And if your answer is "probably about the same," what rung is that judgment on?
Every rate is a function of six dials: who goes in the denominator, how strict the threshold is, how the sample is admitted, who answers, how long the time window is, and self-report versus measurement. Turn each one notch and the same reality emits two numbers that differ several-fold — or in sign — and both are legitimate. So the fastest way to interrogate a suspicious figure is not to argue about whether it's right, but to ask where each of the six dials is set. If nobody can tell you, it doesn't belong in a decision.
One at a time. The denominator sets the magnitude of an identical numerator: early in COVID, deaths over confirmed cases gave a case fatality rate of several percent, while deaths over estimated total infections put the infection fatality rate below 1% — same numerator throughout. The threshold is the decision line, and moving it reshuffles the population instantly. Sample admission decides who never gets counted, most classically by tallying only projects that shipped and excluding those cancelled midway, which inflates the success rate on the spot. The respondent's position shifts answers systematically (next card). The window decides whether you read 7-day or 90-day retention. And self-report versus measurement carries a stable directional gap: self-reported exercise minutes run well above accelerometer readings, and self-reported screen time matches device logs so poorly that the two are not interchangeable.
In 2017 the US lowered the hypertension threshold from 140/90 to 130/80, and adult prevalence jumped overnight from roughly a third to nearly half — the tens of millions of new "patients" had not moved a single millimeter of mercury. Only the ruler moved. Unemployment is starker still: the US publishes six official versions simultaneously. The narrow one counts only the jobless who actively searched in the past four weeks; the broad one folds in people working part-time because full-time work isn't available, plus those too discouraged to keep looking — and it typically runs close to double the narrow one. Neither is falsified, both are "the unemployment rate," and which one gets quoted depends on the story being told.
The dial machine engineers know best is the availability SLA: "99.9%" depends on probe frequency, whether partial degradation counts as an outage, whether the window is monthly or annual, and whether the denominator is requests or minutes. Re-dial the same system and you get 99.99%, or 99% — with customer experience entirely unchanged. Crime rates (reports, charges, and convictions are three different curves), carbon accounting (counting the supply chain or not differs several-fold), and poverty rates (absolute versus relative line) are the same machine. Anywhere a metric precedes an incentive, the dials quietly drift toward the flattering side.
Give each core metric a five-line "definition note" displayed on the same screen as the number: denominator, admission rule, time window, whether the source is instrumentation or a survey, and the date the definition last changed. That last line is the valuable one — a meaningful share of the inflection points in any metric curve are not the business changing but the definition changing. Without a change log, you will read a redefinition as a trend and then decide on it.
Pick a metric you look at every week and, without opening the docs, write down all six dial settings. The ones you can't fill in mean you don't actually know what you're looking at — so does the last decision you based on it still stand?
Same organization, same question: the answer shifts systematically with the respondent's position in the hierarchy, and the direction is predictable — the higher up, the more optimistic. Nobody is lying. Each layer genuinely sees a different reality: leadership sees aggregated success stories and reports filtered on the way up, while the front line sees the specific places things jam. So for a figure like "X% of enterprises have adopted AI," the first question isn't "how many," it's "who answered."
Three channels stack. Filtering: bad news attenuates once per layer, and little survives to the top. Identity: someone evaluating a decision they championed is evaluating themselves, and that stake operates without needing to be noticed. Level-of-abstraction mismatch is the subtlest — an executive answers "are we doing this," a frontline engineer answers "does it work in my hands," and two entirely different questions share one survey wording. Because all three push the same way, the resulting error is not random noise but a stable directional shift, and no amount of sample size will remove it.
Child behavioral assessment has a classic large-scale pooled analysis: when parents, teachers, and the child themselves rate the same child on the same set of behavior problems, the correlation between different types of informant is only about 0.28, versus roughly 0.6 among informants of the same type. Change who is looking and it is nearly a different child — yet clinicians must assemble one diagnosis out of that. Hospital patient-safety-culture surveys reproduce the identical pattern year after year: management rates their own institution's safety climate consistently higher than frontline nurses do. Same hospital; the safety level depends on whom you ask.
Observability has an exactly isomorphic scene: server-side instrumentation computes 99.95% availability while client-side measurement shows 98%, because the server cannot see failed connection setups, timeout retries, or degradation on weak networks. Nobody is deceiving anyone — the vantage point decides what is visible. Economic data works the same way: employer payroll surveys and household employment surveys disagree perennially, and both are right, because they ask different subjects. Architecture reviews too: the author of a design and the engineer on call never agree on "is this system maintainable," and it's the latter who gets woken at 3 a.m.
When a report claims "X% of enterprises have deployed AI," turn to the methodology page and look at the respondent mix: if 70% are director-level and above, that number measures organizational intent, not frontline adoption. When you run tool-adoption surveys on your own team, present results split by level rather than reporting a single average — the gap is the highest-information metric in the whole survey, because it measures directly how wide the distance is between intent and practice.
Your team's last internal survey — did you report an overall average or a breakdown? If you split managers and frontline staff and put the two side by side, what are you afraid you'd see? And doesn't that fear mean you already know?
Compressed into a portable tool, it's three questions: what is the denominator, how strict is the ruler, and what is the person telling you this selling? The first two handle unintentional drift in definitions. The third handles directional choice of definition — the hardest to defend against, because whoever picks the definition rarely needs to fabricate anything; they only need to report the most agreeable result among dozens of equally legitimate calculations. Day 66 covered how a number acquires unearned authority; this is about who calibrated its scale.
The funding effect acts not on the data but on design and definition: which comparator, which endpoint, how long the follow-up, which subgroup to report when the headline result disappoints. Every step stays inside the rules, and together they push the conclusion toward the sponsor's side. That explains an odd finding — screened by conventional quality scores, industry-funded studies are not measurably "worse," yet their conclusions still skew systematically. Quality scores check whether execution followed protocol; they cannot detect whether the definition was chosen before or after seeing the data. So the third question isn't an accusation of bad faith — absent pre-registration, it's the only prior still available.
A systematic review of research on beverages and health found that among interventional studies funded entirely by the relevant industry, the share reaching an unfavorable conclusion was 0%; among comparable studies with no industry funding, it was roughly 37%. Not "somewhat lower" — not a single one. Systematic reviews of drug trials agree: industry-sponsored trials are significantly more likely to report favorable results, and the difference cannot be explained away by standard risk-of-bias assessment. The bias occurs where the ruler is chosen, not where the data is read — which is the hardest link to audit, and the least often audited.
Model benchmark leaderboards are the definitive contemporary battlefield: whoever ships the model picks the eval set, the prompts, and the rivals' versions and configurations, and the same model's ranking can invert completely across leaderboards — without a single number being false. Vendor performance whitepapers and industry "maturity surveys" share the structure. The rule inverts usefully too: when a report's numbers cut against the party publishing them, credibility should go up, because the finding runs against the incentive.
Put the three questions into your vendor-evaluation template, and require every supplied number to fill them in: denominator (which requests, which users are counted), ruler (what counts as success, do timeouts count as failures), and source (who ran the test, on whose dataset). Anything that can't be filled in gets demoted to "vendor claims." Working with AI is the same — when a model hands you an attractive percentage, it can't answer these three either, and your value is chasing the figure back to its primary source.
Think of the last time a number changed your mind — what was its denominator? If you had asked that one question at the time, would you have decided the same way? And the more useful question: why didn't you ask?