Day 50 · 2026.07.09

Data-Driven Management: Let Data Constrain Judgment, Not Excuse You From It

Topic: Data-Driven Management·4 Principles

"In God we trust; all others must bring data." — W. Edwards Deming

This issue's thesis: "Be data-driven" is every big company's mantra, but the real divide isn't whether you use data—it's whether you use it correctly. Misused, data turns on you: a metric gets gamed the moment it's used to judge people; the prettier the dashboard, the more it hides what it never measured; A/B results get retro-fitted into whatever story you wanted; and "insufficient data" becomes a shield for not taking responsibility. This issue gives four tools: spot the Goodhart trap, see through the blind spots of quantitative data with qualitative work, turn opinion fights into falsifiable experiments, and draw the jurisdiction line between intuition and data.

PRINCIPLE 01

Goodhart's Trap: When a Measure Becomes a Target, It Stops Being a Good Measure Goodhart's Trap

metric gamingpaired indicatorsdiagnose, don't judge

Principle + Canonical Quote

Once a metric is used to judge and apply pressure, people optimize "the metric itself" rather than the real goal behind it, and the metric distorts. The cure isn't to drop metrics—it's paired indicators: measure quantity and quality together so cheating shows up on the other dimension.

"When a measure becomes a target, it ceases to be a good measure." — Marilyn Strathern (1997), popularizing Charles Goodhart's 1975 original

"You should guard against overreacting... by pairing indicators, so that both effect and counter-effect are measured." — Andy Grove, High Output Management, Ch.2 (paired indicators)

Metric Drift: The Moment You Apply Pressure, Metric and Real Goal Part Ways

Scene: Setting "Code Coverage Must Hit 80%"

Context: Your manager wants the team to raise unit-test coverage from 55% to 80% this quarter, written into the OKRs.

✗ Pressure a single metric as the target

"80% coverage is a hard target, tied to your review." Three weeks later the number looks great—but engineers wrote a pile of assertion-free empty tests just to get code "executed." The defect rate didn't move, and you destroyed a once-useful diagnostic signal.

✓ State the metric's purpose, and pair it

Demote its status first: "Coverage is a diagnostic gauge, not a review KPI. What I actually care about is defect escape rate."

Then pair it: "I'll watch coverage + mutation score on critical paths. Coverage can be gamed with empty tests, but a test that can't kill mutants exposes itself instantly—modules with high coverage and low mutation score are padding."

A Gaming-Resistance Checkup for Key Metrics

Once it's used to judge, what's the laziest cheat path? (What you didn't think of, the team will.)
Have I paired it with a counter/quality metric that exposes padding? (velocity + rework rate; tickets closed + reopen rate)
Am I using it to diagnose or to judge (decide who's promoted)? The judging use triggers gaming immediately.
The metric went up—do I have independent evidence the real goal went up too, or only the metric?
Who owns the interpretation? If the person being judged also defines and reports it, you've invited cheating.

Common Mistakes

Treating a proxy as the ultimate goal: velocity, lines of code, PR count are proxies, not value itself. Stare at them long enough and the team optimizes the proxy at value's expense.
Single-metric dictatorship: one North Star with no paired guardrail is a green light for gaming.
Judging AND self-reporting: Campbell's Law—the more a social indicator is used for decisions, the more it corrupts the very process it was meant to monitor.

This Week's Exercise + Reflection

Exercise: Pick the metric your team most values, write down "if I were the one being judged, how would I game it most cheaply without really solving anything," then pair it with a counter-metric that exposes that cheat.

Reflection: Which number on my team keeps improving while I quietly sense "the real thing" hasn't? That's most likely an already-collapsed Goodhart metric.

PRINCIPLE 02

The Blind Spot of Numbers: Data Tells You What; Only Stories Tell You Why What vs. Why

qual × quantuser interviewsunmeasured ≠ unimportant

Principle + Canonical Quote

Quantitative tells you what happened and how big; qualitative tells you why and how. Staring only at the dashboard systematically misses "important but not-yet-measured" things—and "can't measure it" usually means "haven't figured out how" (Hubbard: measurement is a reduction of uncertainty, not its elimination), not "shouldn't manage it."

"When the data and the anecdotes disagree, the anecdotes are usually right." — Jeff Bezos, recounted internally at Amazon (i.e., the data may be measuring the wrong thing)

Scene: Retention Drops 3%, the Dashboard Can't Say Why

Context: A feature you own sees weekly retention drop 3 points week-over-week, and several PMs each have a theory.

✗ Dig for more data inside the data

"Let's slice another dimension." Cut by device, region, version—two days later, ten correlation charts, nobody can name a cause, and the loudest voice declares "must be the new onboarding flow."

✓ Quant to locate, qual to explain

Quant narrows it: "The drop is concentrated on new users' day 2; existing users are flat"—numbers pinpoint where.

Qual explains why: Do 15-minute interviews with ~5 day-2 churners, then watch 10 session replays. The truth surfaces: on a certain device the onboarding button is hidden behind the keyboard. This "why" no amount of slicing could produce.

Which One—Qual or Quant?

Am I answering "how big / did it move" (quant) or "why / what's going on" (qual)? Wrong tool guarantees waste.
A number gave me a surprise (anomaly/drop/spike)—did I go ask 5 real people "what happened," instead of spinning inside the data?
The thing I call "can't measure so ignore" (morale, tech debt, trust)—truly unmeasurable, or have I just not bothered designing a rough proxy?
Is my report to leadership "one number + one face"—scale evidence plus a specific person to care about? (echo Day 49)

Common Mistakes + A Note for Women Leaders

Treating "unmeasured" as "nonexistent": tech debt, collaboration friction, psychological safety don't enter the dashboard, so decisions treat them as zero.
Using quant to fake objectivity: "what to measure and what not to" is itself a value judgment—it just buries the bias inside metric selection.
Cherry-picking flattering interviews: only talking to active users / happy employees is qualitative rubber-stamping of a conclusion you already held.

Female Leader's Note A "count only quantifiable output" performance culture systematically undervalues glue work—mentoring, writing docs, cross-team coordination, tending team emotions; contributions that raise others' output while making your own commit count look unremarkable, which research shows fall disproportionately on women and underrepresented groups. As a manager, deliberately write these into promotion cases with concrete examples—it's both fair and a correction of the systematic bias in your own dashboard.

This Week's Exercise + Reflection

Exercise: For one problem where you have "an anomaly but no cause," do 3–5 real interviews or watch a few real usage records. Is the insight qual gave you something no slicing could?

Reflection: How many important variables in my decisions get defaulted to zero because they "can't enter the dashboard"? Which one deserves even a rough measure this quarter?

PRINCIPLE 03

Experimentation: Turn "Whoever's Loudest" Into "Let the Data Refute It" From HiPPO to Falsifiable Hypotheses

A/B testingfalsifiable hypothesispre-registration

Principle + Canonical Quote

Convert opinion fights into hypothesis tests: in advance, write a falsifiable hypothesis, pre-set the success threshold and sample size, and agree "if it misses, we roll back." The precondition is accepting a counter-intuitive fact—most of the ideas you're confident in don't actually work.

"Evaluating well-designed and executed experiments that were designed to improve a key metric, only about one-third were successful at improving the key metric!" — Ron Kohavi et al., Trustworthy Online Controlled Experiments (2020), from Microsoft Bing data, Ch.1

Scene: The PM Is Sure "Blue Button = Higher Conversion"

Context: In review, a senior PM declares "change the CTA button from green to blue, conversion will surely rise" and wants a full rollout.

✗ Let rank/volume decide (HiPPO)

Either you cave and ship it fully, or you two argue "blue vs. green" and the higher rank wins. Either way you learned nothing, and next time you argue again.

✓ Turn the disagreement into a pre-registered experiment

Argue the hypothesis, not who's right: "No need to guess. Write the hypothesis clearly: blue lifts main-flow conversion by ≥ 2%."

Pre-set the decision rule: "Agree upfront—sample size to X, run a full 2 weeks (avoid the novelty effect), ship only if it hits 2% and is significant; if not, roll back, and no one redefines the metric afterward." The disagreement stops being politics and becomes something data can refute.

Lock These Down Before an Experiment Starts

Falsifiable hypothesis: write it as "if X then metric Y changes ≥ Z," not "improve the experience" (which always "wins").
Pre-set success threshold: what counts as success, decided before it runs, in writing—no moving the goalposts later.
Sample size and duration: compute how many samples and how long, so you don't stop the moment you see what you want (peeking).
Rollback condition: what verdict means failure and reverting, agreed in advance, so sunk cost doesn't hijack the decision.
Guardrail metrics: while the primary metric rises, watch that latency, churn, complaints—side effects—don't get wrecked.

Common Mistakes

HARKing (hypothesizing after results are known): inventing "I meant to test this" after seeing results—any noise can be spun into a story. The hypothesis must be locked before you see data.
Only significance, never effect size: with a big enough sample even a 0.01% lift is "significant"—but is it worth the engineering cost? (And don't be fooled by the novelty effect—the first-few-days inflation of a new release.)
Forcing A/B onto decisions that can't be split: strategy, hiring, reorgs can't all be experimented on—don't use "no data" to dodge them; that's Card 4's turf.

This Week's Exercise + Reflection

Exercise: Take a recent "everyone has a point" argument and rewrite it as an experiment card: hypothesis, success threshold, sample size/duration, rollback condition. Even if you never run it, the act itself exposes "actually nobody has evidence."

Reflection: If "most good ideas don't actually work" is true, how much of what my team is confidently rolling out has never been genuinely refuted?

PRINCIPLE 04

Intuition & Data: Draw Each One's Jurisdiction, Let Neither Usurp the Other Drawing the Jurisdiction Line

high-validity environmentsclassify the decisiondon't launder decisions

Principle + Canonical Quote

Data doesn't kill judgment—it constrains it. In regular + fast-feedback environments, trust the data and your trained intuition; for novel, low-frequency, no-precedent decisions, data is only an input and you still have to make the call responsibly. Don't use data to launder an already-made decision, and don't use "insufficient data" to dodge a judgment you should own.

"Intuition can be trusted only when the environment is sufficiently regular to be predictable, and when there is an opportunity to learn these regularities through prolonged practice with rapid feedback." — Kahneman & Klein, Conditions for Intuitive Expertise (2009), American Psychologist

Trust Intuition or Wait for Data? Two Axes

Scene: The VP Already Decided, Asks You to "Add Some Data"

Context: The VP has decided to launch a big project, then asks you to "pull some data to support this direction."

✗ Become the data's whitewasher

You take the hint and cherry-pick supporting numbers into a pretty slide. The decision wasn't changed one bit; data was only used to rationalize a made-up mind—the most common counterfeit of "data-driven."

✓ One line to surface data's real purpose

Ask the decision's nature first: "How locked is this direction? If the data comes back against it, would we actually change?"

Draw the jurisdiction: "If it's an irreversible, no-precedent strategic bet, it's fundamentally a judgment call—I'll give you risk quantification and worst-case, but let's not dress it up as 'the data says do it.' If it's splittable and reversible, what we need isn't a rubber-stamp, it's a real experiment."

Classify the Decision Before You Judge

Is this decision's environment regular, with fast feedback? (fast + regular → trust trained intuition; slow + novel → structured process)
Is it a reversible two-way door (try boldly, back out) or an irreversible one-way door (slow down, demand evidence)? (echo Day 15)
Am I seeking data to possibly change the decision, or to rubber-stamp a decided one? The latter wastes everyone's time.
When intuition alarms, do I treat it as a lead to investigate or an unargued veto? The former is right, the latter risky.
Am I using "not enough data yet" to stall a judgment I should own?

Common Mistakes

Data laundering (a HiPPO variant): conclusion first, data after—disguising a political decision as an analytic one.
Analysis paralysis: using "let's wait for more data" to dodge low-frequency, high-stakes judgments that data can never fully settle.
Treating intuition as a veto: "something feels off" can launch an investigation, but can't override a proposal without evidence—especially when your intuition was never calibrated.

This Week's Exercise + Reflection

Exercise: Take one decision you'll make this week and classify it: how regular/fast is feedback? one-way or two-way door? Then answer honestly—am I seeking data to possibly change my mind, or to rubber-stamp? Write it down for yourself.

Reflection: That "professional intuition" I trust most—was it trained in a regular, fast-feedback environment? If not (e.g., a low-frequency domain I've bet on only two or three times), it may just be luck dressed as experience.

Going Deeper

If every metric gets gamed once it's used to judge, should we still use metrics for performance at all?

Don't, and performance falls back to pure subjectivity—worse. The answer is layering: metrics as diagnostic gauges (low gaming pressure), but performance triangulated with multiple paired metrics + qualitative evidence, with interpretation owned by someone other than the judged. The core is: never let a single metric map directly to a major consequence—the instant it does, Campbell's Law fires.

Isn't "data-driven" itself a bias—systematically letting "the measurable" beat "the important"?

Yes, it's the McNamara fallacy: treating the quantifiable as the only thing that matters and the unquantifiable as nonexistent. The more data-reverent the org, the more it goes systematically blind on tech debt, culture, long-term trust. The fix is to force a seat for the hard-to-measure-but-important—even a rough proxy. The mark of a mature data culture is that it openly admits where its data doesn't reach.

In a big company with a heavy HiPPO culture, how can a mid-level tech lead push experimentation?

Changing culture top-down is above your pay grade, but you can run demonstration wins in your own patch: pick a low-risk argument, quietly run one pre-registered experiment, and let the result speak—most persuasive when it overturns everyone's intuition, including your own, because it proves the method guards against everyone's hubris, not any one person's. Don't frame it as "challenging the boss's judgment," frame it as "saving one full-rollout disaster."

Kahneman says intuition is trustworthy only in regular environments—yet executives make exactly the low-frequency big calls. So is executive "experience-intuition" basically untrustworthy?

In purely low-validity domains (macro forecasts, one-off big bets), yes—research shows experts there don't beat simple models, and confidence often mistakes past luck for skill (echo Day 30). But an executive's value may not be "predicting which bet wins"—it's asking the right questions and taking responsibility under incomplete information; that's judgment, not prediction. Self-protection: separate intuitions from high-validity experience (managed many people → trustworthy) from those in low-frequency domains (bet two or three times → suspect); for the latter, add guardrails with external benchmarks and dissenters.