AI/ML Explained: Same Name, Different Thing — Model Versioning & Eval Reproducibility

A model name is Docker's :latest tag, not a Git commit SHA. It is a mutable pointer, not a content-addressed immutable reference. The "o3" you read about in a paper, the "o3" written into your SLA, and the "o3" the API actually serves six months later can be three different piles of weights. Nobody on your team would use :latest as a rollback baseline — yet the whole industry uses model names as benchmark identity.

Pain point: the minimum requirement for science is referability — "what is the object under study" must be uniquely determinable. Yet benchmark scores, paper conclusions, cost models and SLAs all identify that object with a single string. Once the mapping from name to actual system can change, every conclusion pinned to that name fails silently: no error, no version conflict, just a score that mysteriously drops a few points one day.

Mechanically, one "model name" sits on top of a whole mutable stack, and every layer changes observable behavior:

This is not hypothetical. In December 2024 ARC Prize verified an unreleased o3 preview (high configuration, very large test-time compute) at 87.5% on ARC-AGI-1. The o3 that actually shipped in 2025 scored roughly 41% at the low setting and 53% at medium on the same benchmark. OpenAI clarified that the production o3 is not the same model as that preview (production integrates multimodal capability, and the compute budget the preview used is not available in the product). Same name, thirty-plus points apart — the cleanest specimen of referential drift we have.

The other direction is codename drift: the internal codename "Orion" was widely expected to ship as GPT-5, and was ultimately released in February 2025 as GPT-4.5. There is not even a stable correspondence between rumored codename performance and measured product-name performance.

import anthropic, hashlib, json
client = anthropic.Anthropic()          # needs ANTHROPIC_API_KEY

REQUESTED = "claude-sonnet-4-5"            # the name you request: a mutable pointer
resp = client.messages.create(
    model=REQUESTED, max_tokens=64, temperature=0,
    messages=[{"role": "user", "content": "Reply OK"}],
)
# Key: the response carries the model id the server actually used. Log that,
# not the string you asked for.
print("requested:", REQUESTED, "| served:", resp.model)

# Fingerprint every eval run: if any dimension changed, old scores are not comparable
run = {"served_model": resp.model, "temperature": 0,
       "max_tokens": 64, "prompt_tmpl": "eval-v3", "harness": "lm-eval@0.4"}
fp = hashlib.sha256(json.dumps(run, sort_keys=True).encode()).hexdigest()[:12]
print("run_fingerprint:", fp)   # use this as the primary key, not the model name

Imagine someone hands you a chart reading "MySQL: 18,000 QPS" with no mention of indexes, connection pool, cache hit rate, or hardware. You'd say immediately: that isn't a property of the database, it's a property of the whole system. Agent benchmark scores work the same way — the score belongs to the (model × scaffold × sampling budget) joint system, while the leaderboard shows you a single column: the model name.

Pain point: the leaderboard's data structure is itself misleading. It presents as a map from model name to score, implying the score is an intrinsic property of the model. But on agentic benchmarks (SWE-bench, terminal-style tasks) the model is just a function being called repeatedly; the scaffold around it — which files get retrieved, whether tests are run, how many retries on failure, how candidate patches are ranked — contributes a large share of the score.

The most important confound is collapsing many attempts into one number. Let p be the success rate of a single attempt. If the scaffold allows k independent attempts and can filter the right one using regression tests, the success rate becomes:

pass@k = 1 − (1 − p)^k

Intuition: (1−p) is the probability of one failure, so k failures in a row is that raised to the k-th power, and subtracting from 1 gives "succeeds at least once." The consequence is that as long as p>0, the score climbs monotonically with budget — at p=0.3, pass@1≈0.30 but pass@10≈0.97. A leaderboard that reports a score without k has hidden a freely tunable independent variable.

How large is the effect? Anthropic's own published numbers: Claude 3.7 Sonnet scores 62.3% on SWE-bench Verified, and 70.3% with a custom scaffold. Same model, same benchmark, same day — eight points, enough to flip any two adjacent leaderboard ranks.

import numpy as np

def pass_at_k(n, c, k):
    """Unbiased pass@k estimator (the standard HumanEval formulation):
    c correct out of n samples -> 1 - C(n-c,k)/C(n,k), as a product to avoid overflow."""
    if n - c < k:
        return 1.0          # not enough wrong samples to fill k failures
    return 1.0 - np.prod(1.0 - k / np.arange(n - c + 1, n + 1))

# Same model, same task: single-attempt success fixed at 30% (30 correct out of 100).
# The only thing that varies is how many attempts the scaffold allows.
for k in (1, 2, 5, 10, 40):
    print(f"pass@{k:<2} = {pass_at_k(n=100, c=30, k=k):.3f}")

# pass@1 =0.300 / pass@5 =0.839 / pass@40=1.000
# Model capability did not change by a single bit; only the budget did.
# Reporting a score without k is reporting nothing.

Two teams both say "I implemented JSON parsing," but one accepts trailing commas and the other doesn't — whatever the spec leaves open, the implementation defines. Benchmarks are the same: "MMLU" is not a dataset, it is a four-tuple of dataset + prompt template + scoring method + answer extraction. Change any one of them and the same model reports a score several points apart, with neither side lying.

Pain point: you try to reproduce someone's number, land 4 points off, and spend a week suspecting your environment. The real cause is usually in the scoring protocol, while the paper only said "we evaluated on MMLU." EleutherAI's Lessons from the Trenches on Reproducible Evaluation of Language Models (arXiv 2405.14782) lays this out: model scores are highly sensitive to evaluation setup, which makes cross-method comparison very hard to justify. Their conclusion is that what can genuinely be shared is not a score but the complete evaluation code and configuration — precisely why a harness like lm-eval exists.

Between the dataset and a single number sit five gates, each of which changes the result:

The most overlooked gate is the scoring method. Multiple choice has two mainstream paths: (a) let the model generate, then regex out "A/B/C/D"; (b) skip generation and compare log P(option | stem) across the four options, taking the max. Path (b) has a further fork — whether to normalize by token count:

score_raw = Σ log P(t_i)   vs   score_norm = (1/T) Σ log P(t_i)

Intuition: every token's log probability is negative, so the longer the option, the more negative the sum. Without normalization you systematically favor short options; dividing by token count T turns it into "how natural per token on average," which flips the bias toward long, fluent options. Both are defensible, and both pick different answers — so "the same MMLU" has two scores, and the gap is often larger than the gap between adjacent leaderboard ranks.

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
tok = AutoTokenizer.from_pretrained("gpt2")
m = AutoModelForCausalLM.from_pretrained("gpt2").eval()

def score(ctx, cont):
    """Return (raw log-likelihood sum, length-normalized mean) for a candidate."""
    ids = tok(ctx + cont, return_tensors="pt").input_ids
    n_ctx = len(tok(ctx).input_ids)
    with torch.no_grad():
        lp = torch.log_softmax(m(ids).logits[0, :-1], dim=-1)
    tgt = ids[0, 1:]                          # actual next token at each position
    s = lp[range(len(tgt)), tgt][n_ctx - 1:]   # keep only the candidate's tokens
    return s.sum().item(), s.mean().item()

q = "The capital of France is"
for c in [" Paris", " the beautiful old city of Paris"]:
    raw, norm = score(q, c)
    print(f"{c!r:34} raw={raw:8.2f}  len-norm={norm:6.2f}")
# raw favors the short option, len-norm favors the long one: ranking can invert.
# One model, one question, two equally "correct" protocols -> two benchmark scores.

Using "unit test pass rate" as a proxy for "code quality" hits two independent problems: first, it measures the proxy, not the thing (tests can be weak, and can be gamed by feeding them the answer); second, it is a finite sample and therefore has variance (20 tests and 2,000 tests carry very different confidence). Benchmark scores take both hits identically.

Problem one: construct validity — whether what you measure is what you think you're measuring. Raji et al. 2021, AI and the Everything in the Whole Wide World Benchmark (arXiv 2111.15366), argue systematically that the benchmarks treated as milestones of "general capability" involve fundamentally arbitrary task selection, so reading their scores as progress toward general intelligence has a validity problem at the root.

Two hard pieces of evidence. Contamination: the Scale AI team rebuilt a fresh set of problems matched to GSM8k in distribution and difficulty, called GSM1k (arXiv 2405.00332), and found drops of up to 13 percentage points for some model families, with signs of systematic overfitting (while several frontier families barely moved) — meaning the original score mixed "memorized" with "can do." Weak adjudication: SWE-Bench+ (arXiv 2410.06992) audited patches judged "resolved" on SWE-bench and found roughly 32.7% of passing samples had the solution written directly in the issue or its comments (solution leakage), and about another 31.1% were suspicious because the tests were too weak to verify correctness. A substantial part of that number is not measuring bug-fixing ability.

Problem two: statistical uncertainty. Miller 2024, Adding Error Bars to Evals (arXiv 2411.00640), makes a plain point that is widely ignored: an evaluation is an experiment — the questions are a sample drawn from an unseen super-population, so the score is an estimator and needs error bars. With n questions and accuracy p, the standard error of the mean is:

SE ≈ √( p(1−p) / n )

Intuition: the numerator p(1−p) is the variance of a single Bernoulli right/wrong outcome (largest at p=0.5, smaller as p approaches 0 or 1); dividing by n and taking the square root is "more questions shrink the noise as √n." Plug in n=500, p=0.70: SE≈2.0%, so the 95% confidence interval is roughly ±4 percentage points. The leaderboard gap between 70.3% and 68.9% is therefore usually not significant at 500 questions. And that's the optimistic case — questions from the same repository or template are correlated (clustered), so real variance exceeds the independence assumption.

import numpy as np
rng = np.random.default_rng(0)
n = 500                                # number of benchmark questions
a = rng.random(n) < 0.703              # model A per-question outcome (true rate 70.3%)
b = rng.random(n) < 0.689              # model B (true rate 68.9%)

def se(x):                             # standard error of the mean = sd / sqrt(n)
    return x.std(ddof=1) / np.sqrt(len(x))

print(f"A = {a.mean():.3f} ± {1.96*se(a):.3f}")   # 95% CI is about ±0.04
print(f"B = {b.mean():.3f} ± {1.96*se(b):.3f}")

# Paired difference: compare A and B on the same question, cancelling the shared
# noise source of "how hard this question is"
d = a.astype(float) - b.astype(float)
print(f"A-B = {d.mean():+.3f} ± {1.96*se(d):.3f}")
# Interval straddles 0 -> not significant. Pairing needs fewer questions than
# independent comparison, so it should be the default for a private eval.

• Biderman, Schoelkopf, Sutawika et al. 2024 — Lessons from the Trenches on Reproducible Evaluation of Language Models — the methodological foundation for reproducible evals, and the rationale behind the lm-eval harness
• Miller 2024 — Adding Error Bars to Evals — treating evaluation as an experiment: error bars, paired comparison, sample-size planning
• Raji, Bender et al. 2021 — AI and the Everything in the Whole Wide World Benchmark — the construct-validity critique of "general" benchmarks
• Zhang et al. 2024 — A Careful Examination of LLM Performance on Grade School Arithmetic (GSM1k) — quantifying contamination/overfitting with freshly built, distribution-matched problems
• Aleithan et al. 2024 — SWE-Bench+: Enhanced Coding Benchmark for LLMs — how solution leakage and weak tests inflate "resolution rate"