temperature=0 was never the same as reproducible — what makes two runs diverge isn't sampling, it's the batch size you don't control.
Almost anyone who has run an LLM eval has hit this wall: same prompt, temperature=0, seed pinned — and two runs still differ; prompt A beats B by 2 points and you can't tell if that's a real gain or noise. Most people file it under "model voodoo" and either ignore it or frantically add seeds. The widely-shared 2025 Thinking Machines Lab blog dug out the real culprit: the drift isn't from sampling randomness, it's floating-point non-associativity + server-side dynamic batching — the batch your request lands in is out of your hands, and batch size changes the reduction order of matmul/attention, nudging the per-token logits. This issue doesn't explain "what temperature is" (that's 101) — only four things that bite in production: where the drift actually enters, what seed/system_fingerprint really guarantee, how to make evals stop being flaky, and the throughput bill you pay for determinism.
Break one intuition first: temperature=0 is greedy decoding (take the argmax logit at each step), which in theory is fully deterministic. So where does drift come from? Floating-point addition is not associative: (a+b)+c ≠ a+(b+c) at finite precision differs in the last few bits. And the GPU kernels for matmul, RMSNorm, attention pick different reduction strategies and tiling orders depending on the current batch size — this is the lack of batch-invariance Thinking Machines names: the same sample, run alone vs. packed into a batch of 32 requests, reduces in a different order and produces logits that differ ever so slightly. On most tokens that error is harmless, but hit a near-tie token where two logits are nearly equal and argmax flips, choosing a different word — then autoregression amplifies that fork infinitely and the two trajectories fully diverge. The key: a production inference server's batch size floats with live load, entirely outside a caller's control, so "same request, different moment" simply drifts.
Reproduce it yourself first — turn "voodoo" into something observable by running a fixed input N times and locating the first divergent token:
# Diagnose: run the same request N times, see where outputs fork
outs = [call(prompt, temperature=0, seed=0) for _ in range(20)]
uniq = set(outs)
print(f"{len(uniq)} distinct outputs / 20 runs") # >1 = nondeterministic
# Find the first divergence: align outputs by token, locate first mismatch index
# Divergence usually sits late and on near-tie tokens — confirms batch-reduction driftMake this the first health check when you adopt any new provider or self-hosted stack: measure how nondeterministic it is before deciding whether to pay for determinism.
seed=0 + temperature=0 reproduces — stable on a single low-load box, drifts the moment you hit production concurrency; ② filing nondeterminism as a model bug — it's a systemic property of fp + batching, not a defect; ③ concluding from a single run.The seed parameter on closed APIs is often misread as a "determinism switch." The official wording is far more restrained: same seed and same other params yield "mostly deterministic" output; and it's bound to a system_fingerprint — a fingerprint of the backend weights + infrastructure config. The moment the provider changes inference config (which happens a few times a year), the fingerprint changes and seed's reproduction guarantee is void. A finer trap: the docs state that even when seed and fingerprint match, you may still observe differences; and larger max_tokens / longer output means worse reproducibility (the §1 fork amplifies — more tokens, more chances to hit a near-tie). So position seed correctly: a variance-reduction tool, not a variance-elimination guarantee.
resp = client.chat.completions.create(
model="...", messages=msgs,
temperature=0,
seed=42, # pin the seed
max_tokens=256, # shorter = steadier: don't gamble reproduction at 4k
)
fp = resp.system_fingerprint # record it!
log(fp) # fp changed = your baseline is void, re-run the benchmark
# Right use: pin seed + monitor fingerprint changes + keep output short
# Wrong expectation: bit-level regression assertions / cross-fingerprint comparisonConnect §1–2 to evals: since the same request drifts, your pass/fail verdict flips on samples sitting right at the threshold. You get 82 one run, 85 the next; judging "the new prompt is better" from a single run is likely reading noise. Two engineering exits — pick one, don't mix: (A) want true reproduction — self-host, use batch-invariant kernels (vLLM has a demo), pin down the systemic drift, evals align bit-for-bit; (B) accept nondeterminism — sample each case n≥5 times, turn a single score into a distribution + confidence interval, and judge A/B by whether intervals overlap, not by which of two single points is larger. Most teams on closed APIs can only take (B), so stop treating a single run as ground truth.
# Robustifying eval: repeated sampling → report mean±std, not a single point
scores = [grade(run(case)) for _ in range(5)] # n≥5
mean, std = stat(scores)
report(case, mean, std)
# A/B: check if confidence intervals overlap; don't conclude from 82 vs 85
# Bonus: flag near-tie samples (pass/fail flips across repeats) — noise hotspotsDeterminism has a real price. Batch-invariant kernels give up "pick the fastest reduction adaptively per batch," and in Thinking Machines' measurements the deterministic mode is markedly slower (throughput nearly halved when unoptimized), closed only later with engineering. So don't flip determinism on globally — decide by scenario: debug / reproduce a bug / compliance audit / regression baseline — yes, reproducibility is mandatory; on-policy RL training — yes, if the sampler (inference) and trainer are numerically inconsistent, on-policy quietly degrades into off-policy and the gradient is biased; creative generation / brainstorming / diverse candidates — no, you want diversity; forcing temp=0 for "stability" only makes output dull, handing you the same mediocre answer across calls.
Want · debug / reproduce a prod bug — must replay the same trajectory or you can't localize.
Want · compliance audit / regression assertion — conclusions must be reproducible; use (A) self-hosting or at least pin seed + short output.
Want · on-policy RL — sampler and trainer numerically aligned, else the training target shifts.
Depends · prefix caching hits — you only reap the cache dividend when the input prefix is bit-stable; drift = miss.
Don't · creative / multi-candidate / ensemble voting — diversity is the value; temp>0 is a feature, not a bug.
temperature=0 on a creative task for "consistency" — output turns uniform and rigid; ② disabling batching globally in production for "reproducibility" — throughput halved, cost doubled, and nobody actually needed it; ③ doing RL but running inference and training on two numerical paths — train/inference mismatch makes the curve diverge weirdly.temperature>0 already treats tokens as a probability distribution — you've already accepted run-to-run difference, so you don't mistakenly expect consistency; and two near-tie tokens have near-equal probabilities under sampling, so either pick is within the distribution, unlike greedy which manufactures the "was supposed to be fixed but changed" illusion. So the real lesson isn't "turn off randomness for determinism" but: greedy's determinism is on paper only, and it's unusually sensitive to underlying floating noise — for true reproduction you must fix the kernel's batch-invariance, not just crank temperature to 0.