Think of an LLM as an API that always returns 200 OK: query a record that doesn't exist and it won't return 404 or null—it fabricates a perfectly-formatted response on the spot. The problem isn't that it "wants to deceive you"—it's that its architecture has no "not found" return value at all. At every step it must emit a token; "silence" or "unknown" is never the default option.
First, dispel a misconception: hallucination is not a bug, it's a mathematical inevitability of the training objective. An LLM does next-token prediction—essentially density estimation over language, learning a distribution P(next word | context). This objective contains no fact-checking term whatsoever: the model is rewarded for "sounding like the training data," not for "being correct."
OpenAI's Kalai et al. (2025) nailed this down: even with 100% error-free training data, the cross-entropy objective itself forces hallucination. The intuition—generating a valid answer is no easier than a binary classification task "is this statement true?"; and for a rare fact seen only once or twice in training (a person's birthday, a paper's author), the model has insufficient statistical signal to tell it apart from plausible wrong answers, so the error rate has an irreducible lower bound.
The nastier layer is evaluation incentives: nearly every benchmark scores "1 point for correct, 0 for both wrong and blank." Under this rule, the expected score of guessing is strictly higher than honestly saying "I don't know"—just like on a standardized test, leaving it blank beats guessing. So the entire training-evaluation pipeline systematically trains the model into a confident test-taker, not an honest knower.
from openai import OpenAI client = OpenAI() # needs OPENAI_API_KEY # Ask about a purely fictional entity—the model has no "not found" return r = client.chat.completions.create( model="gpt-5", messages=[{"role": "user", "content": "Describe the contributions of 2019 Turing Award winner Zael Kirmani"}], logprobs=True, # get per-token log probabilities ) msg = r.choices[0].message.content print(msg) # → likely a fluent but entirely fabricated biography # Key observation: token probabilities of the fabrication are often NOT low— # the model "confidently fabricates." High confidence != grounded. lp = r.choices[0].logprobs.content print(sum(t.logprob for t in lp) / len(lp)) # mean logprob
Whether calibration is good is easiest to grasp via weather forecasting: a forecaster who said "70% chance of rain" 100 times is well-calibrated if it actually rained on about 70 of those days. That's separate from how "accurate" he is—he can always say 70% and never 100% (not sharp enough), but as long as the frequencies match, his confidence is a trustworthy signal. Whether an LLM's confidence can be used as an alert threshold depends on the same thing.
The model attaches an implicit confidence (token probability) to every answer. Calibration asks: does its stated confidence match the actual accuracy? Bin predictions by confidence; ideally "in the confidence-0.8 bin, exactly 80% are correct." The metric is ECE (Expected Calibration Error)—the weighted absolute difference between each bin's "average confidence" and "actual accuracy," closer to 0 is better:
The classic fix is Temperature Scaling (Guo et al. 2017): don't touch model weights, just divide the output logits by a learned temperature T, then softmax. T>1 flattens an over-peaked distribution, pulling confidence back in line with true accuracy—one parameter, post-hoc, no accuracy loss. Note it fixes "the confidence number is untrustworthy," it doesn't fix hallucination itself: calibration tells you how much to believe, but won't make a wrong answer right.
import numpy as np def expected_calibration_error(confs, correct, n_bins=10): # confs: confidence per prediction (0~1); correct: whether right (0/1) bins = np.linspace(0, 1, n_bins + 1) ece = 0.0 for i in range(n_bins): # predictions falling in this confidence bin m = (confs > bins[i]) & (confs <= bins[i + 1]) if m.sum() == 0: continue acc = correct[m].mean() # actual accuracy of this bin conf = confs[m].mean() # mean confidence of this bin # gap × fraction of samples in bin, accumulate ece += (m.sum() / len(confs)) * abs(acc - conf) return ece # Overconfident model: confidence broadly above true accuracy → large ECE confs = np.array([0.9, 0.9, 0.8, 0.95, 0.7]) correct = np.array([1, 0, 0, 1, 0]) # 5 questions, only 2 right print(f"ECE = {expected_calibration_error(confs, correct):.3f}")
Picture an honest advisor who says "I'm not too sure about this", optimized for half a year against a "customer satisfaction KPI"—he gradually becomes a salesman who's always emphatic. Sounds more credible, more reassuring, yet is actually less accurate. That's exactly what RLHF does to a model: to please human preferences, it sacrifices the honesty of its confidence. You know the old distributed-systems tradeoff—sacrificing consistency for p99 experience.
A counterintuitive fact: the raw model right after pretraining (the base model) is actually quite well-calibrated—it only does next-token prediction, so its probabilities are its true statistical beliefs about language. But after RLHF (Reinforcement Learning from Human Feedback, aligning the model to human preferences) and instruction tuning, calibration degrades markedly. The GPT-4 technical report has a famous figure: pretrained GPT-4's reliability curve hugs the diagonal, while after RLHF the curve visibly bulges toward the overconfident side.
The mechanism lies in the shape of human preference. When annotators score responses, they systematically prefer answers that are confident, fluent, and give a clear conclusion, and dislike hedging like "it depends…" or "I'm not sure." The reward model learns this preference, and PPO optimization squeezes probability mass onto a few high-confidence phrasings—this is mode collapse: the distribution collapses from "honestly spreading out uncertainty" into "always looking very certain."
This explains a daily observation: aligned models like ChatGPT / Claude read as more assured than raw base models, and therefore lower your guard more easily. It's not that the model "got dumber"—it was optimized into a tone that feels good. After RLHF, the certainty of tone and the reliability of content have become decoupled.
from anthropic import Anthropic client = Anthropic() # needs ANTHROPIC_API_KEY # One prompting trick against overconfidence: explicitly ask for a "calibrated" # numeric confidence, and require thinking "when would I be wrong" first— # forcing the model to spread out its uncertainty prompt = """Answer the question, then give a confidence 0-100. Confidence definition: across all questions where you give this confidence, exactly that fraction should be correct. First list scenarios where you might be wrong, then give the score. Question: What is the exact depth in meters of the deepest point in the Pacific?""" r = client.messages.create( model="claude-opus-4-8", max_tokens=400, messages=[{"role": "user", "content": prompt}]) print(r.content[0].text) # "Calibration definition + reason-about-errors first" usually yields a more # honest score than just asking "how confident are you"
Like a cache-miss signal: internally the system actually knows this was a miss, but if it doesn't expose the miss signal to the upper layer, the caller wrongly assumes every access hit. A model is the same—internally it has a signal distinguishing "I know" from "I'm fabricating" (e.g. the entropy of the output distribution, internal activations), but by default it doesn't report it. Having it self-evaluate means explicitly exposing that cache-miss signal.
The core question: does the model know what it doesn't know? Anthropic's Kadavath et al. (2022, the paper is literally titled "Language Models (Mostly) Know What They Know") gave a somewhat optimistic answer: large models to a substantial degree "know whether they know." They used two operational probes:
The mechanistic intuition: when the model is confident, the output distribution is sharp (a few tokens with very high probability, low entropy); when unsure, the distribution is flat (many tokens with similar probability, high entropy). This "internal uncertainty" is a genuine signal—the problem was never that the model lacks a sense of boundary, but that the default generation pipeline throws this signal away, emitting only the final answer. Having it explicitly do a P(True) self-eval picks the discarded signal back up.
Of course it's "Mostly," not "Always": the more a question is unseen in training and requires multi-step reasoning, the less reliable the self-eval—echoing the statistical origin from section 1, the rare low-frequency region is exactly where self-knowledge also fails.
from anthropic import Anthropic client = Anthropic() def answer_with_self_eval(question): # Step 1: answer normally a = client.messages.create( model="claude-opus-4-8", max_tokens=300, messages=[{"role": "user", "content": question}] ).content[0].text # Step 2: a fresh independent call to assess P(True) of the answer above # independent call avoids it "defending" what it just said check = client.messages.create( model="claude-opus-4-8", max_tokens=10, messages=[{"role": "user", "content": f"Question: {question}\nCandidate answer: {a}\n" "What is the probability this answer is correct? Reply one number 0-1."}] ).content[0].text return a, float(check.strip()) ans, p_true = answer_with_self_eval("Who formulated Bayes' theorem?") # low p_true → trigger human review / route to RAG, don't trust directly