SKILL 31 / SAFETY · DEEP DIVE

Truth-First Skills

Making it say "I don't know," making it run the command before saying "done" — and why this whole class fails open

2026-10-02 · BigCat

An unglamorous opening: truth-first does exist as an Agensi listing, but a search engine attributed it to Addy Osmani's agent-skills — and that repo's 25 skill directories contain no such folder. Which is exactly the disease this class of skill treats. Below: 9 real, installable or readable entries with install commands and a first usable prompt, plus the one criterion worth memorizing — no "unknown" tier means it is binary, and binary fails.

Same series → Skill 21 Source Grounding · Skill 7 Writing Evals for a Skill · Skill 27 Hooks & Deterministic Automation

// WHY THIS MATTERS

A model lies in two ways: it invents facts (citations, version numbers, URLs) and it invents completion ("tests should pass now"). Skill 21 handed the first to tool grounding; this issue takes the second, plus something harder — making "I don't know" a legal value in the output contract. That is the entire craft here: these skills do not change what the model knows, only the precondition for it to speak.

// 01

What it kills for you: turning "I think it's fine" into "the gate is shut, so I may not say fine"

Claim: the four that actually earn their keep all install within one screen. Commands first; the criteria come in the second half.
# 1. The headliner: truth-first (an Agensi marketplace listing)
npx agensi install truth-first

# 2. The hardest completion gate on the official marketplace (one skill inside Superpowers)
/plugin install superpowers@claude-plugins-official

# 3. "Read the official docs first" and "get disproved first"
npx skills add addyosmani/agent-skills --list   # see the 25 folder names first
npx skills add addyosmani/agent-skills

# 4. The CLAUDE.md school: paste in permission to say I don't know
/plugin marketplace add instantX-research/anthropic-anti-hallucinate-skills
/plugin install anti-hallucinate@anthropic-anti-hallucinate-skills

The first usable move

Don't wait for auto-trigger. Next time it says "should," paste this back:

You just said the tests "should" pass. Run verification-before-completion:
which command proves that sentence? Run it in full in this message,
paste the exit code and the failure count, then state a conclusion.

What each of the four kills, and when not to install it

An SEO site filed truth-first under Addy Osmani — that is the counter-example itself. Go look at addyosmani/agent-skills: 25 skill folders, no truth-first. So this series' rule matters doubly here: aggregator sites are for discovering candidates only; names come from the GitHub repo directory or the official marketplace page.
// 02

Mechanism: three designs, and not one of them is "be careful"

Claim: none of the effective truth-first skills says "please be careful." They write a gate, a third tier, and a table that pre-rejects excuses.

1. The gate function: a claim must pass through

BEFORE saying "done," you must complete: 1 IDENTIFY which command proves this sentence? 2 RUN run it in full (fresh, not last time's) 3 READ full output, check exit code, count failures 4 VERIFY does the output actually support the claim? - no -> state the real status, with evidence - yes -> state the claim, with evidence 5 ONLY THEN are you allowed to speak Skipping any step = lying, not verifying

This beats "please verify first" on one point: speaking itself becomes a governed action, so "Great!" and "Perfect!" fall inside the gate too — that SKILL.md lists expressing satisfaction among its red flags.

2. Three tiers: binary squeezes "unknown" into "known"

This is the most reusable idea in the issue. With only right/wrong available, anything uncertain must be filed under one of them, and the model's default bias is "right." So every serious attempt adds a third tier: truth-first's Verified / Inferred / Unknown, verify-claim's Verified / Refuted / Inconclusive, source-driven-development's UNVERIFIED. The official side is blunter: the first basic strategy on Reduce hallucinations is literally Allow Claude to say "I don't know", with a sentence you can copy verbatim — "If you're unsure about any aspect or if the report lacks necessary information, say 'I don't have enough information to confidently assess this.'" (per that page, 2026-10).

3. The rationalization table: write down every excuse in advance

Both of the skills above carry a two-column table: excuse on the left, rebuttal on the right. "Should work now" → run it. "I'm confident" → confidence is not evidence. "Just this once" → no exceptions. "Different words, so the rule doesn't apply" → spirit over letter. And the sharpest: "fetching docs wastes tokens" → hallucinating an API wastes more.

Why write it that way? Because the common failure of a rule like this is not incomprehension, it is circumvention: rephrase it, verify half of it, "this case is special." Writing the dodges into the skill closes the exits in advance — the fastest signal of whether one of these was made seriously.
// 03

Use and limits: five more entries, and how to accept-test them

Claim: don't install three of the same kind — they differ in trigger moment and in surface, and overlapping ones just fight. Sort out who owns which segment.

Only anti-hallucinate below ships install commands; the other four are single-folder skills — copy that folder from the repo into ~/.claude/skills/ or the project's .claude/skills/.

Four criteria, for choosing among the nine above

  1. Only a reverse eval counts as acceptance: anything passes the positive cases. What you must construct are three cases where the evidence does not exist — a function in a file that isn't there, a version number that can't be looked up, confirming tests pass in an environment that won't run. One pass condition: it says it doesn't know. Run the no-skill baseline first, per Skill 7, or you cannot measure what this one added.
  2. Is there a third tier: the output spec must carry Unknown / Inconclusive / UNVERIFIED. "Verified" with no "couldn't verify" is binary — drop it.
  3. Is there a rationalization table and a red-flag list: the hard sign of serious work (see above). Principles without closed exits get dodged by tomorrow with "this case is special."
  4. Which side its trigger sits on: read the description in its frontmatter. "Before claiming complete / committing / opening a PR" governs invented completion; "when the user asks for facts" governs invented facts. Don't expect one skill to cover both — what you stack is different segments, not duplicates.
A real way to get this wrong: install truth-first and verification-before-completion together, then complain that every sentence needs a command. Both want evidence, but one wants system state (configs, logs) and the other wants an exit code. If you aren't debugging infrastructure, turn the former off.
// 04

Cost and alternatives: this class fails open, so the real gate isn't here

Claim: Skill 29's rule that safety skills must fail closed is violated right here — these are all prompt text, the model may ignore them, and nothing errors when it does.

Two bills to accept up front

The ladder of alternatives: harder as you go down, and pricier

one prompt the official "say I don't know" line; free, this turn only ↓ CLAUDE.md persistent across the project; costs context permanently ↓ skill triggerable, room for a long spec; fails open (all nine live here) ↓ hook Stop event, exit 2 / {"decision":"block"}; the model cannot dodge it ↓ CI the only layer that stops what was already pushed

How to go one layer down: Claude Code's Stop hook fires when it wants to finish, and exit code 2 prevents it from stopping and continues the conversation; the JSON form is {"decision":"block","reason":"..."}, and hookSpecificOutput.additionalContext can feed "3 test failures remain, continue" straight back in (per the official hooks reference, 2026-10). In other words: "did it run the tests" belongs to a hook, not to a skill.

Putting uncertainty into the output contract

If this lives in your own program rather than a CLI chat, don't lean on prompting — lean on schema. Structured Outputs pins the response to your schema via output_config.format (type: "json_schema"), so status: "verified" | "inferred" | "unknown" and evidence can be required fields — no confident prose can route around the unknown tier any more. Limits worth knowing: no recursive schemas, no minimum/maxLength-style constraints, additionalProperties must be false, and output may not match the schema on a refusal or when max_tokens is hit (per that page, 2026-10).

// ONE-PAGE ACTION CARD

Do these three today: 1. /plugin install superpowers@claude-plugins-official, then invoke verification-before-completion by hand right before it would say "done" — the most direct payoff of the nine. 2. Copy the official line into your CLAUDE.md: "If you're unsure … say 'I don't have enough information to confidently assess this.'" — free, project-wide. 3. Write three reverse-eval cases where the evidence does not exist, and run the no-skill baseline first.

Add as needed: source-driven-development (framework code) · doubt-driven-development (irreversible or cross-module decisions) · truth-first (debugging and system-state questions; read its SKILL.md first) · discernment-nudge (answers going to non-technical readers) · fact-checker (numbers and quotes in a draft) · one Stop hook (pin "no finishing without a test run" there).

Explicitly don't: install two of the same evidence-demanding kind · treat this class as a CI gate (it fails open) · copy skill names or authors off aggregator sites · accept-test with positive cases only · run a completion gate over exploratory scripts · think "I noted it might be outdated" equals having checked.

// GLOSSARY

false completion
Claiming tests passed, the bug is fixed, or the build succeeded without running the verification. The sibling of invented facts; this issue treats it.
three-tier grading
Output tiers like Verified / Inferred / Unknown. The third one is the point — binary forces the uncertain into "certain."
fail-open / fail-closed
Letting through on failure versus blocking on failure. A skill is the former (ignoring it raises nothing); hooks and CI are the latter.
rationalization table
The two-column "excuse to rebuttal" table inside a SKILL.md, pre-closing rephrasing, half-verification and "just this once."
Structured Outputs
The Claude API's output_config.format (json_schema), pinning output to a schema; it can make "unknown" a required field.

// THREE THINGS THAT DON'T ADD UP

If the model can ignore it, is this class of skill worth anything?
Yes, but the value is not a guarantee — it is a changed default. With no external constraint the model's default is to produce an answer that sounds complete; these skills move the default to "find the evidence first." The right mental slot for one is a checklist on call, not a door. Which yields a counter-intuitive conclusion: invoking it by hand beats waiting for auto-trigger — calling it the moment you read the word "should" hits far more often than hoping it fires itself at exactly that moment. If you want a door, go down a layer to a Stop hook.
Won't truth-first grading just give me something that marks everything Unknown?
It can, and this is the most common real failure of the class, not a theoretical risk: you rewarded "I don't know," so it treats not knowing as the safe exit. Two counters, both inside things named above — turn it on only for high-stakes work (verify-claim says to skip low stakes outright), and don't test only whether it will say it doesn't know. The reverse eval has to run both ways: build three more cases where the evidence is entirely sufficient and see whether it still dares to conclude. One-directional acceptance reliably produces an assistant that hands every question back to you — that is not honesty, it is deflection.
Why is "make it say I don't know" so much harder to accept-test than "make it check"?
Because checking leaves traces and not knowing doesn't. A check means tool calls, fetched URLs, command output — countable in the transcript. Whereas "it should have said it didn't know and didn't" is only detectable if you already know the right answer, which is precisely what you lack in the situations where you'd use it. So acceptance here requires manufacturing the evidence gap yourself: you remove the evidence by hand, which makes you the only party who knows the answer key. That also explains why opinion on this class is the most divided of any — most people cannot test it and go on feel.

// FURTHER READING