A viral post made the rounds recently: "How to Make Claude Code Stop Making Stuff Up When It Doesn't Know." It described a 4-layer setup — honesty rules in CLAUDE.md, verification protocols, linter hooks, and a fact-checker subagent — to catch Claude fabricating functions, faking test results, and confidently delivering nonsense.
The post was good. The problem it named is real. But the solution is a duct-tape stack, and every layer has a gap the next one is supposed to cover. Four moving parts, all configured by the user, all depending on the agent's willingness to follow the rules that are supposed to catch it breaking the rules.
I built a single hook that replaces all four. Here's why the layers don't hold, and what does.
What the 4 layers actually are
Layer 1: Honesty rules in CLAUDE.md. Tell Claude to admit uncertainty, cite file:line when referencing code, and ask before adding dependencies.
Layer 2: Verification protocol. More CLAUDE.md instructions forcing Claude to check files exist before importing them.
Layer 3: Linter/type-checker hooks. A PostToolUse hook that runs tsc or a linter after every file write. If Claude invented an import, the checker fails.
Layer 4: Fact-checker subagent. A second agent invoked before commits whose job is to verify claims made by the first.
Each layer is reasonable in isolation. Together, they share a structural problem.
The structural problem: layers 1, 2, and 4 ask the agent to police itself
CLAUDE.md rules (layers 1 and 2) work by telling Claude to behave differently. But Claude's fabrication problem exists because it's optimised to look helpful, and admitting uncertainty feels unhelpful. You're asking the same optimisation that causes the lie to also catch the lie. Sometimes it works. Sometimes Claude reads the rule, weighs it against the pressure to look competent, and guesses anyway. The viral post itself acknowledges this: "Punish honesty once and Claude goes back to guessing."
Layer 4 — the fact-checker subagent — is a second model reviewing the first. This is the "two AI reviewers" trap I've written about before: if both models share the same training, the same helpfulness pressure, and the same blind spots, the second one can be talked out of a finding just as easily as the first. A model reviewer is not an independent check. It's a correlated one.
Only layer 3 — the linter hook — is genuinely independent. tsc doesn't care what Claude claimed. It reads the file and either the types resolve or they don't. That layer works because it doesn't involve a model.
What a single deterministic hook does instead
I built Groundtruth around that same principle: nothing reads the code, so nothing can be talked out of the verdict.
It's a Claude Code Stop hook. When the agent says "done," Groundtruth reads three things from outside the agent's control — the request, the transcript (what the agent actually ran), and git diff HEAD (what actually changed) — and renders a verdict before the turn ends.
No LLM. No network. No API key. One hook, not four layers.
Here's what it catches, mapped against the 4-layer stack:
"Tests pass" — when nothing ran
The viral article's fix: a Stop hook that runs the test suite before Claude can declare done.
Groundtruth's approach: it checks whether a test command actually executed in the session transcript. If Claude claims "tests pass" but no test runner appeared in the Bash history, it fires:
🔴 false test/build claim — "tests pass", but no test command ran this session
The difference: the article's hook runs the tests (which costs time and can fail for environment reasons). Groundtruth checks whether they were run. If they weren't, the claim is false regardless of what the tests would have shown. Both are valid; mine is cheaper and catches the specific lie — "I ran the tests" when you didn't.
Made-up imports
The article's fix: tsc as a PostToolUse hook catches fabricated imports when the type checker fails.
Groundtruth's approach: for JS/TS (where imports are path-relative), it resolves the import path against the file tree. If the target doesn't exist, it fires:
🟡 phantom ref — new import of './utils/tokenHelper' but that file doesn't exist
For package-qualified languages (Python, Go, Rust, Java, C#), it abstains rather than false-flag — because a missing package import isn't provably wrong from the file tree alone. This is deliberate: a check that false-fires teaches you to ignore the card, which breaks everything.
"Done" — but the file isn't in the diff
The article doesn't cover this. Groundtruth does:
🟡 silent no-op — claimed src/upload.test.js, but it is absent from the diff
The agent said it created a file. The diff says it didn't. No model needed to catch that.
Rules your agent could see and overrode anyway
The article's layer 1 puts rules in CLAUDE.md and hopes Claude follows them. Groundtruth compiles rules from your docs into deterministic predicates — your doc says "use pnpm not npm," so a regex checks the diff for npm install. The agent can't rationalise past a regex.
These rules are proposed, never auto-armed. You review them with /groundtruth-rules and approve what's correct. If one fires wrongly, silence it by id. The permission gate is yours, not the model's.
Stubs and placeholders
The article doesn't catch // TODO: implement this or raise NotImplementedError left in added code. Groundtruth does:
🟡 stub/placeholder in added code: // TODO: real backoff — single attempt for now
If the agent claims a feature is done and the diff contains a placeholder, those two facts contradict. No judgment call needed.
Why "don't involve a model" matters
The viral article's own best insight — buried at the end — is that the linter hook works because it doesn't involve a model. That's the whole thesis of Groundtruth, applied to every check instead of just one.
A model reviewer (layer 4) can be reasoned out of a finding. Claude is very good at explaining why a seemingly-wrong thing is actually fine. "The test file wasn't needed because the function is trivial" sounds plausible to a second model that shares the same optimisation for helpfulness. It doesn't sound plausible to a diff that simply notes the file isn't there.
The 4-layer approach works partially because 25% of it is genuinely independent (the linter). Groundtruth works fully because 100% of it is.
What Groundtruth does NOT do
I'll be precise, because overclaiming is the exact failure mode this tool is built to catch.
- It does not judge code quality. It doesn't know if your algorithm is correct. It knows if the agent claimed it was tested when it wasn't.
- It does not do semantic evaluation. "Did the agent rationalise past a rule" requires understanding intent, which requires a model. That's roadmap, not shipped.
- It does not replace testing. It catches the lie that testing happened. You still need the tests.
- It runs in the same trust domain as the agent — tamper-evident, not tamper-proof. The honest security scope is in SECURITY.md.
The honest comparison
| Check | 4-layer manual setup | Groundtruth |
|---|---|---|
| "Tests pass" with no test run | Stop hook that runs tests | Checks transcript for test execution |
| Fabricated import |
tsc PostToolUse hook |
Path resolution against file tree (JS/TS); abstains elsewhere |
| File claimed but not created | ❌ Not covered | ✅ silent no-op detection |
| Stub/TODO left in "done" code | ❌ Not covered | ✅ placeholder scan on added lines |
| Project rules violated | CLAUDE.md (hope-based) | Deterministic regex from your docs |
| Fact-checking claims | Subagent (model-based, correlated) | Diff-anchored (no model, uncorrelated) |
| Config required | CLAUDE.md edits + hooks JSON + subagent file |
/plugin marketplace add + restart |
| Can the agent talk its way out? | Layers 1, 2, 4: yes. Layer 3: no. | No. |
Try it
/plugin marketplace add akahkhanna/groundtruth
/plugin install groundtruth@groundtruth
Restart Claude Code. Every turn gets a warn-only verdict card. No config needed. Arm your project rules with /groundtruth-rules approve-all when you're ready. Turn on blocking with /groundtruth-block on once you trust the precision.
MIT. No LLM. No network. No API key.
The 4-layer article was right about the problem. The fix is one layer, not four — and the layer that works is the one that doesn't involve a model.
I write these while building Groundtruth and EraPin. The 74% false-positive story — how we found it, how we killed it — is in FIXES.md.
Top comments (6)
The correlated-reviewer point is the load-bearing one, and it's why the single hook wins for the mechanical class: tsc reads the file, not Claude's story about it. But watch the boundary you just drew. A linter covers invented imports and type errors end to end, which is exactly the class where the agent's self-report was never needed anyway. It's structurally blind to the semantic lies: picking a worse-but-passing approach, or claiming a test passed when the test asserts nothing. So the hook doesn't replace layer 4's job, it replaces its implementation. You still need an independent check for 'did the claim match reality,' it just can't be another model that shares the first one's blind spots. The version that holds there isn't a reviewer at all, it's execution: run the test and diff the actual output against the claim, fail closed on mismatch. Ground truth over a second opinion. Curious where your hook draws the line between 'checkable by the toolchain' and 'only checkable by running it.'
You’re right on both counts, and I want to be precise about where I agree and where the line actually sits in the implementation.
You’re correct that the hook doesn’t replace layer 4’s job — it replaces its implementation. The need for an independent “did the claim match reality” check doesn’t go away. What changes is the mechanism: execution evidence over a second opinion. That’s exactly how Groundtruth is built — it reads the transcript (what actually ran) and the diff (what actually changed), not the agent’s narrative about either.
On the boundary you’re asking about — “checkable by the toolchain” vs “only checkable by running it” — that’s an explicit, documented line in the architecture.
The deterministic engine (shipped, per-turn) handles the mechanical class: claim vs diff, presence of a test run in the transcript, stubs in added code, phantom imports via path resolution, directive violations via regex. These are all cases where the claim can be checked against an artifact the agent can’t author — the diff or the transcript.
The semantic class — picking a worse-but-passing approach, a test that passes but asserts nothing, an agent that rationalised past a rule it could see — needs judgment, which means a model. That’s roadmap, explicitly not shipped, and I’ve been careful not to claim it because overclaiming is the exact failure mode the tool is built to catch.
So the honest answer to your question: the line is drawn at “can a deterministic predicate over the diff/transcript decide this without a judgment call.” Everything below that line is shipped. Everything above it is labeled roadmap and waiting for a semantic layer that I’m not willing to ship until its false-positive rate is proven — because a check that false-fires teaches users to ignore the card, and then it catches nothing.
Your framing — “ground truth over a second opinion” — is better than mine. I might steal that.
This is the cleanest statement of that boundary I've seen, and the roadmap honesty is the rare part. One push on where the line sits: part of what you filed under 'semantic, needs a model' is still deterministic, it just needs more execution rather than more judgment. 'A test that passes but asserts nothing' has an oracle, mutate the code under test and check the test still fails. Vacuous assertion, no judgment call, it just costs a second run. So the line might be 'checkable against an artifact you already hold' (diff, transcript) vs 'checkable only by producing a new one' (mutation, property fuzz), with the model tier sitting above both. What survives even that is the genuinely oracle-free class: 'picked a worse-but-passing approach.' Worse relative to what? The intent was never written down, so there's no artifact to diff and the model is scoring against a spec that doesn't exist. That's the one I'd ship last too. Worth treating 'run more to check' as its own tier before 'ask a model,' since they fail in opposite directions: mutation over-reports, the judge under-reports.
Fair catch — that's a real misfiling in my own roadmap (9d and regression-blind are execution-checkable, not judgment). "Run more to check" is its own tier. One nuance: a produced artifact is computed by the agent's own code, so it isn't a trust anchor the way the diff and transcript are — a test asserting on a hash of the source file gets a perfect mutation score while asserting nothing. So I'd call the tier empirical, not deterministic (mutation is flaky and side-effectful), and put it in the CI rung, not the per-turn hook. Your opposite-directions point is the sharp part: since a false positive is fatal here and mutation over-reports, the model-free tier is the dangerous one — the charter binds by failure direction, not mechanism. The oracle-free residual I'd never ship as a verdict at all; the tool checks you did what you said, not that it was right. Though the cheap end is real: a just-added test with no calls in its body provably can't fail, and there's nothing to delegate to — that ships as a warn.
The abstain-on-package-imports choice is the right call - a checker that false-fires gets ignored within a week, and then it catches nothing. Two edges I would test against the transcript check: does it verify the test command's exit code, and that the run happened AFTER the last write? "Tests pass" can be technically true from a run that predates the final edit, and presence-only matching would bless that stale green. Partial runs are the same shape - a filtered
npm test -- --grep fooshows up in Bash history and reads as "tests ran" while covering one file. Ordering (last test run postdates last mutation) plus exit status looks like the minimal pair that closes both without reintroducing a model.@skillselion — spot on, both were open. The exit code was captured on every tool result and simply never read — failure was inferred from a stdout string, so an OOM'd Killed passed as green. Ordering was structurally impossible until I rebuilt the transcript parser to pair and sequence-stamp calls; "last run postdates last mutation, plus exit status" is exactly the pair it became. Partial runs get the same treatment — fires only on an "all tests pass" claim where every run was narrowed. All shipped as warn-with-abstain. Thanks for the read — it even flushed out a matching bug in multi-agent setups.