A few weeks ago I asked an AI coding agent to add rate limiting to an API.
Thirty seconds later: "Done. All 14 tests pass."
Three tests passed. The other eleven did not exist.
This is not a model-quality complaint. The agent wasn't malicious; it was
doing what agents do: reporting the outcome it intended, not the outcome it
measured. And once you notice it, you see it everywhere in AI-generated
codebases. READMEs saying "fully tested" over a single smoke test. "Build
passes" committed one hour before a dependency bump broke the build. The
docs claim; nobody replays.
Humans do this too, but humans have a natural brake: writing "14 tests pass"
in a README when you know it's 3 feels like lying. Agents have no such
brake. The claim is just text that matched the pattern of success.
Replaying the claims
I built Umbra partly to make this measurable. Its HONEST axis reads the
claims a repo makes (in READMEs, changelogs, agent artifacts like
CLAUDE.md), then replays them against reality in a locked-down Docker
sandbox:
$ npx umbra-scan --deep
Claim receipts:
CLAIM FAILED: "14 tests pass" — README.md:7 — actually 3 tests pass, 0 fail
CLAIM FAILED: "build passes" — README.md:9 — actually build exits 1
CLAIM VERIFIED: "All tests pass" — CLAUDE.md:3 — 3 tests pass
Every claim gets a receipt. And because a repo that lies about its tests
will lie about anything, a verified-false claim caps the whole trust score
below passing. We call it the liar cap. Harsh? Maybe. But trust is the
product.
Why this matters more than another linter
Static analysis answers "is this pattern dangerous?" That question was
designed for a world where a human read the code before merging it. The
vibe-coding world has a different failure mode: nobody read it, and the
entity that wrote it also wrote the status report.
So Umbra's score mixes four axes: SAFE (the static rules: leaked keys,
missing RLS, alg:none JWTs), CLEAN (the slop: dead exports, unused deps),
RUNS (does it actually build and boot, verified in a sandbox), and HONEST
(the claims, replayed). The last two don't exist in traditional tools
because traditional tools never had to ask "but did anyone check?"
Try it on the last thing your agent wrote
cd your-project
npx umbra-scan --deep # needs Docker for the sandbox
Open source, MIT, fully local. If it catches your agent lying, I want to
hear the story. If it accuses a repo unfairly, that's a severity-one bug:
file it.
Top comments (0)