My own tool lied to me for three weeks: “declared” and “works” are different claims

Arsen Ask — Fri, 04 Sep 2026 15:57:57 +0000

My repo had 55 passing checks. On someone else's machine, none of them ran.

I write code with agents. Every repo I hand them has an AGENTS.md full of sentences: always run the tests, never commit secrets, no TODO without a filed task. The agent reads them, agrees with them, and then does whatever it wants.

Not out of malice. A sentence has no exit code.

So I built a small thing that turns those sentences into commands. A manifest, .aqk.yml, lists the checks the repository promises to hold. doctor --run reports which of them actually execute right now — as opposed to which are merely declared.

Those are two different claims. This post is about the distance between them, because I fell into it myself.

Three weeks of green

I had 55 self-checks, a green pipeline, and a tidy manifest. Then someone who wasn't me ran the tool for the first time, on their own project: Django, on Windows.

Ten minutes. Four defects. The worst one looked like this.

The tool picks between a native recipe — if ruff or vulture or eslint is installed, use it, it catches more — and a portable one, written in sh, that works everywhere and catches less. It decided whether a program existed like this:

command -v ruff

On Windows the shell is cmd.exe. There is no command builtin there. The answer came back "not installed" for every program on the system. Every check silently fell back to the weakest option, and the run reported green.

Green because nothing ran.

Absence of signal turned out to be indistinguishable from success — the exact failure this whole thing exists to prevent. Fifty-five of my own checks never saw it. Someone else's machine saw it in ten minutes.

Three things worth more than the tool

1. A check that cannot go red is indistinguishable from a working one.

Later I found one of my own tests written so that it could not fail. The scenario it set up made the failing condition unreachable. I noticed by accident, while changing something unrelated.

So the rule is not "write the test first". It's "watch the test fail first". You write the failing check, you run it, you see red, then you write the code. Those are different disciplines, and only one of them proves anything.

2. "It's in the config" is not "it runs."

If you have never watched your own pipeline go red on purpose, you don't know whether it can. Most people have never tried. I hadn't, on Windows — I couldn't, and I quietly let "I can't check that" become "that's probably fine".

3. A badge nobody recomputes is a claim, not a fact.

Compliance badges get drawn once and then live their own life. A month later the README asserts something the repository stopped doing, and nothing anywhere goes red.

So the badge command in my tool refuses to print anything while a gate is failing, and badge --check fails your pipeline on the day the README stops matching the run. My own badge is verified that way on every push. It's the only version of a badge I could justify shipping, given that the entire premise is "don't trust claims, run them."

If you want to look

npx agent-quality-kit doctor

It only reads. It writes no file and sends nothing anywhere — you can point it at a repository you haven't decided anything about. Node 18+, zero dependencies on purpose: a tool people install into someone else's repo with one command should be readable in one sitting. MIT.

https://github.com/arsen-ask-lx/Agent_Quality_Kit

Honestly, where this stands

It has one real user besides me. Everything I could tell you about it being useful is therefore unproven, and I would rather say that than imply adoption I don't have.

Which is also the ask. If you run it and it lies to you, an issue is worth more to me than a star — the last person who did that found four bugs in ten minutes, and every one of them was real.

DEV Community: Arsen Ask

My own tool lied to me for three weeks: “declared” and “works” are different claims

Three weeks of green

Three things worth more than the tool

If you want to look

Honestly, where this stands