Correction (2026-08-07). A full review of kibsu found three more scorer defects — chiefly that the scorer could not see Title-case directives ("- Must run the tests before merging."), an ordinary style every collection uses. All three are fixed in kibsu 0.3.0 (scorer 0.6.0), and the survey was re-measured at the same pinned commits. This time the headline moved: the median is 9.4%, not 11.1%. Instruction counts grew about 17% in every collection (total 17,361 -> 20,286), and the percentages computed on the larger, truer denominators came down almost everywhere: the range is now 2.1% to 20.9% (was 2.2% to 23.3%). The phantom line 134 / 56 (42%) is now 159 / 69 (43%); "three of eight mandate zero artifacts" holds. My own workspace baseline below moved the same direction (28.65% -> 27.1%), doctrine stayed a hard 0.0% at every pre-registered stage under both instruments, and the lab still scores 4 of 5. Ablation runs pin every moved number on the Title-case fix alone — the other two fixes were real but had nothing to bite in this corpus. Once again the error ran in this article's favor: checkability in the wild is even lower than published, which sharpens the argument below rather than weakening it. Full round, bias directions, and the commands to reproduce both sides: CORRECTIONS.md.
Correction (2026-08-01). An audit of kibsu's own scorer found five measurement bugs. All five are fixed in kibsu 0.2.0, and this survey was re-measured at the same pinned commits with the corrected scorer. The body below is preserved as published; these figures moved: "five of the eight collections mandate zero artifacts" is now three of eight (the corrected scorer finds mandates the old one had filtered out or missed - both newly counted sets turned out phantom); the phantom line "103 distinct / 44 never existed (43%)" is now 134 / 56 (42%); total instructions counted 17,611 -> 17,361; the range's top 22.9% -> 23.3%. The headline median did not move: 11.1%, and the pre-registered baselines reproduce digit-for-digit under both instruments. One of the five bugs had been inflating the phantom rate - the error ran in this article's favor. Every correction, its bias direction, and the commands to reproduce both sides: CORRECTIONS.md.
I didn't set out to build a developer tool. For a long time now I've been working with AI on everything in my life — daily conversations about my daughters, planning projects, ideas for ones that don't exist yet. The goal was always the same: ease my life, get more done, and break the barrier between human and AI — stop treating it as a search box, start treating it as a partner.
Somewhere along the way, the partnership got serious. The workspace where my projects live grew an instruction system for AI coding agents — the files everyone is writing now: AGENTS.md, CLAUDE.md, a skills directory, rules for how agents should plan, log, and verify their work.
Then I asked an uncomfortable question: is any of it actually followed?
Not "do the agents seem to follow it." Could anyone tell, from the repository alone, whether an instruction was followed? For most of my rules, the answer was no. My own audit found that the two checks my instructions said must run before every commit were invoked by nothing — no CI, no hook, no scheduled task. The rule had been enforced, for its entire life, by whoever remembered. Replaying my last 200 commits, the index-freshness rule alone would have failed on 29 of 61 eligible commits — roughly half. My instructions were not rules. They were hopes with formatting.
So I wondered whether everyone else's are too. I wrote a tool and measured.
What I measured, and the two honest limits that come before the numbers
I analysed eight public agent-instruction collections — 1,332 instruction units, 17,611 individual instructions — each at a pinned commit SHA, with the raw per-repo JSON published alongside the tool.
An instruction counts as CHECKABLE if a reviewer could tell from the repo whether it happened: it's a tick-box, or contains a runnable command, or names a concrete file artifact, or refers to an exit code, a diff, an assertion. Everything else is CLAIMABLE — the only evidence it happened is the agent saying so.
Two limits, before any number, because they bound everything below:
First: checkability is not quality. Nobody — including me — has demonstrated that low-checkability instructions produce worse outcomes. The demonstrated finding is narrower and, I think, more important: nobody can tell either way. That is the weaker claim, the more defensible one, and the only one worth acting on.
Second: the metric is deliberately biased upward. Every ambiguous instruction counts as checkable — a bare filename mention counts, a bare command word counts. Every figure below is a ceiling. The true numbers are lower. And the metric reads only documents: one surveyed project ships a separate eval harness that drives real agents and grades them — real enforceability this method cannot see. Document analysis has a floor it cannot measure below.
The numbers
Across the eight collections, the median proportion of procedural instructions that are checkable is 11.1%. The range: 2.2% to 22.9%. The best collection in the sample still leaves three-quarters of its procedural instructions unverifiable; the weakest leaves 98%.
Two findings surprised me more than the median.
Five of the eight collections mandate zero output artifacts. Not "promised and missing" — never promised anything at all. No file their instructions require to exist. Nothing that survives the session. Nothing a reviewer could ever audit.
The other three do promise artifacts — and of the 103 distinct artifacts they mandate, 44 have never existed in any commit of their repositories. Forty-three percent. I call these phantom artifacts: an instruction no model has ever been caught skipping, because nothing was ever looking. (The check walks full git history, not just the current tree — an artifact that existed once and was deleted is not a phantom.)
So the field splits into two failure modes: collections that make no verifiable claims at all, and collections whose claims fail at roughly the rate of a coin flip weighted for optimism.
One genre note, because scoring everything on checkability would be the obvious way to get this wrong: instructions about how to think — "name the assumption before building" — produce judgment, not files. They score ~0% checkable, and that is the genre working, not a defect. The headline figures are procedure-only for exactly that reason.
The tool
The measurement tool is kibsu — open source, zero dependencies, Python 3.8+, and it writes nothing to the repositories it reads:
pip install kibsu
python -m kibsu report /path/to/any/repo
It ends every report with "Nothing was written to this repo — run git status to confirm", and that line is the design philosophy in one sentence. The survey's raw evidence, with pinned SHAs per repository, is in the repo: https://github.com/M-Bajalan/kibsu
The first repository it publicly diagnosed was its own, and the verdict was "0 of 5 ready." It launched by telling the truth about itself. I intend to keep it that way.
Update (Jul 28): there's now a companion lab — a complete, cloneable agent-governed
workspace (fictional company, seeded data, commit gates, machine-checkable tasks) that
scores 4 of 5 on kibsu's own report, with the command trail proving it:
https://github.com/M-Bajalan/kibsu-lab — point any coding agent at its AGENTS.md.
What happens next — and how you'll catch me if I cheat
My own workspace's instruction system measures 28.65% procedure-only checkable — about 2.6× the public median, and still failing seven of ten instructions. The plan is to improve it using the tool's own findings and publish the before/after.
That plan has an obvious attack: "you optimised for your own metric." It's partly correct and can't be answered with numbers — so I've pre-registered the terms instead. The baseline is committed, pinned to a SHA, recorded before a single skill was improved. The prediction: procedure-genre scores will rise; doctrine-genre scores are expected to stay ≈0 — because doctrine produces judgment, not artifacts, and a rising doctrine score would mean I gamed the metric. Every rewrite publishes as a text diff, so you judge each change rather than trusting a percentage.
Update (Aug 8): the pre-registered experiment ran — and then the instrument changed twice underneath it, which is exactly the scenario the pre-registration existed for. Cycle 1's improvement survived both scorer corrections (28.65% -> 31.4% under the instrument that measured it; 27.1% -> 29.3% under the corrected one), and doctrine-genre checkability stayed a hard 0.0% at every stage under every instrument — the disqualification clause never fired. Both re-baselines are in PREREGISTRATION.md, every moved number in CORRECTIONS.md.
The tool is Act 1 of three: diagnose. Act 2 builds what's missing — the maps, indexes, logs, and commit gates that let any agent walk into a repository cold and continue from the last task. Act 3 measures whether it worked, across commits, over time. Almost no developer tool measures its own effect. That, more than any number above, is the point.
One more reason all of this is public. Everything I know about working with AI, I learned from people who shared what they found — I take from community learning all the time. When I saw what this way of working did for my own life, I wished everyone could use it and see the difference. This is my part, given back.
kibsu is Akkadian, from kabāsu, "to tread": the track left behind, the course of conduct, the way of calculation. The dictionary entry turned out to be the architecture.
Built with AI coding assistants under my direction, gating, and review — the commit trailers record which ones, because a project about verifiable claims shouldn't have an unverifiable author line.
Top comments (11)
That title nails a real problem. Agent instructions often read like contracts, but unless something checks them, they are closer to wishes.
I like converting the most important instructions into observable assertions: did it read the repo, did it avoid unrelated files, did it run the focused test, did it preserve the user's changes? The instruction becomes much stronger when the environment can catch a miss.
"Closer to wishes" — that's the finding in three words. Your observable-assertions list is
the stronger form of enforcement, and worth being explicit about: kibsu deliberately measures
only what a reviewer can check from the repository itself, so runtime assertion harnesses
like you describe sit above what it can see (the article's Honest Limits section names
exactly this). If you want to see the instruction→assertion conversion done end to end,
that's what the companion lab is: three agent tasks, each with a machine-checkable done-state
— including one that verifies the agent continued half-finished work instead of restarting
it. github.com/M-Bajalan/kibsu-lab
That observable-assertions framing is the key. Instructions only become operational when you can point to a trace and say: this rule fired, this rule was skipped, this exception was logged. Otherwise the instruction is just a nice sentence in the prompt.
The trace is the part nobody ships. Everyone writes the rule — me very much included — but almost nobody logs "rule fired / rule skipped." The closest thing I have is a pre-commit gate that prints which rule blocked and why, and two nights ago it blocked the very agent that helped build kibsu, over an artifact-cap rule it had helped write. That one logged line settled an argument no percentage in my article could. Do you log rule-firings in your own setup today, or is this a gap you're seeing from the inside too? I haven't yet found anyone who ships that trace.
This is a gap I see from the inside too. I have traces and reports for some workflows, but true rule-firing logs are still uneven. The strongest version would be boring and mechanical: rule id, evidence found, action taken, skipped reason, and artifact link. Your pre-commit example is the right shape.
Boring and mechanical" is exactly what a trace format should aspire to be. That five-field shape — rule id, evidence found, action taken, skipped reason, artifact link — is going into kibsu's queue as the spec for a rule-firing log, and this thread gets the credit when it ships. Thank you for sharpening it.
Love that. A rule-firing log sounds unglamorous, but it is the kind of artifact that makes agent behavior reviewable after the fact. If it ships, I would especially watch whether skipped reasons get written as clearly as successful actions.
Your watch-point is now literally the acceptance criterion: it's recorded on the kibsu issue that carries your five-field schema (kibsu#12), with a proposed blind test — show a reviewer a fired entry and a skipped entry; if they can tell which is which by detail level rather than by the action field, the skipped side is underwritten. This thread keeps writing the spec.
Since you'd appreciate the shape of it: kibsu 0.2.0 shipped this week as a correction release — an audit of the scorer itself found five measurement bugs, and the fixes went through five rounds of independent fresh-eyes verification. Eleven fix commits, and not one measurement changed; every defect after the first round was in the prose describing the numbers. The article above now carries the correction note, and the repo has a CORRECTIONS.md indexing every moved figure with its bias direction. The strongest lines in the verifier's own reports turned out to be the ones recording what it didn't check and why — your point, demonstrated on us before the trace log even exists.
That blind test is a strong acceptance criterion. If a reviewer cannot tell a fired rule from a skipped rule by the quality of the evidence, the log is just theater. The skipped side needs as much accountability as the fired side.
I ran the same audit on my own repo and landed close to your number. An instruction the agent can't be measured against is really just a preference, and it drifts the moment nobody's watching. The handful that actually held were the ones I'd turned into a check that fails the build. The rest sat in the file looking authoritative and doing nothing. Documentation is fine, as long as it isn't counted as enforcement.
Running it on your own repo and landing near the median is exactly the replication this
needed — thank you. And your observation matches what the data showed at scale: the promises
that survive are the ones wired to something that can fail. One nuance the tool tries hard to
respect: not everything should be checkable — instructions about how to think score ~0% and
that's the genre working, not failing. The problem is only when unbacked prose gets counted
as enforcement, which is your closing line exactly. If you're open to sharing your number
(even roughly) I'd love to collect community data points next to the survey's eight.