Correction (2026-08-07). A full review of kibsu found three more scorer defects — chiefly that the scorer could not see Title-case directives ("- ...
For further actions, you may consider blocking this person and/or reporting abuse
That title nails a real problem. Agent instructions often read like contracts, but unless something checks them, they are closer to wishes.
I like converting the most important instructions into observable assertions: did it read the repo, did it avoid unrelated files, did it run the focused test, did it preserve the user's changes? The instruction becomes much stronger when the environment can catch a miss.
"Closer to wishes" — that's the finding in three words. Your observable-assertions list is
the stronger form of enforcement, and worth being explicit about: kibsu deliberately measures
only what a reviewer can check from the repository itself, so runtime assertion harnesses
like you describe sit above what it can see (the article's Honest Limits section names
exactly this). If you want to see the instruction→assertion conversion done end to end,
that's what the companion lab is: three agent tasks, each with a machine-checkable done-state
— including one that verifies the agent continued half-finished work instead of restarting
it. github.com/M-Bajalan/kibsu-lab
That observable-assertions framing is the key. Instructions only become operational when you can point to a trace and say: this rule fired, this rule was skipped, this exception was logged. Otherwise the instruction is just a nice sentence in the prompt.
The trace is the part nobody ships. Everyone writes the rule — me very much included — but almost nobody logs "rule fired / rule skipped." The closest thing I have is a pre-commit gate that prints which rule blocked and why, and two nights ago it blocked the very agent that helped build kibsu, over an artifact-cap rule it had helped write. That one logged line settled an argument no percentage in my article could. Do you log rule-firings in your own setup today, or is this a gap you're seeing from the inside too? I haven't yet found anyone who ships that trace.
This is a gap I see from the inside too. I have traces and reports for some workflows, but true rule-firing logs are still uneven. The strongest version would be boring and mechanical: rule id, evidence found, action taken, skipped reason, and artifact link. Your pre-commit example is the right shape.
Boring and mechanical" is exactly what a trace format should aspire to be. That five-field shape — rule id, evidence found, action taken, skipped reason, artifact link — is going into kibsu's queue as the spec for a rule-firing log, and this thread gets the credit when it ships. Thank you for sharpening it.
I ran the same audit on my own repo and landed close to your number. An instruction the agent can't be measured against is really just a preference, and it drifts the moment nobody's watching. The handful that actually held were the ones I'd turned into a check that fails the build. The rest sat in the file looking authoritative and doing nothing. Documentation is fine, as long as it isn't counted as enforcement.
Running it on your own repo and landing near the median is exactly the replication this
needed — thank you. And your observation matches what the data showed at scale: the promises
that survive are the ones wired to something that can fail. One nuance the tool tries hard to
respect: not everything should be checkable — instructions about how to think score ~0% and
that's the genre working, not failing. The problem is only when unbacked prose gets counted
as enforcement, which is your closing line exactly. If you're open to sharing your number
(even roughly) I'd love to collect community data points next to the survey's eight.