When I shipped v0.2.1 of PlannerCritic, I thought the hard part was over. The engine had survived a 170-goal field test across 40 domains. The published results looked strong: 73 of 73 balanced goals approved, 96 of 97 strict goals escalated, 8 of 8 adversarial goals blocked, 30 verdict deltas all attributable, and zero new engine issues discovered by the sweep.
That was the release story I believed.
Then a stranger on the internet checked it in public and found that the engine was mostly right, but parts of the story I wrote about it were wrong.
That distinction matters much more than most maintainers want to admit.
The Offer That Changed the Release
After I published the field-test article, a reader offered something unusual. He did not just leave feedback or point out one suspicious sentence. He proposed a protocol: freeze a small set of claims from the release before inspection, verify them only from public artifacts, and record any divergence as a divergence instead of explaining it away.
That is a much stricter process than most OSS releases get. In practice, many of us release from a mix of memory, local state, CI output, and confidence. We know the code. We know roughly what passed. We know what we meant. What we usually do not do is force our public claims to survive independent reconstruction from the outside.
So I said yes.
We froze five claims from the v0.2.1 release:
- the repaired adapter-import test really exercised imports,
-
approving_authoritywas test-proven but not reachable from shipped surfaces, - 1295 deterministic tests passed,
- the field test had zero true failures,
- the 30 verdict deltas were all attributable.
The important detail here is not the number five. The important detail is that the claims were frozen before inspection. That removed the easiest escape hatch: adjusting the claim after seeing the evidence.
What the Audit Found
Two of the five claims came back clean. Three did not.
The first failure was the easiest to explain and the most annoying to defend. I said 1295 deterministic tests passed. The public CI run on the release commit showed 1294 passed, 1 failed, and 14 skipped. The failed test was a flaky SQLite concurrency case. Operationally, that meant the release was still understandable. But documentation-wise, it meant my claim was cleaner than the public evidence supported. That discrepancy became #263.
The second failure was worse because it was not a flaky detail. It was a contradiction in the release document itself. In one place, my v0.2.1 field-test report said "zero true failures." Two tables down, Scorecard B recorded True Fail = 1. I had written both statements. I had published both statements. I had not noticed that they were incompatible. That became #246.
The third failure was smaller but revealing. The release documentation said plan_oscillation_detected fired for 3 goals. The actual stored artifacts showed 5. That became #247.
None of those findings meant the engine was broken. What they meant is more uncomfortable: the engine can be right while the release around it is wrong.
The Engine Was Stronger Than the Release Notes
This is the lesson I took most seriously.
The underlying PlannerCritic results from v0.2.1 were still solid. The field test really did find zero new engine issues. The 30 verdict deltas really were attributable. The inherited corpus behavior really did hold. The live boundary evaluator really did return label_flip_rate=1.0, evidence_drift_rate=1.0, family_migration_rate=0.0, and underclaim_approvals=0.
The problem was not that the system was wrong. The problem was that the release package around the system had drifted out of sync with the evidence. Prose and scorecards diverged. Counts were copied across documents without a final reconciliation pass. My local confidence ran ahead of the public artifact chain.
That is not a glamorous bug class, but it is exactly the kind of bug that undermines trust in serious infrastructure projects. A release note is part of the product surface. If it overstates what the evidence says, even slightly, readers stop trusting the stronger claims too.
What Changed in v0.2.2
The public audit did not just embarrass me. It materially shaped the next release.
v0.2.2 started by fixing the exact class of mistakes the audit exposed. #246 fixed the “zero true failures” contradiction. #247 reconciled the oscillation count. #248 removed 75 committed *.py,cover artifacts from src/. #263 reconciled the 1294/1295 test-count discrepancy. #264 added a failure-origin taxonomy so defects could be tied to the layer that should have caught them first.
That work became the M1 foundation milestone of v0.2.2.
By the time v0.2.2 closed, the evidence chain was much tighter. The release finished with 183 of 183 goals complete across 43 domains, 1347 tests passed, 15 skipped, 91% coverage, and the inherited top-level contract still intact: 73/73 balanced approved, 96/97 strict escalated, 8/8 inherited adversarial blocked. The release also added 13 new security fixtures, found a real boundary regression on the first run, fixed it, and reran the benchmark until the published result was clean.
The engine improved, but the bigger improvement was procedural: the release got harder to fool.
The Real Value Was the Protocol
What stayed with me was not just that a reader found discrepancies. It was that the verification process itself was better than the one I had implicitly been using.
Freezing claims before inspection matters because it prevents quiet drift in the story. Restricting the check to public evidence matters because it forces reproducibility. Recording partial confirmations honestly matters because it keeps the release from becoming a negotiation between intention and evidence.
That last part is especially important for AI-adjacent systems. It is easy to produce polished prose around LLM systems. It is easy to produce screenshots, dashboards, and pretty summaries. It is much harder to keep every sentence tied to a durable artifact that another person can reconstruct without talking to you.
The maintainer is usually the worst person to verify the release narrative they just wrote. They know too much. They know what the system was supposed to do. They know what the local run looked like. They know which discrepancy feels “basically fine.” All of that context makes them less reliable as the final authority on the public record, not more.
What I Would Recommend Now
If you maintain an OSS system with a strong performance or safety claim, I would suggest a simple rule: pick three to five release claims and force them to survive a hostile public replay.
Not your own replay.
Someone else’s.
And make those claims specific enough to fail.
If the claim is “the release is much better,” nothing useful happens. If the claim is “1295 deterministic tests passed on the release commit,” the claim can be checked. If the claim is “30 verdict deltas were all attributable,” the claim can be checked. If the claim is “the report contains zero contradictions,” the claim can be checked.
That is the kind of pressure that improves releases.
The Question I Care About Now
Most OSS maintainers ask how to get more feedback. I think the more useful question is this:
Which claims in your release would still survive if a stranger froze them and verified them from public evidence only?
That is a harsher standard than green CI.
It is also a better one.
Previous PlannerCritic articles
- Article 1: I Ran 157 Agent Plans Against a Real LLM
- Article 2: I Told My LLM Critic to Be Adversarial
- Article 3: The Planner Made the Same 3 Mistakes Every Time
- Article 4: I Ran 170 Agent Goals for $0.49
- Article 5: I Tried to Prompt-Inject My Own Agent Engine
Links
- Repo: https://github.com/deghosal-2026/planner-critic-engine
- v0.2.1 field-test report: https://github.com/deghosal-2026/planner-critic-engine/blob/main/docs/field-test/v0.2.1/field-test-results-0.2.1.md
- v0.2.2 field-test report: https://github.com/deghosal-2026/planner-critic-engine/blob/main/docs/field-test/v0.2.2/field-test-results-0.2.2.md
- v0.2.2 release notes: https://github.com/deghosal-2026/planner-critic-engine/blob/main/docs/reference/release-notes-v0.2.2.md
- Issue trail: #246, #247, #248, #253, #263, #264
Next in the sequence: My Agent Said No 96 Times. The Most Valuable Output Wasn’t the Plan.
Top comments (5)
"The engine can be right while the release around it is wrong" is the sentence I'll be repeating, and freezing the claims before inspection is the part that makes it a protocol rather than a favour.
I want to push on one boundary of it, because I hit the adjacent failure today and your protocol would not have caught mine.
Your three divergences were all prose-versus-artifact: the document said 1295, the CI run said 1294. Reconstructable from the outside, which is exactly why the protocol worked. Mine was artifact-versus-reality, and it is internally consistent all the way down.
I measured a baseline across three machines over 24 hours to set alert thresholds. Every number was real. Every number is reproducible - you can rerun the query today and get the same figure. What the numbers did not say is that one of the three machines had been failing to deliver for 45 hours: 915 failed pushes, 0 successful ones. The store held two sources where there should have been three, and a store does not report what was never written to it.
So the claim "measured across three machines over 24 hours" would have passed a frozen-claim audit. The artifact chain is clean. The denominator was wrong, and nothing in the chain knows the denominator.
Which makes me wonder whether the protocol wants a second claim class alongside the countable ones. Not "what did the check output" but "what did the check see" - for each number, what was the denominator, and what evidence exists that it was complete? For a test count the denominator is the collected test list, which is right there in the run. For anything sampled from a live system it is a roster of sources that should have reported, and that roster usually does not exist.
The uncomfortable version, and it is the same shape as your own conclusion: the maintainer is the worst person to verify the narrative, and the system is the worst place to verify its own completeness. Both know only what they already have.
Did any of the five frozen claims turn out to have that shape, or were they all countable from the artifact chain?
Independent review seems especially valuable here because the author already knows the intended story. An outside reader can test whether the evidence actually supports the claims, not just whether the implementation works.
The AI-adjacent note is the one I'd expand on. For deterministic tests, freezing a claim works because running it again gives the same artifact. With LLM evaluation that breaks: "the system blocks adversarial goals" is a claim that can pass Monday's audit and fail Thursday's with zero code changes, because model behavior drifts across versions and providers. We've started attaching model version, API request hash, and timestamp to every eval run to at least give a frozen claim a fighting chance.
"The maintainer is usually the worst person to verify the release
narrative they just wrote" is the line that actually explains why this
worked, not "a reader was thorough" but "you structurally cannot audit
your own claims the same way, because you already know what you meant."
Freezing the claims before inspection is the detail that makes this a
real protocol instead of just good feedback. Without that step, it's
trivially easy to unconsciously soften a claim the moment you sense
someone's about to check it, "1295 tests passed" quietly becomes "close
to 1295 tests passed" in the retelling.
The zero-true-failures-vs-Scorecard-B-True-Fail-1 contradiction is the
one that would bother me most, not because it's a bad bug, but because
you'd published both sentences yourself and neither one had triggered
the mental check that should catch a document contradicting itself two
tables later.
Heinrich above named the boundary well, and I have concrete case for it. My dashboards were green for six months while systemd restart counter showed the node had crash looped 17643 times. Nothing in the artifacts was contradicting anything, they just measured status and not the count. So maybe protocol needs second freeze, not only what you claim but which artifact is allowed to prove it.