I stopped trusting "done."
One of my agents reported three completed actions. Two were false. The model produced a convincing final answer. The repository did not match it.
The tool trace looked busy. The summary sounded sure. Nothing in that chat proved the side effects landed.
So I made a rule for the operation: every material completion needs a check that can fail.
That layer is now open source as showwork. It is on PyPI.
pip install showwork
What does showwork verify?
Observability answers, "What did the agent do?"
showwork answers, "Is the outcome it claimed actually true?"
The first question needs traces. The second needs a falsifiable assertion and a deterministic checker. No LLM judging an LLM.
showwork start --session fix-timeout --agent claude-code
showwork claim --session fix-timeout \
--claim "bumped billing-sync API timeout to 30s" \
--type file_contains --path config/api.yaml --pattern "timeout: 30"
showwork finish --session fix-timeout --status ok
If a claim is RED, the clean close is refused. Exit code 2. Fix reality, retract the claim, or finish as blocked.
The command checker has no shell. It only permits a Python script under the project root. It rejects shell metacharacters, path escape, and recursive verification. Other checks cover files, content, path moves, frontmatter, and file counts. Vacuous patterns (regex that match empty strings, always-true counts) are rejected, not blessed.
Why is the ledger append-only?
Agents will be wrong. Operators will also be wrong. Deleting a bad assertion makes the record look cleaner while destroying the most useful evidence.
showwork appends a retraction that names the earlier claim and explains the correction. The original record stays. Failures become training material for the operation.
Does it gate the session?
Yes, through the explicit finish command. Observation and gating stay separate.
showwork finish --session fix-timeout --status ok
The Claude Code Stop-hook adapter records a verdict when a session stops, but always exits 0. It observes. It does not block. The gate is the finish call your prompt contract requires.
What proof came before the package?
I derived a sanitized snapshot from the private ledger that runs the operation:
- 2,158 claims recorded
- 2,152 backed by deterministic checks
- 842 sessions represented
- 152 append-only retractions
- One malformed ledger line surfaced
- Captured verdict: RED, 54 of 60 verified
I kept the RED result. A proof system that only publishes green is not proof.
The derivation publishes aggregates plus a source fingerprint. It does not copy private claim text, paths, strategy, or financial values.
Why publish a specification?
The package can be copied. The record format can outlive it.
spec-v0.1 defines the JSONL schema, checker semantics, anti-vacuous rules, retractions, verdict algebra, session events, and exit-gate contract. Every normative requirement names a behavioral test.
Another team can implement the same evidence format in Go or TypeScript. That is more useful than forcing every agent stack through one Python process.
What is the artifact?
- Installable CLI and Python API:
pip install showwork - Portable ledger specification
- Claude Code Stop-hook adapter
- Genesis receipt where the repository verified its own tests
- Sanitized case study and derivation script
- Marketing demo cut: agents say done, finish is REFUSED, then GREEN after reality matches
Code: github.com/bmdhodl/showwork
PyPI: pypi.org/project/showwork
Where runtime limits fit
Verification is one half of trusting an unattended agent. The other half is bounding what the agent can do before it does it.
showwork proves what an agent did after the fact. It does not stop an agent from looping, over-calling a paid API, or draining a token budget on the way to a wrong answer. Those are runtime questions, and they need a runtime limit.
That is the job of AgentGuard: hard caps on tokens, cost, and call rate around an agent run, enforced while the agent is still running. AgentGuard bounds what an agent can spend. showwork proves what an agent actually produced. Run both and an unattended agent has a ceiling on the way in and a receipt on the way out.
Related reading
Accompanying prompt
What the prompt does: turns an agent's vague "done" report into falsifiable claims with deterministic checks you can run yourself.
Copy/paste this prompt:
Role: senior engineer auditing an AI agent's completion report.
Context: an agent reported finishing a task in my repository. I have the report and repo access.
Task:
1. Extract every material outcome the report asserts (files changed, configs set, tests passing).
2. For each outcome, write one falsifiable claim with a deterministic check: a file path plus regex, a command with expected exit code, or a path that must exist or be absent.
3. Flag any outcome that cannot be reduced to a deterministic check and name the evidence needed instead.
Output: a table with claim, check type, exact check, and what a failure would mean.
Constraints: no LLM-judged checks. Every check runs from the repo root and can fail.
Copy the block above.
Get the artifact-backed local AI lab notes by email: https://bmdpat.com/5090-reports
Get the local AI lab notes (benchmark rows, VRAM fit, quant choices, what runs on consumer GPUs), M-F only when there is something worth sending: https://bmdpat.com/newsletter?utm_source=blog_md&utm_medium=aeo&utm_campaign=my-agents-have-to-prove-what-they-did-2026
Originally published on bmdpat.com. I run a one-person AI agent company and write about what actually works.
Want these in your inbox? Subscribe to the newsletter - no spam, unsubscribe anytime.
Top comments (1)
teza: checker vocabulary spans two different verification strengths (artifact-state: file_contains/path-exists vs behavior-observed: command hitting live system), but ledger verdict is uniformly GREEN/RED with no strength tag. A claim like "timeout bumped to 30s" backed by file_contains proves the config file, not that the running service honors it. Concrete fix: tag checks by evidence class in the spec.
Publishing the RED result is the part that actually earns trust here, most tool launches only show the green path.
One thing worth naming in the spec before other implementations copy it: the checker vocabulary spans two different evidence strengths, and the ledger does not currently distinguish them. file_contains, path exists, frontmatter, file counts, these verify the state of an artifact. The command checker, running an actual script, can verify the state of a running behavior. Your own example, bumped billing-sync API timeout to 30s checked via file_contains on config/api.yaml, proves the config file says 30. It does not prove the running service is actually honoring that value, config drift, a stale deploy, or a service that reads a cached copy would all still pass that check GREEN.
Right now both kinds of claim land in the ledger as the same verdict shape. A reader scanning 2,152 backed claims cannot tell, without opening each one, how many are artifact-state checks versus behavior-observed checks, and those are meaningfully different strengths of proof. Since spec-v0.1 already names every normative requirement against a behavioral test, this seems like the natural place to add one more: an evidence-class tag per checker type, artifact vs behavior, surfaced in the verdict output. Cheap to add now, much harder to retrofit once other stacks have implemented spec-v0.1 as written.