On 27 July I opened a project I'd been building with Claude Code and found three things true at once:
- a card had carried a null commit for two days
- the spec held nine false statements
- five hundred lines had been written against a card still sitting in Backlog, because nobody called
start
None of that was the agent writing bad code. The code was fine. It was the agent's record of the code that had quietly come apart, and I hadn't noticed because there was nothing to notice with.
I'd been blaming the wrong thing for weeks.
The shape every one of these tools has
I tried the obvious fixes. A stricter CLAUDE.md. A tracker with better prompts. Hooks that nagged. They all helped a little and none of them touched the actual problem, which is structural:
The agent does the work, and the agent writes its own report card.
That's it. That's the whole failure mode. Your tracker is a filing cabinet: the agent says "done, tests pass", the board stores the string "done, tests pass", and nothing anywhere asks is that true?
So the only thing standing between a claim and your project's memory is you, reading the diff. You are the verification step. That's why you can't walk away while it works, and it's why the board and the repo drift apart the moment you stop watching.
Advice doesn't fix this. I know because I wrote the advice. The entire MCP server for this project was built without start ever being called — the card sat in Backlog while five hundred lines were written against it. If the author of the protocol drifts inside one session while holding it in context, the protocol isn't a mechanism. It's a wish.
What I built instead
I stopped trying to make the agent more reliable and started asking a different question about every fact on the board: who has the authority to assert this?
Did the command pass? The machine says so. When the agent hands a card back, the tracker runs your project's check first, and only grants the status if it exits zero. Crucially the check is an argv array declared by a human in project config — no tool the agent can reach may write it. The agent can select a declared check; it cannot define one. An agent that could write the command that grades it would just be grading itself with extra steps.
Did the work land? Git says so. If a card's commit is already an ancestor of main, the board is corrected at session start without being asked. Forward only: it fills blanks and confirms landed work, and never overrules a decision you made — because no commit records intent.
Is this note still true? The diff says so. Every note records the sha it was true of. Later the board tells you how far the tree has moved since, and distinguishes "nothing has landed" from "I can't check". Everywhere else, a note from three weeks ago looks exactly like one from this morning.
Do two things contradict? The board says so, unprompted. A view for claims git contradicts, branches no card owns, and cards closed without a check ever running.
Storing your board in git is storage — several tools do that. Letting git overrule your board is arbitration. That's the difference, and it's the whole product.
The part I got wrong, twice
Two things I'd tell anyone building in this space.
Verification is about which surface may establish a command, not about escaping it. I made checks argv arrays, ran them with shell: false, and thought I was done. Then I found that the local web UI's PUT replaced the whole document — including the checks map — and it's unauthenticated by design. So anything that could reach that port could install ["/bin/sh", "-c", "…"] as the check, and the next hand-back would run it. I reproduced it end to end: 200 OK, schema-valid, payload executed. shell: false is no defence when the argv is a shell.
And "measured" means "measured on my machine" until CI exists. A grace window for capturing a check's trailing output was measured carefully — a hundred bytes to a million, ten runs each, plus deliberate CPU saturation. All of it on one 8-core laptop. The first 2-core CI runner it ever met dropped a line on the first job. Three separate timing assumptions in this project have now failed that way.
The honest limits
A pass proves a declared command exited zero on a named tree. It does not prove the work is correct. An agent that writes a passing test for broken code defeats this completely, and the tool says so on the card rather than in a footnote.
Until you declare a check, it proves nothing at all — cards move on the agent's word, like everywhere else. There are no dependency graphs. And it's one developer, one machine: no accounts, no permissions, no team features.
It was built using itself
Every feature was used to build the next one, and the board in the repo is the real one — 75 cards and 274 notes, about 43,000 words, written by the agent as it worked, including the mistakes. A locking bug that silently lost writes. A safety check whose error handling turned a crash into total silence. A test that passed against a file the tracker itself had just modified — caught by the feature that had shipped hours earlier, which promptly caught its own author.
That last one is my favourite thing in the project. The tool's first real catch was the tool catching me.
github.com/albertoclemente/shipward — MIT, zero dependencies, no build step, Node 20+. 547 tests.
If you run coding agents, I'd genuinely like to know whether this happens to you too: an agent closing something that later turned out not to hold. I don't know yet whether I'm unusual.
Top comments (4)
The part I like is making the check an argv array owned by config. That sounds small, but it removes the agent's easiest escape hatch. I would still want the tracker to store the raw command output too, because a green exit code without the transcript is hard to debug later.
Failures keep the output — exit code, duration, head and tail of the log with the
elision stated. Passes don't: just the check name, argv, exit, duration, sha, and
whether the tree was dirty.
The reason is that the note is the memory. Every future session re-reads it, so a
full test log is a cost paid forever, not just by the run that produced it. And a
green run at a clean sha you can re-derive — check it out, run the same argv, the
transcript comes back.
The dirty tree is where you're right. A pass over uncommitted changes isn't
reproducible from the sha, so there the transcript was the only copy. That I
should fix.
The machine-owned check is the right boundary. I’d also bind every successful check result to the exact Git tree it evaluated.
An exit code alone can become stale immediately: the agent can run the check, modify another file, then hand the card back using a result produced against different state. Recording the commit/tree hash, command, exit code and output digest makes the claim reproducible.
For dirty worktrees, either refuse completion or hash the relevant files before and after the check and fail if they changed. The useful invariant is not just “this command passed,” but “this command passed against the exact artifact now being marked complete.”
"This command passed against the exact artifact now being marked complete" — that's the invariant, said better than I've managed it.
Most of it's there: argv, exit code, duration, sha, dirty flag, anchored to the tree
the check ran on rather than HEAD at write time. And the agent never carries a result at all — done() runs the check itself.
But you've found something I hadn't. I read the head before the run and never again, so a tree that changes during the check is recorded against pre-run state. Your before-and-after hash is the fix, and I don't have it.