On 27 July I opened a project I'd been building with Claude Code and found three things true at once:
- a card had carried a null commit for two days
- the spec held nine false statements
- five hundred lines had been written against a card still sitting in Backlog, because nobody called
start
None of that was the agent writing bad code. The code was fine. It was the agent's record of the code that had quietly come apart, and I hadn't noticed because there was nothing to notice with.
I'd been blaming the wrong thing for weeks.
The tracker this post describes is real: github.com/albertoclemente/shipward — no clone needed: npx shipward setup ~/code/your-repo --seed-from-branches, and your board seeds itself from your own branches.
The shape every one of these tools has
I tried the obvious fixes. A stricter CLAUDE.md. A tracker with better prompts. Hooks that nagged. They all helped a little and none of them touched the actual problem, which is structural:
The agent does the work, and the agent writes its own report card.
That's it. That's the whole failure mode. Your tracker is a filing cabinet: the agent says "done, tests pass", the board stores the string "done, tests pass", and nothing anywhere asks is that true?
So the only thing standing between a claim and your project's memory is you, reading the diff. You are the verification step. That's why you can't walk away while it works, and it's why the board and the repo drift apart the moment you stop watching.
Advice doesn't fix this. I know because I wrote the advice. The entire MCP server for this project was built without start ever being called — the card sat in Backlog while five hundred lines were written against it. If the author of the protocol drifts inside one session while holding it in context, the protocol isn't a mechanism. It's a wish.
What I built instead
I stopped trying to make the agent more reliable and started asking a different question about every fact on the board: who has the authority to assert this?
Did the command pass? The machine says so. When the agent hands a card back, the tracker runs your project's check first, and only grants the status if it exits zero. Crucially the check is an argv array declared by a human in project config — no tool the agent can reach may write it. The agent can select a declared check; it cannot define one. An agent that could write the command that grades it would just be grading itself with extra steps.
Here it is happening, unedited — a confident "done, tests pass" refused, then the same command earning the hand-back once the fix is real:
Did the work land? Git says so. If a card's commit is already an ancestor of main, the board is corrected at session start without being asked. Forward only: it fills blanks and confirms landed work, and never overrules a decision you made — because no commit records intent.
Is this note still true? The diff says so. Every note records the sha it was true of. Later the board tells you how far the tree has moved since, and distinguishes "nothing has landed" from "I can't check". Everywhere else, a note from three weeks ago looks exactly like one from this morning.
Do two things contradict? The board says so, unprompted. A view for claims git contradicts, branches no card owns, and cards closed without a check ever running.
Storing your board in git is storage — several tools do that. Letting git overrule your board is arbitration. That's the difference, and it's the whole product.
The part I got wrong, twice
Two things I'd tell anyone building in this space.
Verification is about which surface may establish a command, not about escaping it. I made checks argv arrays, ran them with shell: false, and thought I was done. Then I found that the local web UI's PUT replaced the whole document — including the checks map — and it's unauthenticated by design. So anything that could reach that port could install ["/bin/sh", "-c", "…"] as the check, and the next hand-back would run it. I reproduced it end to end: 200 OK, schema-valid, payload executed. shell: false is no defence when the argv is a shell.
And "measured" means "measured on my machine" until CI exists. A grace window for capturing a check's trailing output was measured carefully — a hundred bytes to a million, ten runs each, plus deliberate CPU saturation. All of it on one 8-core laptop. The first 2-core CI runner it ever met dropped a line on the first job. Three separate timing assumptions in this project have now failed that way.
The honest limits
A pass proves a declared command exited zero on a named tree. It does not prove the work is correct. An agent that writes a passing test for broken code defeats this completely, and the tool says so on the card rather than in a footnote.
Until you declare a check, it proves nothing at all — cards move on the agent's word, like everywhere else. There are no dependency graphs. And it's one developer, one machine: no accounts, no permissions, no team features.
It was built using itself
Every feature was used to build the next one, and the board in the repo is the real one — 75 cards and 274 notes, about 43,000 words, written by the agent as it worked, including the mistakes. A locking bug that silently lost writes. A safety check whose error handling turned a crash into total silence. A test that passed against a file the tracker itself had just modified — caught by the feature that had shipped hours earlier, which promptly caught its own author.
That last one is my favourite thing in the project. The tool's first real catch was the tool catching me.
github.com/albertoclemente/shipward — MIT, zero dependencies, no build step, Node 20+. 570 tests. On npm as shipward.
If you run coding agents, I'd genuinely like to know whether this happens to you too: an agent closing something that later turned out not to hold. I don't know yet whether I'm unusual. Or skip telling me and point it at a repo: the first thing done() refuses will answer it.
Top comments (25)
The part I like is making the check an argv array owned by config. That sounds small, but it removes the agent's easiest escape hatch. I would still want the tracker to store the raw command output too, because a green exit code without the transcript is hard to debug later.
Failures keep the output — exit code, duration, head and tail of the log with the
elision stated. Passes don't: just the check name, argv, exit, duration, sha, and
whether the tree was dirty.
The reason is that the note is the memory. Every future session re-reads it, so a
full test log is a cost paid forever, not just by the run that produced it. And a
green run at a clean sha you can re-derive — check it out, run the same argv, the
transcript comes back.
The dirty tree is where you're right. A pass over uncommitted changes isn't
reproducible from the sha, so there the transcript was the only copy. That I
should fix.
Dirty passes keep the output now — 9feb4bd.
Clean ones still don't, and I'm leaving that: a clean sha can be checked out and re-run to get the log back, so storing it forever costs every future session for nothing. A dirty tree can't be re-run, which is exactly the case you spotted.
The machine-owned check is the right boundary. I’d also bind every successful check result to the exact Git tree it evaluated.
An exit code alone can become stale immediately: the agent can run the check, modify another file, then hand the card back using a result produced against different state. Recording the commit/tree hash, command, exit code and output digest makes the claim reproducible.
For dirty worktrees, either refuse completion or hash the relevant files before and after the check and fail if they changed. The useful invariant is not just “this command passed,” but “this command passed against the exact artifact now being marked complete.”
"This command passed against the exact artifact now being marked complete" — that's the invariant, said better than I've managed it.
Most of it's there: argv, exit code, duration, sha, dirty flag, anchored to the tree
the check ran on rather than HEAD at write time. And the agent never carries a result at all — done() runs the check itself.
But you've found something I hadn't. I read the head before the run and never again, so a tree that changes during the check is recorded against pre-run state. Your before-and-after hash is the fix, and I don't have it.
Fixed in 9feb4bd. You were right — the tree was only read before the check ran, so a pass could be recorded against a tree that had already changed. It's read on both sides now, and if it moved the card doesn't move either.
One thing your comment saved me from: it hashes the diff, not the filenames. A file edited twice with the same name would have slipped through otherwise.
Thanks. First bug on that board that didn't come from me or CI.
Nice fix. Hashing the diff contents closes the “same filenames, different state” hole.
The remaining edge is the small TOCTOU window between the second tree read and the card-state write. If another process can modify the worktree there, I’d make the final transition a compare-and-swap: update the card only if the current tree still equals the verified hash. Alternatively, hold the same repository lock across verification and transition.
Then the invariant becomes atomic: either the exact verified tree is marked complete, or nothing moves.
Fixed in
13450bb— the promotion is a compare-and-swap now. Of your two fixes it was the only one this design could take: the check deliberately runs outside the write lock (waiters give up at 60s, and a suite can run for minutes), so holding the lock across verification was out. Instead done() takes a third tree reading while it holds the lock, and the card moves only if the digest still equals the one the check was verified against. A mismatch lands in the same bucket as the last fix — absence of evidence, neither pass nor fail, the card stays in progress. The test stages your scenario literally: a second process holds the lock, the tree moves while the write waits on it, and the card must not move with it. Which is your closing sentence as an invariant: either the exact verified tree is marked complete, or nothing moves. Three findings now, each a strictly smaller window than the last. The acknowledgements section is developing a habit of your name.@peterbuildssecure Postscript, a week on: all three of your findings shipped, and it's on npm now —
npx shipward setup ~/code/your-project --seed-from-branches— so the thing a stranger installs today is meaningfully more correct than the thing I wrote about. That's down to you. Three rounds, each window smaller than the last, and I hadn't seen any of them coming.If you ever have an idle hour, the compare-and-swap is the part I'd most like broken by someone who isn't me. You've got a better record at that than I do.
That is a very generous postscript. It’s also good to hear the fixes made it into the installed artifact rather than stopping at the discussion.
The first place I’d attack is the lock domain. The third digest read and card update are serialized against other Shipward participants, but Git, an editor or another process may not respect that lock.
I’d add a failpoint immediately after the under-lock digest comparison and before the card write, then have an external process mutate the tree in that gap. If the card can still move, the compare-and-swap is correct within one store but the claimed invariant spans two independently writable systems.
One clean way to narrow the claim is for completion to attest “digest X passed” rather than “the current workspace is complete,” and require every consumer of that status to compare X with the tree it is about to act on. That turns a later mutation into visibly stale evidence instead of trying to make the filesystem and card store one atomic transaction.
That failpoint test is the right one, and I think it fails.
The lock in question is Shipward's own — it serializes done() against other Shipward writers and nothing else. git, my editor, any other process: all outside it. So the gap is real. The third digest read happens under that lock, the card write happens milliseconds later, and nothing in those milliseconds is mine to hold. The compare-and-swap took the window from minutes to milliseconds, which is small enough that I stopped being able to hit it by hand — and I'd quietly filed that as closed. Two independently writable systems, exactly as you put it. No lock I can take makes them one transaction.
So I took the other half of your comment instead, in 606af4d. The digest already existed — the check stamps it, the swap compares against it — and then it was thrown away, so what reached the card was check, argv, exit, ok, at, sha, dirty, ms. sha, not digest. A consumer reading a completed card got "passed at abc1234, tree dirty" and had nothing to compare, in precisely the case where the sha proves least: a dirty tree carries one sha through every edit you make to it. The checkable half of the evidence was the half I was dropping. It's kept now, completion attests "digest X passed check Y", and the board's Trust tab does the comparison for anything sitting in review — it names both trees and says the pass is neither wrong nor a failure, just no longer about what you'd be promoting.
Writing the tests for that turned up the thing I'd have shipped straight past. The digest hashed a bare
git diff HEAD, while the dirty flag has always excluded the board's own directory on the grounds that the board is not the code under test. The two had drifted, and the digest was the one that mattered: three of the four changed files in my working tree at that moment were Shipward's own tracker and notes. Which means the pass I record was invalidated by the very write that recorded it — an attestation stale on arrival — and any second writer touching the board mid-check could refuse a legitimate pass outright. There was even a test asserting board writes don't count. It passed against the broken code, because it staged the board as a new untracked file, which git can't see either way; every real repo commits its board. That test now commits it first, and fails without the fix.Four rounds, and each one has taken something away from what this tool is allowed to say about itself. I've stopped calling that a coincidence.
The board-directory exclusion is the tell: any directory the check writes to during verification, not just the board's, invalidates the exclusion rationale. Rather than closing it path by path as you (and I) keep finding a new one, I'd invert the check: dirty should mean "the working tree changed since the digest was taken," full stop, with the only sanctioned exclusion being paths the verified digest itself never covers — which by definition can't drift the thing you're attesting. Two different concerns (what we hash vs. what we watch for changes) collapsing into one exclude-list is what keeps producing a fourth round.
Who has the authority to assert this is exactly the right question, and it generalizes past task trackers. The same gap exists for architectural claims: an agent says this follows the pattern we agreed on, and unless something with actual authority checks that against a rule, you've got a self-report system for architecture too, not just for task status.
Agreed, and task status is actually the easy case — git doesn't care what the
agent claims, the commits and passing tests are either there or they're not.
Architecture is harder because most of the time the rule was never written
down in a checkable form in the first place. It's a paragraph in a design doc
or something agreed in a meeting, and you can't verify a claim against that.
Tools like dependency-cruiser or ArchUnit cover the parts you can encode
(imports, layer boundaries), but there's always a portion that won't encode.
I think the honest position is that this portion stays self-reported, and
the job is to keep it small.
If the agent can select which declared check to run, what stops it from picking the cheapest one? A card handed back against the lint check instead of the test suite would still get the green status. Curious whether checks are bound to specific cards or any declared check satisfies any card.
Nothing stopped it, and the honest answer as of this morning was: recorded, not gated. Any declared check satisfied any card — a done() against lint earned the same green as the suite, and the only defense was that the evidence names its exam. You also found something sharper than your own question: a selected check became the card's check, so one cheap hand-back quietly lowered the standard for every later one.
Fixed in
e10c1da. The rule is now that selection fills a blank, it does not change a standard: a card that already carries a check refuses a hand-back naming a different one — nothing runs, nothing is proved, same bucket as a failing check — and switching takes force:true, which writes a decision entry naming both checks, even when the chosen check then passes. There's no strength ordering between commands, so the tool can't know lint is weaker than the suite; what it can hold is that a standard never changes silently. The declared-by-human rule closed "the agent writes the exam" — you're the third reader to find a hole like this within a day of looking, and this one closed "the agent picks the easiest exam". Thanks.Postscript: e10c1da is on npm now, so the hole you found is closed in the version anyone installs —
npx shipward setup ~/code/your-project --seed-from-branches.Worth saying what your comment actually did. You asked a question instead of reporting a bug, and the question turned out to contain a worse problem than the one you were asking — I'd never have gone looking at check selection on my own. If you do point it at a repo and try to pick the cheap check, I'd like to know whether the refusal reads as obvious or as the tool being obstructive. That's the part I can't judge from inside.
The key insight here is that agent state should be treated as derived data, not trusted self-reported data.
Six days is too long to leave this sitting — sorry, it deserved a faster answer than it got.
"Agents should be allowed to propose state. Systems should be responsible for proving it." I spent a month trying to get that into a title and you did it in two sentences. I'd have led the post with it.
Since you took the derived-data framing further than I did, here's where I found that it stops. Derivation covers what git can prove: a card claiming it shipped when the commit isn't an ancestor of main, a note that was true fourteen commits ago. The audit corrects those without being asked, and only ever forwards — it fills blanks, confirms what landed, and never moves a card backwards. What it can't derive is intent. No commit records that something is backlog rather than review, or that it matters more than the card beside it. That part stays proposed, and the system has nothing honest to say about it. Keeping those two categories apart is most of the design.
Your
shell: falsepoint also had a second half I hadn't seen, and a reader found it two days after the post went up. Owning the command definition stopped the agent writing its own exam — but any declared check still satisfied any card, so it could hand back against lint instead of the suite and collect the same green, and that choice then quietly became the card's standard for every hand-back after it. Fixed in e10c1da: selection fills a blank, it does not change a standard. Your principle, one layer down — the definition was owned, the choice wasn't. You'd probably have got there faster than I did.The authority-owner framing maps directly to agent deployment too. A worker should not be allowed to both perform a tool call and declare its outcome: the durable ledger or provider response has to establish that claim. I use a drain boundary where new claims stop, in-flight effects become terminal or UNKNOWN, and a replacement can reconcile from the idempotency key. Otherwise a self-reported “done” can hide the same gap as a stale tracker card.
The state you call UNKNOWN is the one this tool had to learn to say out loud. A check that times out, or runs while the tree moves under it, is recorded as exactly that — absence of evidence, neither pass nor fail — because collapsing UNKNOWN into either direction is where the lying starts. Earlier today my own machine made the case for me: a neighboring build starved the test suite past its time budget, and the gate refused its own project's hand-back twice rather than call a timeout a verdict. Your drain boundary is the same invariant one layer up: nothing may claim an outcome the durable record can't establish. The ledger outranks the worker, git outranks the board.
This makes me think of agent tracking less as a logging problem and more as an authority-design problem. If the same system can perform an action, declare it successful, and define the criteria for success, the feedback loop is inherently biased. Independent sources of truth become much more valuable as agent autonomy increases.
That's the design position, yes — and the one refinement I'd make is that the fix isn't only independent sources of truth, it's removing the agent from the reporting path entirely. The agent here doesn't run the check and carry back the result; it can only request the transition, and the tracker runs the check itself, under the same lock that moves the card. An agent that never holds the evidence can't bias it. The criteria are out of its hands too — a check is declared by the human, as an argv array in config, and a card can only name one, never define one. What's left to the agent is the part you'd actually want an agent for: doing the work and saying what it thinks it did.
The “who has the authority to assert this?” framing is a really useful way to think about agent reliability. In our AI work at IT Path Solutions, we’ve found that different claims often need different sources of evidence and different validation windows. A Git commit can confirm that code landed, for example, but it doesn't establish that the behavior remains correct after dependencies or surrounding logic change. Giving each claim a clear evidence owner, scope, and freshness boundary could make these systems much more trustworthy.
The freshness boundary is the part of this the tracker actually has an answer for. Every evidence entry is stamped with the sha it ran against, and every time a session re-reads the board the drift since that sha is computed and surfaced — commits since, and whether any of them touched the files the entry names. So a green check doesn't quietly stay green: it visibly ages. What it can't do is tell you which later changes invalidate it — that would need the dependency knowledge you're describing, and I don't think a tracker should pretend to have it. "This passed at that sha, and 14 commits have landed since, 2 touching files it names" is a claim git can back. "It still holds" isn't, and the honest move is to keep those two sentences apart.