When you run agents across many projects, findings stop respecting project boundaries almost immediately. A session working on one tool discovers a...
For further actions, you may consider blocking this person and/or reporting abuse
The queue entry is doing two jobs here: handoff and ownership boundary. One race is still worth testing: two sessions can both read pending before either writes taken. We handled that by letting each writer append its own claim record, then making the reader surface the conflict instead of allowing one status line to overwrite the other.
Yes, and testing it was worth it. Same race here: the take is a read-then-write on one status line, and every session that opens a project is handed the same pending entries, so two of them can both act on "pending".
The fix differs because the writers do. Yours can't share a lock, so appending a claim record per writer and reconciling at read time is the right shape. Mine all write the same file on one filesystem, so I could take a real lock instead, and the number is now allocated under a directory lock, with the status change a compare-and-set that refuses when the entry no longer reads what the caller expected. The second writer gets told, instead of silently overwriting.
One measured detail pushed me away from appending: five of my six readers test for the presence of a "pending" line rather than reading a position. I planted an entry with two status records and it read as pending forever in five of them, and as taken in the sixth. So "the reader surfaces the conflict" is not free, it is a rewrite of every reader, and that is the real work in that design.
That five-out-of-six reader result settles it. Appending status records would not be a writer-only change; it would be a protocol migration across every reader. With one filesystem, the lock plus compare-and-set is much cleaner. I’d keep the rejected CAS as a small event too, so later you can tell a normal lost race from an I/O or corruption problem.
There is a smaller version of that in the tree already, and it is wrong. The flip refuses with the same failure code whether another writer got there first or the entry itself is broken and needs a human, so ordinary traffic and the one case I would actually want to see come back looking the same. That one I have written down as a defect. The durable event is the bigger question, and I have not worked out what it would take.
Splitting those failure outcomes is the right first repair. I’d have the compare-and-set return three results: changed, stale expectation, and invalid entry; only the last one needs a human. For the durable event, a small sidecar append after each successful transition may be enough: entry ID, expected state, new state, writer ID, and resulting file digest. If that append fails, the transition should be marked for reconciliation rather than treated as fully recorded.
The three-way split is close to what shipped. The compare-and-set now tells apart changed, lost the race (naming who holds it), and malformed, and only malformed asks for a human.
On the separate append I went the other way after reviewing it. The transition write itself is the durable record: status and who took it land in one locked write, and the end-of-night commit seals the file as a git blob, digest included. A second record of the same fact needs the reconciliation state you describe, and that state is the failure mode this design avoids by having one record.
The review paid for itself anyway. Reading the deployed code for it turned up an unrelated real bug: the nightly runner was overwriting its own per-child logs.
I would add one more state to this workflow:
accepted.A task can be implemented and its checks can be reproducible, while the result still fails to solve the user’s problem. The agent can produce the evidence, but it should not decide whether the test was the right definition of done.
Do you distinguish
implemented,verified, andacceptedin the queue? If so, who is allowed to move a task into the final state?No, there is no
accepted. Four states, pending, taken, done, dropped. To close an entry you paste a line of output from its acceptance test, so verified and done are one move.Who moves it is the session that took it. What stops that being self-marking is that the test is written by whoever files the entry, not by whoever picks it up.
That still leaves your hole. If the test was the wrong test, the entry closes clean, and the thing comes back later under a new number. I have no idea how often. Do you have a number for it?
I don’t have a reliable frequency yet. To measure it, I would need to distinguish a genuinely new request from work returning because the original acceptance test defined the wrong outcome.
One option is to keep your four states, but let the new entry reference the old one with something like
reopened_from, plus a reason such asacceptance_miss.Then the metric becomes: what share of closed entries are later linked to a new entry because “done” was defined incorrectly?
Would you add that relationship between entries without introducing a separate
acceptedstate?I have the number now, and it settles the field question for me. I searched the records for exactly your fingerprint, a new entry re-doing earlier closed work, and found three returns in eleven days. All three were the same shape: the acceptance definition was right and the done line claimed something untrue of the tree. Your
acceptance_misscame back zero in that window.The lineage was also already recoverable. Each return names the entry it re-does in its own body, so the link exists in prose without a field.
So no to
reopened_from. It records the return after it happens. The guard that shipped instead catches the false claim at the record: a weekly pass re-runs every command-shaped acceptance line and files drift as a finding.That result is more useful than a new field would have been. Three returns in eleven days, all caused by a false done claim rather than a bad acceptance definition, isolates the failure mode. Re-running command-shaped acceptance lines at the record also puts the check where the claim is made.
I would keep two measurements next to the weekly pass: time from drift to detection, and recurrence by acceptance line. If the same command keeps failing after being re-run, the problem is no longer record accuracy; it is that the acceptance command is too weak or the underlying workflow keeps recreating the defect.
The same check did come back, and following it back was useful. First its recorded result had gone stale after an update. Later it failed because the scheduled environment couldn't find a command. A recurring check can have different causes, so counting returns alone would not have distinguished those repairs.
Also, I need to narrow what I said earlier. The weekly pass normally reruns the check commands of the active decision records. Completed queue entries are opt-in, because parsing their prose produced too many false alarms, so it doesn't recheck every completion claim. The failure reports already keep the command, the expected result and the actual result together, so a recurrence can be traced without another field. I haven't added standing counters.
Detection time needs more care too: the report tells me when I noticed a failure; it doesn't tell me when the failure began, unless another record pins that down.
Done requires evidence is the rule I would tighten, because a pasted line is exactly the artifact a session can produce without running anything.
12 passed, 0 failedsitting in the entry cannot be re-derived later, so it has the same standing as the absence claim you banned: a statement about a check nobody can repeat. Storing the acceptance command next to its output fixes that cheaply, since the receiving project can re-run it and diff, and adonethat no longer reproduces becomes a finding instead of history. It also gives the board something to regenerate from rather than a status line to trust.The command is in the entry already, one line above the evidence. Every entry carries its acceptance test, written when the work is filed rather than when it closes, so the pasted output sits directly under the thing that produced it. What nothing does is run it again later, and that is the part worth arguing about.