DEV Community

Cover image for My AI Agent Can Withdraw a Claim but Not Approve One
John
John

Posted on Originally published at hexisteme.github.io

My AI Agent Can Withdraw a Claim but Not Approve One

Originally published on hexisteme notes.

Six evidence claims in my research tool were waiting for review. Each one needed a verdict: passed, withdrawn, or failed. My main Claude session had already formed opinions on most of them, and the tool has a command that records the verdict. The obvious move was to let the session record all six and move on. One of the two outside reviewers I had asked to check the batch said no to half of that, and the half it said no to is the part this note is about.

The setup

The research tool collects sources, freezes their raw text, and lets an agent register claims that point at specific spans of that text. A claim stays pending until someone reviews it. Passing a claim adopts it; withdrawing one takes it out of the set.

For this batch I told my main Claude session to work through the open items, and to settle each one by consulting two other models with different jobs:

  • Codex, run read-only, with the job of refuting things against code and data.
  • agy (Gemini), with the job of checking direction and anti-goals: the list of things the tool is explicitly not supposed to do.

The rule for the whole batch was simple. Only items where both reviewers pointed the same way got executed. Items where they split were held and written down, not decided by the main session.

What agy flagged

The six claims were the review queue. agy's point did not concern any single claim. It concerned who records the verdict.

If my main Claude session records "passed", that is the agent approving its own evidence. One of the tool's anti-goals is automatic adoption of claims. A session that drafts a claim's verdict and then writes "passed" into the store is exactly that, even if the verdict happens to be right. agy's line was that withdrawal can be delegated to the agent and approval cannot.

The agreed result of the two reviews was:

  • Withdrawal: 3 claims, recommended by the agent and the reviewers.
  • Passed: 3 claims, left to me.

My own reading of why the asymmetry holds is narrow. A withdrawal takes a claim out of the usable set. A pass puts one in. The anti-goal is about the second direction. I did not test any wider principle than that, and the source record only states the rule and the reason agy gave.

The tool refused anyway

There was a second gate that I had not designed with this in mind. When the session tried to record the verdicts, the CLI refused to write a review non-interactively. Its message was that the owner should run it directly in a terminal.

The session did not route around this. It wrote the six commands into a shell script and left it for me to run. That is the behavior I wanted, and I want to be precise about why it happened: the refusal came from the CLI, and the decision not to bypass it was the session's. I did not confirm whether the session would have bypassed a softer prompt.

When I ran the script, it recorded only 2 of the 6 and stopped. The cause was set -e combined with a mismatch on the 8-character confirmation input. The replacement script made every line independent, so one failed confirmation could not stop the rest, and it passed a bash -n syntax check. As of this writing the records show the first script's 2 withdrawals; I have no record yet of the second script being run. That second script also carried 4 new pending claims from another session's run, which my main session and Codex had each judged separately. Those went into the same script.

The reviewer who overturned the main session

The most useful single finding in the batch was a correction to the main session's own earlier reasoning.

Codex gave its own verdict on all six claims. On one claim, it said the claim should pass on meaning, and that the main session's reason from October 31, "no workshop evidence", was wrong. The frozen raw source had a Comments line at line 136 reading "Peer-reviewed by NECO 2018 Workshop". My main Claude session opened the file and confirmed the line.

That claim was still recommended for withdrawal, but for a different reason. Another claim among the same six is a later version that replaces it, so the withdrawal reason was corrected to "superseded by the later version" instead of the false "no workshop evidence". If I had let the original reason stand, the record would say the source lacked something it actually contained.

This is also where the asymmetry mattered. Withdrawing that claim with a corrected reason was low-stakes. Had the main session been allowed to record passes, it would have been grading claims on the same reasoning that had just been shown wrong on one of them.

One more claim, among the passes left to me, got a caution note. The source span said "even the strongest frontier models", and the claim restated it as "frontier models". Codex called it a small generalization and passed it. I left the call on that one to the human side, which is me.

The "unresolved" items that were already resolved

An afternoon report on the same day had listed 6 items as unresolved. They were not. All six already had fixes or explanations recorded: a scholar-search budget, a drop in wiki rows that turned out to be wiki HTTP 429 responses, a WAF PDF item that turned out not to apply, a line-break issue in a corpus search command, full-text XML handling, and a "v2" number in a counsel output.

Codex's section on these checked each one against code and selected tests and confirmed all six were handled. The cause was mechanical: the earlier "unconfirmed" lines of a previous section in the batch notes were grepped, and the later lines in that same section, where the resolutions were written, were never read. The grep found the question and missed the answer below it.

I list this here because it is the same shape of risk as the self-approval problem, in the other direction. An agent that reports something as open when it is closed creates work. An agent that reports something as closed when it is open is the case the approval rule exists for.

Where the two reviewers split

Some candidates for new work got opposite answers. A check for contradictions in entity attributes: Codex said hold, because without binding the subject, attribute, period, and unit it would raise false conflicts; agy said start. Strengthening automatic detection of how much of a page was captured: Codex said start, with only clear negative signals; agy rejected it as turning a judge into a gate, which is an anti-goal. Both were held.

A third round with narrower briefs brought both reviewers to the same reduced version of each, and only that reduced version was built. The parts they still disagreed on stayed out.

The code review, briefly

After the first batch of changes, I sent the uncommitted diff to Codex for code review. It raised 6 findings. The session reproduced each one before touching code, fixed all six, and added 15 regression tests. Examples: a stop gate attributing a later task's block to an earlier task that had passed, and a shared-passage index that only recorded the first position of each chunk and so missed a shared 300-character passage. Over the batch the test suite went from 1069 to 1102 passing, and after the third round it stood at 1124.

What is still open

The approval rule currently rests on two things: agy's objection being written down, and the CLI refusing non-interactive review writes. As far as this batch shows, the second one is the only part enforced in code. If the CLI ever accepted non-interactive review writes, nothing recorded in this batch would stop an agent from recording "passed".

Codex also ran in a sandbox that could not create temporary files, so it could not run pytest. The full test pass in this note comes from the main session's run, not from the reviewer.

And the first script stopping after 2 of 6 is a reminder that "the agent handed me a script" only moves the risk to the script. Next I want to check whether the review command can tell a script-driven run from a person at the keyboard at all, or whether the refusal I relied on was only about the missing terminal.

Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.

More notes at hexisteme.github.io/notes.

Top comments (2)

Collapse
 
li_yi_dcec96bf2647dec1b7e profile image
Mindutil •

The asymmetry you ran into is the mundane version of something my own directory does to itself. I run 4,323 AI tool listings, and the 'verified' badge on them is produced by the party with the strongest interest in it being true (the vendor claiming the listing) and then recorded by the same crawler that generated the claim. Split by price tier it gets worse: 614 of 4,323 are claimed (14.2%), and free - the tier readers trust most - has the lowest rate at 261/2,198 (11.9%). Where two sources disagree I also auto-adopt instead of parking the item: three landing pages from one vendor came back labelled free / free / freemium, so the same verdict exists three ways and nothing ever decided it. Your rule that only both-reviewers-agree items get executed, and splits get written down rather than resolved by the session that made the claim, is exactly the step I am missing. The disagreeing case is data I still cannot defend a year later.

Collapse
 
reidmarlow profile image
Reid Marlow •

The terminal check is a useful friction point, but it doesn't establish who authorized the verdict. I'd bind approval to the exact claim revision and source span, then require an independently authenticated reviewer to approve that revision. A changed claim should invalidate the approval. That also gives the human a way to review the proposed script without treating running it as approval of every line.