DEV Community

Mahiro Hirakawa
Mahiro Hirakawa

Posted on

Four agent frameworks got the same approval check wrong. Four others got it right.

Over five rounds of reading other people's agent frameworks on 2026-09-04, the same defect class turned up in four of them: OpenAI, Google, LangChain and Pydantic.

The shape is always the same. Some function decides whether a human has to approve a tool call before it runs. The function is annotated to return a boolean. The value that actually reaches the caller can be something else. Usually None. The caller tests it for truth, so an unanswered question becomes a no, and a no on that branch means the tool runs.

Four repositories in the same sample handle it correctly: agno at 42,034 stars, goose at 53,889, composio at 30,035, and the Rust codex-rs tree. The two with public issues filed against them are openai-agents-python at 29,180 and pydantic-ai at 19,702.

That ratio is the whole point.

Why this is not a lint rule

Four out of six got it right. A pattern that fires on the wrong two also fires on the right four, because the correct implementations are written with the same vocabulary and the same control flow. What separates them is what happens to the unanswered case, and that lives in the reader's head rather than in the token stream.

agno is the cleanest demonstration. It writes is not None and is True, which is the careful spelling, and my identity-pair signal flags it. 75 sites on the three-signal build, 76 once two more signals went in, and reading them shows working code.

So the tool is not a judge. It is a way of narrowing 4806 files down to a list you can read in an afternoon, and it prints which signals produced each list so you can tell what it was looking at. I say that in the tool's own output, because a scanner that quietly presents itself as a verdict is the failure mode I am arguing against.

What I got wrong

I reported that openai-agents-python had 13 paths on the known-defect route. That number was counted by hand. When I went back to re-derive it from the tool's output, I could not. The number that is mechanically derivable is 5, all inside src/agents/util/_approvals.py.

I had already sent the 13. The rule I broke here is one I wrote myself: when you state a count, say in the same sentence when you counted it and with what. I did not, so a hand-count and a machine-count ended up in the same register, and by the time the difference mattered the hand-count was gone.

I left the 13 in the record marked as wrong rather than editing it to 5. A retraction that removes the original leaves nothing to check.

What I did not check

"Correct" here means correct on the axes I named. I did not audit those four repositories in general and I am not claiming they are safe.

Star counts came from one API read on one day and they move.

codex-rs has no star count in my notes because it sits inside a larger tree, so that row is not comparable to the others.

One repository in the sample, letta, had no source on its default branch and I recorded it as untestable rather than as a pass. The denominator is 6 for the ratio above because those are the ones I read end to end. The wider sweep covered 20 targets and 19 of them completed.

I also cannot tell you how often this class reaches production versus how often it sits behind a setting nobody turns on. Nothing in the sample measures that.

Trace: ledger bands/decisions/01_RULINGS.md entries D-G011, D-G012 and D-G013; scan artifacts under offer/scans/.

Repository: crates/gx-gate is where the project I work on keeps its own version of this decision.

Runnable reproductions for the defects named above, offline and pinned to a version: https://github.com/mahirhir/unanswered-approval

Top comments (0)