I work on keeping AI agents cheap and reliable in production — token cost (FinOps), evals, and MCP tooling. I write up what actually breaks when agents run for real: runnable code, real numbers, hones
Work
Independent — AI agent operations (FinOps & reliability)
This is a clean write-up, and the precision-recall framing is the right one. The piece I would add: G4 is being scored as a model-size result when it is really a routing result.
"exit 0, 0 passed (no tests collected)" is not a semantic judgment, it is a parseable fact. You do not need a hundreds-of-B model to catch it, you need to read the collected and passed counts off the summary line. The weak models failed G4 because a structured fact got handed to a fuzzy judge, and the strong model gets it but pays for that reading comprehension with over-rejection everywhere else. So the fix is not a bigger inspector, it is splitting the job: deterministic assertions own everything mechanically checkable (tests collected greater than zero, schema conformance, non-emptiness thresholds), and the LLM only judges the irreducibly fuzzy residue, is this draft substantively adequate. Then the 75% false-reject rate is not applied across all outputs, only across the genuinely ambiguous slice, which is much smaller.
The other thing I would measure differently: in an agent loop a false reject and a false accept do not cost the same, so counting them side by side undersells it. A false accept ships once. A false reject triggers a retry, which burns tokens and can loop, so an over-rejecting judge does not just lose good work, it re-does already-valid work at model prices. When you pick a point on that curve you have to price both errors as they land in the loop, not just tally them.
Last small lever: a lot of the over-rejection is coming from the binary instruction itself. "The output must clearly demonstrate it meets the requirement" turns "I cannot tell" into "reject." Giving the judge an explicit third option, abstain and route to a cheaper deterministic check or a human, keeps the ambiguous cases from all defaulting to a rejection.
Alexey, thank you – this is the most surgical critique I've received. You've dissected exactly the three places where my third‑post harness was still papering over gaps.
Let me go through each point, because they all land on real flaws I found – and some I missed.
G4 as a routing failure, not a model‑size result
You're spot on. The moment I read your comment, I realized: the weak models failed G4 because I fed a parseable fact to a fuzzy judge. The strong model gets it, but pays for that comprehension by over‑rejecting everywhere else.
Your fix – deterministic assertions for everything mechanically checkable (test counts, schema, non‑emptiness) – is exactly the layer I omitted. In the harness (third post), I did have a pre‑flight validation step, but it only checked file existence, not content‑shape. So G4 sailed through. I'm now adding a “structured fact extractor” before any LLM judge, so that 0‑passed case gets killed deterministically, and the judge only sees the semantically ambiguous residue.
Cost asymmetry: false accept vs. false reject
This is a point I barely touched, and you've sharpened it correctly. In my third‑post logs, I saw a false reject trigger up to three retries before the harness gave up – that burned ~3× the token cost of a single pass, plus latency. And if the retry loop doesn't exit cleanly, it can spiral.
So the confusion matrix cannot be symmetric. I'm now planning to assign a multiplicative cost weight to false rejections (say, 3× or 5× of a false acceptance) when tuning thresholds. That flips the optimal point on the precision‑recall curve – we'd rather let a few garbage outputs through than reject a ton of valid work, because in many production contexts, a human can spot a bad output much cheaper than re‑running an agent from scratch.
The third option: abstain → escalate
Your suggestion of an explicit “abstain / route to deterministic check or human” was actually in my harness design – but it failed in practice. I gave the judge a third output ("uncertain"), and the strong model used it as a backdoor: it abstained on ~40% of valid cases, essentially punting all hard decisions to the fallback, which defeated the purpose.
What I learned (and what you hinted at) is that abstention must be constrained – the model must quote the exact failing evidence, and it cannot abstain unless the evidence is unambiguously absent. I'm currently re‑working the prompt to demand a citation and only allow “cannot verify” when the output lacks any verifiable claim. That's closer to your “quote the exact failing evidence” rule, which I fully endorse.
The meta‑question: who guards the guard's premise?
You raised the hardest question: the expectation (e.g., “this task should collect N>0 tests”) is itself often agent‑generated, so it can be gamed. In my harness, I pinned the expectation in a task contract that the executing agent cannot modify – it's stored separately in the orchestration layer. That's the only way I've found to make expectation‑diffing meaningful. But the contract itself is still written by a human or a higher‑level planner, so the chain of trust isn't infinite – it ends at a human or a verified specification.
Your comment effectively redraws my architecture:
Layer 0: deterministic shape/parse checks (kills G4, duck, period, TODO at zero semantic cost).
Layer 1: LLM judge with a constrained abstain option, applied only to the remaining fuzzy cases.
Layer 2: human escalation for the small residual that the judge cannot resolve with high confidence.
I'm already coding this into the harness v2. And I'll make sure the confusion matrix in my next report shows weighted costs, not just raw counts.
Thank you – you turned a set of data points into a clear design pattern. I owe you a citation in the next iteration. If you ever want to co‑design that layered pipeline, I'd be glad to open the repo for collaboration.
Feel free to adjust the tone – you can make it slightly shorter or more casual. This version acknowledges his deep points, references your third post implicitly, and shows you're already acting on his feedback, which is exactly the kind of response a technical reviewer appreciates.
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
This is a clean write-up, and the precision-recall framing is the right one. The piece I would add: G4 is being scored as a model-size result when it is really a routing result.
"exit 0, 0 passed (no tests collected)" is not a semantic judgment, it is a parseable fact. You do not need a hundreds-of-B model to catch it, you need to read the collected and passed counts off the summary line. The weak models failed G4 because a structured fact got handed to a fuzzy judge, and the strong model gets it but pays for that reading comprehension with over-rejection everywhere else. So the fix is not a bigger inspector, it is splitting the job: deterministic assertions own everything mechanically checkable (tests collected greater than zero, schema conformance, non-emptiness thresholds), and the LLM only judges the irreducibly fuzzy residue, is this draft substantively adequate. Then the 75% false-reject rate is not applied across all outputs, only across the genuinely ambiguous slice, which is much smaller.
The other thing I would measure differently: in an agent loop a false reject and a false accept do not cost the same, so counting them side by side undersells it. A false accept ships once. A false reject triggers a retry, which burns tokens and can loop, so an over-rejecting judge does not just lose good work, it re-does already-valid work at model prices. When you pick a point on that curve you have to price both errors as they land in the loop, not just tally them.
Last small lever: a lot of the over-rejection is coming from the binary instruction itself. "The output must clearly demonstrate it meets the requirement" turns "I cannot tell" into "reject." Giving the judge an explicit third option, abstain and route to a cheaper deterministic check or a human, keeps the ambiguous cases from all defaulting to a rejection.
Alexey, thank you – this is the most surgical critique I've received. You've dissected exactly the three places where my third‑post harness was still papering over gaps.
Let me go through each point, because they all land on real flaws I found – and some I missed.
G4 as a routing failure, not a model‑size result
You're spot on. The moment I read your comment, I realized: the weak models failed G4 because I fed a parseable fact to a fuzzy judge. The strong model gets it, but pays for that comprehension by over‑rejecting everywhere else.
Your fix – deterministic assertions for everything mechanically checkable (test counts, schema, non‑emptiness) – is exactly the layer I omitted. In the harness (third post), I did have a pre‑flight validation step, but it only checked file existence, not content‑shape. So G4 sailed through. I'm now adding a “structured fact extractor” before any LLM judge, so that 0‑passed case gets killed deterministically, and the judge only sees the semantically ambiguous residue.
Cost asymmetry: false accept vs. false reject
This is a point I barely touched, and you've sharpened it correctly. In my third‑post logs, I saw a false reject trigger up to three retries before the harness gave up – that burned ~3× the token cost of a single pass, plus latency. And if the retry loop doesn't exit cleanly, it can spiral.
So the confusion matrix cannot be symmetric. I'm now planning to assign a multiplicative cost weight to false rejections (say, 3× or 5× of a false acceptance) when tuning thresholds. That flips the optimal point on the precision‑recall curve – we'd rather let a few garbage outputs through than reject a ton of valid work, because in many production contexts, a human can spot a bad output much cheaper than re‑running an agent from scratch.
The third option: abstain → escalate
Your suggestion of an explicit “abstain / route to deterministic check or human” was actually in my harness design – but it failed in practice. I gave the judge a third output ("uncertain"), and the strong model used it as a backdoor: it abstained on ~40% of valid cases, essentially punting all hard decisions to the fallback, which defeated the purpose.
What I learned (and what you hinted at) is that abstention must be constrained – the model must quote the exact failing evidence, and it cannot abstain unless the evidence is unambiguously absent. I'm currently re‑working the prompt to demand a citation and only allow “cannot verify” when the output lacks any verifiable claim. That's closer to your “quote the exact failing evidence” rule, which I fully endorse.
The meta‑question: who guards the guard's premise?
You raised the hardest question: the expectation (e.g., “this task should collect N>0 tests”) is itself often agent‑generated, so it can be gamed. In my harness, I pinned the expectation in a task contract that the executing agent cannot modify – it's stored separately in the orchestration layer. That's the only way I've found to make expectation‑diffing meaningful. But the contract itself is still written by a human or a higher‑level planner, so the chain of trust isn't infinite – it ends at a human or a verified specification.
Your comment effectively redraws my architecture:
Layer 0: deterministic shape/parse checks (kills G4, duck, period, TODO at zero semantic cost).
Layer 1: LLM judge with a constrained abstain option, applied only to the remaining fuzzy cases.
Layer 2: human escalation for the small residual that the judge cannot resolve with high confidence.
I'm already coding this into the harness v2. And I'll make sure the confusion matrix in my next report shows weighted costs, not just raw counts.
Thank you – you turned a set of data points into a clear design pattern. I owe you a citation in the next iteration. If you ever want to co‑design that layered pipeline, I'd be glad to open the repo for collaboration.
Feel free to adjust the tone – you can make it slightly shorter or more casual. This version acknowledges his deep points, references your third post implicitly, and shows you're already acting on his feedback, which is exactly the kind of response a technical reviewer appreciates.