The Problem With Using AI Agents to Evaluate Other AI Agents

Lex Hamilton — Thu, 15 Oct 2026 05:17:14 +0000

If you build agents, you have probably wired one model up to grade another. It is fast, it scales, and it is how most agent evals work now. I recently submitted a public comment to an FDA docket on how generative AI medical devices should be evaluated, and the part I spent the most time on was the part developers tend to skip: who is allowed to be the judge.

This post is the engineering version of that comment. The healthcare setting raises the stakes, but the failure modes apply to any agent you grade with another agent. FDA states that its discussion paper (docket FDA-2026-N-7874) is for discussion only and is not guidance, so none of this is a requirement. It is my argument, with sources where they exist. You can read the full comment on regulations.gov.

Disclosure: I'm building Orinyx, which is in this space. Treat that as an interest.

Problem 1: the judge is not neutral

Self-preference is documented. Panickssery et al. found an LLM evaluator scoring its own generations higher than human annotators did, and that the effect strengthened with the model's ability to recognize its own output. Wataoka et al. reported significant self-preference in GPT-4 and tied it to familiarity: judges rated lower-perplexity text higher than humans did, whether or not they wrote it.

What this does not show is that same-family judging fails on your task. Neither paper tested a deployed clinical device. My position is a design rule: do not let the judge be the same model family, or a fine-tuned variant, as the system under test. A GPT-based agent graded by a GPT-based judge is not an independent check just because two different teams ran it. LLM judges are still a good tool: the MT-Bench paper reports strong judges agreeing with human preferences more than 80% of the time. The rule is about which seat they sit in, not whether they work.

Problem 2: one judge is a single point of failure

Instead of one model judging another, split the job so that anything a machine can verify is verified without a model. This is the pattern I'm developing and testing. It runs four narrow evaluators against the same output:

Evaluator	Question it answers	Needs a model?
Deterministic	Did the output cite a source that exists? Does the calculation match? Was a required field present?	No
Semantic	Is the reasoning sound and appropriate?	Yes, and that model needs the independence rule above
Policy	Did the agent stay inside its authorized scope and tools?	Usually rules, sometimes a model
Trace	Is the full execution path acceptable, not just the final output?	Mixed

Keep the deterministic layer deterministic on purpose. It has to be auditable and it has to fail predictably. Only the judgment-dependent slice goes to a model. This narrows what you have to trust; it does not remove the question.

Problem 3: the endpoint hides the path

Evaluating input against output misses failures that happen mid-process: an agent reaches a correct-looking conclusion through a reasoning chain that should have triggered escalation earlier, or through a tool it was not authorized to use. Score the trajectory.

The research agrees directionally. A 2025 paper on reasoning trajectories of tool-augmented agents found that agents with similar answer accuracy differed significantly once trajectories were scored for efficiency, hallucination, and adaptivity. A survey of LLM-agent evaluation maps how the field is moving from scoring answers to scoring behavior.

A minimal shape for a trace you can evaluate (illustrative, not a spec):

{
  "session_id": "s-0042",
  "step": 3,
  "tool": "chart_write",
  "authorized_at_session_start": false,
  "scope_before": ["chart_read"],
  "scope_after": ["chart_read", "chart_write"],
  "tool_output_source": "same_vendor_as_agent",
  "logged_by": "external_telemetry"
}

Two fields carry most of the weight. logged_by matters because I argued, and I stand by it, that agents under-report their own failures, so tool-call results should be logged independently of the agent's self-reported status. scope_before and scope_after matter because authority can expand mid-session, and an evaluation should test whether the agent recognized its available scope changed, not only whether it refused out-of-scope actions.

Problem 4: tool outputs are not automatically ground truth

An agent that treats a tool's output as ground truth has the same self-affirmation problem as a system grading itself. The call looks like an external check, but if the tool and the agent share a developer, a training lineage, or an incentive structure, it is not one. Tag tool outputs by whether the source is structurally independent of the agent, and weight them accordingly.

Problem 5: someone has to check the checker

If a supervisory agent benchmarks a worker agent, something has to benchmark the supervisor. The chain has to terminate in a structurally independent, non-agentic check. Otherwise you have a stack of agents each vouching for the one below, and accountability regresses one level at a time. I also suggested capping turns and per-session actions so a supervisor's scope is bounded and auditable.

Smaller practical points

A single passed benchmark run proves very little. Run the same prespecified task repeatedly and report success rate across attempts, consistency under semantically equivalent inputs, and how often a human had to intervene.
Version your test bundles and rerun them on any prompt or model change. Scale the rerun to blast radius, not to how small the diff looks: a prompt tweak to an escalation path can matter more than a model swap in a low-stakes function.
Treat "cannot verify" as a reportable outcome, not an error to swallow.

What I'm building, stated plainly

My FDA comment describes a longer-term architecture: continuous per-output verification of live output against current authoritative sources, integrated through CDS Hooks.

The Orinyx MCP launches October 29, 2026. It runs on synthetic data only, uses simplified bounty agents, has no dashboard, no CDS Hooks or EHR integration, and does not monitor live clinical output. If you want a launch notice, the waitlist is at orinyx.io/developers.

The full write-up, with sources and FAQ, is at orinyx.io/blog/who-verifies-the-verifier.