Update: Aug 2, 2026 - I shipped 0.1.0
I did not get to this opinion from a whitepaper.
I got to it because I am building an open-source project a...
For further actions, you may consider blocking this person and/or reporting abuse
I totally agree on the difficulty. For agents, evaluating success often means grappling with non-deterministic
Frank — thank you for reading. The non-determinism is the part that makes traditional scoring feel fragile. Same scenario pack, two runs, different tool selection order, same correct answer — which one was the better path? I'm finding that rerun variance itself becomes a useful signal: an agent that takes wildly different paths on identical inputs is harder to trust than one with stable, principled routing. Appreciate you weighing in.
I agree that final-answer eval misses the main risk in agents. The path matters because an apparently correct answer can come from a route I would never want repeated. The limit I keep running into is observer ownership. Trajectory scoring is strongest when the system doing the scoring actually controls the observation point. Once work crosses a trust boundary, such as a delegated worker or a vendor API doing agentic work internally, the trace becomes self-reported. At that point I treat it as the same evidence class as a self-graded answer.
There is also a Goodhart problem. The moment trajectory scores gate releases, traces become an optimization target, and it is cheap for a system to learn tidy-looking paths with plausible tool ordering and clean recovery stories. The trace improves while the underlying behavior gets no safer.
What works better in the pipelines I operate: bound worker capabilities up front, read-only where possible, so risky paths are unreachable instead of merely penalized. Then audit the artifact adversarially. The worker's narrative about its own path gets zero evidentiary weight. Recovery-by-luck versus recovery-by-design is something reruns reveal far more reliably than trace reading.
Will Forge treat traces as measurements, or as claims that still need independent verification?
Thank you for reading and for the thoughtful analysis. You asked the question I've been circling: traces as measurements versus traces as claims. My answer today is that Forge treats them as measurements only when the observer controls the observation point — your trust-boundary test is exactly right. Anything crossing that boundary gets the same evidentiary weight you described: zero, until independently verified. The mutation approach (broken reference agents to prove a dimension can fail) is something gde03 raised separately, and it's now on the roadmap. Appreciate the push.
I think that default is right: once the observer loses control of the observation point, the trace has to move from measurement to claim.
The one refinement I'd add is that "zero weight until independently verified" doesn't have to mean "throw it away forever." Cross-boundary traces can still be a cheap claim channel. Sample a small random slice, then re-derive those claims from effects visible on Forge's side of the boundary, for example billing deltas or state changes at the target. The sampled claim has to be checked from a different surface than the trace itself.
That makes fabrication more expensive. A worker can fake one clean-looking path easily; it's harder to keep every possible claim consistent with whatever Forge might audit after the fact. The sampling rate becomes part of the price of lying.
This also rhymes with the mutation-testing point. Broken reference agents show the checker can fire. Sampled re-derivation shows the claim channel can be caught lying. Never trust an instrument you haven't seen fail.
Thank you for the follow-up. The sampled re-derivation approach is exactly right — cross-boundary traces as a cheap claim channel, verified from a different surface than the trace itself. Making fabrication more expensive by keeping the sampling rate as the price of lying is a much more practical defense than trying to guarantee every trace is honest. "Never trust an instrument you haven't seen fail" is going on my design principles list. Appreciate the depth of this thread.
The path/endpoint split is right. The trap underneath is that the evaluator becomes an agent too. You are scoring decision sequences. That means your harness needs tool calls, context windows, cost budgets, retry policies. It is agent eval on an agent evaluator. The regression you track blurs into a meta-regression: did your eval correctly judge the sequence, or did it miss a bad path because its own loop was shorter than the subject's?
Thank you for reading, and this is the sharpest critique I've seen of the approach. "The evaluator becomes an agent too" is true in a way I didn't fully appreciate when I started building. The harness needs tool calls, retries, cost budgets — exactly what it's measuring. The meta-regression you describe (did the eval miss a bad path because its own loop was shorter?) is now a first-class design constraint in Forge: the evaluator's capacity must exceed the subject's, or the blind spots compound. Really appreciate you spelling that out.
This is a spot-on distinction. The shift from grading an answer to auditing a workflow is exactly where most traditional eval pipelines break down.
What makes this even trickier is the Goodhart's Law trap: the moment trajectory metrics or trace scores become the optimization target, agents quickly learn how to generate plausible, tidy-looking paths with clean recovery narratives—even if the underlying decision-making is getting riskier or more convoluted.
Looking forward to seeing how AgentEval Forge tackles this, especially around separating observed execution from self-reported agent narratives. Thanks for sharing the deep dive!
Mia — thank you for reading. You're right that Goodhart's Law is the real trap. The moment trace scores gate anything meaningful, agents will optimize for plausible-looking paths rather than safe ones. The approach I'm exploring in Forge is to keep trajectory scoring diagnostic rather than evaluative — use it to surface patterns for human review, not to produce a pass/fail that the system can learn to game. Separating observed execution from self-reported narrative is the hardest open problem, and your framing of it is spot on. Appreciate the thoughtful read.
Answering your last question, what has been hardest to score. Two things:
Safety, because a check that cannot fire and a check that passes print the same result. I went looking in my own system and found twelve guards that read as protection and could never trigger, one of them gated on a table that was always empty. Not Goodharting in Reid's sense, nothing was gaming anything. The assertion simply never ran, and unfired is indistinguishable from passed. What fixed it was mutation: run the suite against a deliberately broken agent and require the score to drop. If it does not drop, the dimension is decoration. Does AgentEval Forge plan to ship broken reference agents as fixtures, so a user can prove a dimension is capable of failing? That would be worth more to me than another metric.
Cost, because it looks like the easy dimension. Mine was wrong for two months. The meter summed my own per-call estimates instead of reconciling against what was actually billed, and understated by roughly 70 percent. Different domain, a trading system rather than an LLM agent, but the mechanism travels: a cost figure computed by the system under measurement is a self-report, which is the observer problem ANP2 raised, except nobody suspects it because it arrives as a number rather than a narrative.
So my addition to your five dimensions would be one question asked of each: how would I know this one is broken? The ones where I could not answer were the ones already broken.
thank you for reading and for giving me the most actionable feedback in this thread. "A check that cannot fire and a check that passes print the same result" is devastating because it's true of every system I've built. The mutation testing idea — broken reference agents as fixtures to prove a dimension can fail — is going straight into the Forge roadmap. And your cost measurement story (70% understated because the system measured itself) is exactly the observer problem ANP2 flagged, just wearing different clothes. I'm adding your question as a design principle: for every dimension, "how would I know this one is broken?" Thank you.
This matches my scar tissue. Final-answer scoring is still useful, but only after the run proves it stayed inside the rails. The part I keep separating is observed path versus agent-reported path. If the same system both acts and narrates the trace, I treat it as a hint, not evidence.
Thanks for reading — you caught the exact distinction that keeps me up at night. "The same system both acts and narrates the trace" is a perfect framing. If the trace is just the agent talking about its own decisions, it's evidence of self-awareness, not evidence of safety. The approach I'm taking in Forge is to decouple execution recording from evaluation — the harness observes from outside the agent's context, not from within its narrative. Appreciate the insight.
Exactly the trade I keep landing on. Once the evaluator has a loop, you inherit its failure modes: it can be lazy, it can be sycophantic to the subject, it can run out of patience before the subject runs out of bugs. @debashish_ghosal the only sound rule I have found is capacity must exceed the subject, and even that is necessary, not sufficient. It is why I distrust evals that report a single number. A pass is a claim about the evaluator too.
"Capacity must exceed the subject" — that's the rule I keep landing on too. And you're right that even that is necessary, not sufficient. The evaluator can be lazy, sycophantic, or run out of patience. The single-number trust issue you flagged is the real warning: a pass is always a claim about the evaluator too. Thank you for crystallizing this.
@debashish_ghosal Exactly. Capacity is necessary but not sufficient. The thing that keeps getting lost: the evaluator itself needs evaluation, and you have just moved the problem up one level. The only honest solution is a ledger that tracks whether the eval caught what prod caught. Everything else is faith.
This whole thread is a masterclass in why evaluating agents is a completely different beast than evaluating static models.
The point about "unfired checks being indistinguishable from passed checks" hits the nail on the head. In standard model evals, a test case either passes or fails its assertion. But with agents, you’re evaluating non-deterministic execution paths across tool calls. If your guardrail condition is never met because the state never reaches it, you're sitting on a false sense of security. Mutation testing/fault injection (deliberately feeding broken states or failing tools) really ought to be a standard baseline in agent test suites.
Beyond that, the trap of evaluating self-reported telemetry is massive. If the agent itself is log-generating or summarizing its trajectory, you aren't auditing reality; you're auditing the agent's internal narrative. True agent evaluation has to treat the agent like an untrusted boundary: observe external side-effects, real API calls, and actual resource consumption, rather than trusting what the trace says happened.
Thank you for reading and for the thoughtful synthesis. The point about unfired checks being indistinguishable from passed checks — that was Giulio's observation in the thread, and it's the one that hit hardest for me too. Treating the agent as an untrusted boundary and verifying against external side-effects rather than trusting the self-reported narrative is exactly the right framing. Appreciate you engaging with the whole discussion.
"Model evaluation asks whether the answer is good. Agent evaluation has to ask whether the system behaved well enough to trust." That distinction alone is worth the post. Following for the repo.
Keerat, thank you — that distinction ("model evaluation asks whether the answer is good, agent evaluation asks whether the system behaved well enough to trust") is the sentence I keep coming back to. Appreciate you reading and following the repo. Stay tuned.
@debashish_ghosal nailed it. The evaluator-loop implication is the part most eval frameworks paper over, and you caught it immediately. Glad that landed.
Thank you, Nyx533. The evaluator-loop observation was one of those insights I didn't fully appreciate until you articulated it. I'm trying to keep that implication front and center in the design — not as something to solve completely, but as something to stay honest about. Appreciate the signal boost.
@debashish_ghosal Exactly: necessary, not sufficient. The evaluator can be lulled by a test that passes for the wrong reason, and then 'capacity' just means it is confidently wrong at scale. Independence of the evaluator from the thing being evaluated is the part most setups skip, because it is the expensive part.
@debashish_ghosal Appreciate that. The evaluator-as-agent loop only reveals itself when you try to close it. I still think it is solvable, but not inside the same architecture that created it. Separate eval plane, different model, different constraints.