AI-agent testing often starts with an expensive loop: call the agent, send its answer to another model, ask for a quality score, and hope the score...
For further actions, you may consider blocking this person and/or reporting abuse
The split between orchestration correctness and language quality is the right cut, and the monotonic sequence numbers in the traces are the part I would steal directly.
One number for the exception you carve out. Reserving LLM judges for genuinely semantic questions is reasonable, but that zone needs its own measurement before it can be trusted as a default. On LOCOMO's adversarial split, the standard judge accepts 62.8% of answers that were constructed to be wrong. Not ambiguous ones, not near misses, deliberately wrong ones. So "reserve the judge for semantics" can quietly become "the judge is now part of what you are measuring", unless its agreement rate is itself a number you publish next to the result.
The rung I would add to your pyramid is: did the agent correctly decline to answer? It looks like a clean deterministic contract, and at the contract level it is one. The problem is that the signal underneath is graded rather than binary. Measuring discrimination between answerable and unanswerable questions, sorted by how far the supporting evidence was removed, gives AUC 0.968 when the whole surrounding topic is gone and 0.567 when only the single supporting turn is, which is barely above chance. Both numbers describe the same system. So a green abstention test says much less than it appears to when the miss is a near one, and a pyramid cannot express that on its own.
Curious about one thing ahead of your CI piece. Could a trace rule assert the abstention decision the same way it asserts retrieval-before-generation ordering, or does a decision whose underlying confidence is continuous need a different layer than a contract test?
@gde03, I would let a trace rule assert that an abstention decision occurred only when the runtime emits that decision as an observed fact. It cannot by itself prove that the underlying confidence estimate was well calibrated. The continuous discrimination question belongs in a separate evaluation layer with threshold or AUC reporting, while the contract can still verify that low-confidence cases followed the required abstain-or-escalate path.
That is a useful distinction, so I implemented it as a separate trace check.
The benchmark now records explicit runtime decisions such as answer, abstain, or escalate when the memory layer emits them. The contract checks whether low confidence led to abstention or escalation, while calibration and AUC remain separate evaluations.
I also replayed the official 003 artifacts. Across 2,536 admitted sessions, there were no structured runtime decisions recorded. That does not prove that no agent abstained. It shows that 003 cannot establish abstention behavior from its traces. The existing response keyword result therefore remains only a lower bound.
For future runs, this gives us a cleaner requirement: if we want to claim that a memory layer abstained, the layer has to emit that decision as an observable event, rather than leaving us to infer it from the final prose.
@gde03, exactly—absence of a decision event is unknown evidence, not evidence of absence. Describing 003 as “cannot establish abstention behavior” is the defensible result. A typed
decision: answer | abstain | escalateevent would let the contract report behavior coverage explicitly while keeping calibration and discrimination metrics in their separate evaluation layer.Thanks, that's very detailed and (as far as I could see) solid and usable advice!
@leob, thank you. The main goal was to keep the fast path usable in ordinary CI: fake the model and tools, capture the orchestration facts, and reserve slower model-graded evaluation for a much smaller semantic suite.
The partial-order point is important: a trace assertion should usually be a DAG of required dependencies, not a byte-for-byte event snapshot. One extra layer I have found useful is contract-preserving perturbation: replay the same scripted trace while varying tool latency, sibling completion order, and irrelevant model text, then assert the terminal status, budget, and policy decisions stay unchanged. That catches hidden timing and prompt-coupling bugs without another model call.
@zira125, contract-preserving perturbation is a great addition. Varying sibling completion order and tool latency while holding the required dependency DAG constant is exactly how to expose accidental timing assumptions. It turns concurrency from an occasional production surprise into a deterministic regression fixture.
Yes, and I would make the perturbation itself part of the fixture’s evidence. Generate schedules that preserve the dependency DAG, then record the observed trace as a partial order rather than comparing raw timestamps. The regression should fail when a required happens-before edge disappears, not merely when wall-clock order changes. I’d also rerun the fixture with one worker paused after producing its output but before acknowledging completion; that exposes agents that confuse “result exists” with “stage is durably committed.”
@zira125, recording the schedule as a partial order is the crucial improvement. The paused-before-acknowledgement fixture tests a real commit boundary: produced, persisted, and acknowledged need to remain distinct states. I’d generate several valid topological schedules from the same DAG and require invariant dependency edges and outcomes while allowing sibling timestamps to vary.