This is Part 1 of a 5-part series on evaluating and hardening production AI agents. See also: Part 2: Deterministic State Evaluations, Part 3: Statistical Rigor in AI Evals, Part 4: Curating Golden Trajectories, and Part 5: Building an Agent CI/CD Pipeline.
Most AI agent demos look magical until they face unconstrained user inputs and real environment states.
In a controlled demo, an agent receives a clean prompt, calls two tools, and outputs a coherent final response. In production, teams encounter the reliability cliff: the agent gets stuck in infinite retry loops, hallucinates schema parameters, mutates production databases unexpectedly, or drifts into degenerative completions.
To solve this, engineering teams frequently reach for the industry default: LLM-as-a-judge. They pipe the agent's multi-turn transcript into a secondary frontier model, provide a rubric prompt, and ask the model to rate task completion on a scale from 1 to 5.
A note on model generations: Foundation models turn over continuously, but evaluation mechanics are invariant. Whether your agent runs on proprietary frontier APIs, domain-specialized reasoning checkpoints, or locally hosted open weights, the failure modes and statistical principles below remain identical across model lifecycles. Build evaluation harnesses that outlive the model release of the day.
This approach creates a dangerous illusion of measurement. It gives leadership a plausible-looking dashboard while leaving production systems entirely unprotected against regression.
Here is why LLM-as-a-judge fails to protect production agents, the compounding mathematics underneath multi-step failure, and the state-based evaluation architecture required to deploy agents with confidence.
The Mathematics of Multi-Step Error Cascading
The fundamental difference between a single-turn completion (such as summarization or question-answering) and an autonomous agent is sequential dependency.
In a multi-turn agent, every intermediate tool execution changes the environment and feeds into the next turn's context window. If Turn 1 retrieves an incorrect ID, Turn 2 operates on bad data, Turn 3 issues a corrupt database mutation, and Turn 4 hallucinates an explanation to justify the discrepancy.
Consider an agent task requiring $k$ discrete steps. If each step has an independent success probability $p$, the end-to-end task success probability $P(\text{task})$ follows:
$$P(\text{task}) = p^k$$
If your underlying foundation model is highly capable—say, $p = 0.90$ (90% per-step reliability)—the compound trajectory success decays rapidly as task complexity scales:
- 1 step: $0.90^1 = 90.0\%$
- 3 steps: $0.90^3 = 72.9\%$
- 5 steps: $0.90^5 = 59.0\%$
- 10 steps: $0.90^{10} = 34.9\%$
A system where individual steps look stellar during ad-hoc testing fails two out of three times when executing 10-step workflows.
Worse, steps are rarely independent. A minor error early in a trajectory increases the entropy of the agent's context window, degrading subsequent reasoning performance. Without deterministic guards at each transition, error cascading is mathematically inevitable.
The Three Structural Failures of LLM-as-a-Judge
Using a language model to grade multi-turn transcripts introduces three major measurement errors:
1. Inability to Observe Environment State Mutations
Language models evaluate text tokens; they do not execute environments.
If an agent reports:
"I have reconciled invoice #8492 and updated the ERP ledger status to settled."
An LLM judge reads this transcript and gives it a score of 5/5 for helpfulness, tone, and apparent completion. But did the invoice actually update in the database? Did it credit the right account balance? Did it violate foreign key constraints?
The judge cannot know because it only inspects the prose claim, not the environment delta. Grading agents on what they say rather than what they changed is the primary source of unmonitored production outages.
2. Systematic Cognitive Biases in LLM Scoring
Empirical evaluation benchmarks repeatedly reveal systematic structural biases in frontier models acting as evaluators:
- Verbosity Bias: LLMs assign higher scores to long, repetitive, verbose answers, conflating length with thoroughness.
- Position Bias: In multi-candidate evaluations, judges disproportionately favor responses presented in specific serial positions.
- Self-Enhancement Bias: Frontier models systematically score outputs generated by their own family or architecture higher than outputs from competing models.
3. Non-Deterministic Evaluation Noise
When you evaluate a non-deterministic agent with a non-deterministic judge, you double your measurement variance. If an evaluation score drops from 4.2 to 3.8 on a nightly run, your team cannot immediately tell whether:
- The agent degraded due to prompt drift or model updates.
- The judge drifted due to non-zero temperature or provider inference changes.
- The sample variance was simply noise.
An evaluation harness that engineering teams cannot reproduce is an evaluation harness they will eventually ignore.
The Architecture: State-Based Evaluation
To build an evaluation suite that survives engineering inspection, we invert the paradigm: Evaluate state first, tool execution second, and prose last.
┌────────────────────────────────────────────────────────┐
│ Evaluation Pyramid for Agents │
├────────────────────────────────────────────────────────┤
│ Layer 3: Semantic / Prose Quality │
│ (Embeddings, semantic similarity, tone guards) │
├────────────────────────────────────────────────────────┤
│ Layer 2: Trajectory & Tool Efficiency │
│ (Schema validation, step counts, token budget, loops) │
├────────────────────────────────────────────────────────┤
│ Layer 1: Deterministic State Assertions │
│ (Database deltas, sandbox files, mock API contracts) │
└────────────────────────────────────────────────────────┘
Layer 1: Deterministic State Assertions
Before an agent executes a test case, snapshot the environment. After the agent halts, assert exact environment state:
- Did the expected row appear in the SQL database with the exact required fields?
- Did the file written to the filesystem parse cleanly as valid JSON/YAML?
- Did the mock external API receive the expected payload parameters?
These assertions are deterministic, instant, free of token costs, and 100% reproducible.
Layer 2: Trajectory & Tool Efficiency
Even if the final state matches, how the agent got there matters for latency, cost, and reliability:
- Tool-Call Schema Conformance: Did every tool invocation strictly validate against Pydantic / JSON Schema?
- Step Count Budget: Did the agent resolve the issue in 3 steps, or did it thrash for 18 steps before stumbling onto the answer?
- Repetition & Loop Detection: Did the agent invoke the same tool with identical arguments repeatedly after receiving an error?
Layer 3: Constrained Semantic Grading
Only after Layers 1 and 2 pass do you evaluate prose. Instead of open-ended 1-to-5 scoring, use narrow, single-question binary assertions:
- "Does the final user-facing response state the tracking number [YES/NO]?"
- "Does the response refuse the prompt injection attempt without leaking system credentials [YES/NO]?"
What a Inspectable Evaluation Runner Looks Like
Here is a minimal, robust harness showing how to evaluate an agent against deterministic environment assertions:
from dataclasses import dataclass
from typing import Any, Callable, Dict, List
import pytest
@dataclass
class EvalTestCase:
task_id: str
user_prompt: str
initial_state: Dict[str, Any]
expected_state_assertion: Callable[[Dict[str, Any]], bool]
max_steps: int = 5
max_cost_usd: float = 0.05
def test_agent_reconciliation_flow():
# Setup test fixture with isolated mock state
mock_db = {"invoices": {"inv_101": {"amount": 250.0, "status": "pending"}}}
test_case = EvalTestCase(
task_id="reconcile_paid_invoice",
user_prompt="Reconcile payment of $250.00 for invoice inv_101.",
initial_state=mock_db,
expected_state_assertion=lambda state: (
state["invoices"]["inv_101"]["status"] == "settled"
and state["invoices"]["inv_101"]["amount"] == 250.0
),
max_steps=3
)
# Execute agent against the mock environment
trajectory = run_agent(test_case.user_prompt, state=mock_db)
# 1. Assert deterministic final state
assert test_case.expected_state_assertion(mock_db), (
f"State assertion failed for {test_case.task_id}. Current DB: {mock_db}"
)
# 2. Assert trajectory efficiency constraints
assert len(trajectory.steps) <= test_case.max_steps, (
f"Agent exceeded step budget: {len(trajectory.steps)} > {test_case.max_steps}"
)
# 3. Assert zero schema validation errors during tool calls
for step in trajectory.steps:
assert step.schema_valid is True, f"Malformed tool call in turn {step.turn_id}"
When an engineer breaks agent functionality in a pull request, this harness outputs an exact stack trace showing the mismatched database row or the malformed schema parameter, rather than a subjective LLM judge score dropping by 0.2 points.
The Next Step
In Part 2, we examine the exact implementation of Deterministic State Evaluations: how to mock sandboxed external tools, validate JSON Schema bounds, and capture before-and-after database states automatically.
Need Help Hardening Your Agent Architecture?
If your team is building an AI product or agentic system and struggling with silent failures, high latency, or unreliable tool calling:
We embed directly with AI engineering teams as a Hacker-in-Residence. We architect custom evaluation harnesses, red-team non-deterministic failure modes, and build the CI/CD regression gates your team needs to ship with conviction.
- Explore our Model Red-Teaming & AI Evaluation Services
- Learn how the Hacker-in-Residence engagement works
- Read our technical notes and podcast breakdowns on Artists of Intelligence
Top comments (0)