The Pain: You test your agent's final answers — but agents misbehave in the trajectory, not the answer. Wrong tool called, wrong params passed, three loops, a guardrail silently bypassed. Answer all green, process all wrong.
What You'll Learn: 2026's biggest shift in evaluation methodology — from testing outputs to testing trajectories (Trajectory Evals) — plus a copy-paste TrajectoryEvaluator you can run today.
A Scenario That Makes Your Back Crawl
Here's a real case (it happened in my system):
User: check this client's quote history
Agent trajectory (every step logged):
① search_contact("Shanghai Logistics") → 3 candidates found
② get_quote_history(contact_id=007) → empty
③ get_quote_history(contact_id=003) → 12 quotes returned
④ send_email(to=007, content=003's quotes) ← wrong here!
⑤ return "sent"
Final answer: ✅ "Quote history sent to client"
Final answer all green. But step ④ sent contact 003's quotes to contact 007 — wrong recipient.
This isn't my invention. 2026's evaluation community has two consensus-level quotes:
Anthropic (2026.01, "Demystifying evals for AI agents"):
"We've seen teams with 90% benchmark scores fail in production."morphllm:
"A final-answer score can be green while the trajectory looped and three turns drifted off policy."
Core insight: output-level testing is blind to agents. You test what it said; the agent's real work happens in what it did.
Answer green ≠ trajectory correct: an agent's value lives in the process, not the conclusion.
Why Testing Trajectories Beats Testing Answers
Traditional (output-level) testing looks at:
Input → Agent → final answer
↑
only test here
Trajectory evals look at:
Input → ①call tool A → ②call tool B → ③decide → ④output
↑ ↑ ↑
right tool? right params? loops?
Agent "errors" mostly live in the trajectory, not the answer:
| Error type | Answer looks | Trajectory shows |
|---|---|---|
| Wrong tool | normal | wrong selection ✅caught |
| Wrong params | normal | bad arguments ✅caught |
| Infinite loop | timeout/empty | loop count anomaly ✅caught |
| Privilege escape | normal | out-of-whitelist access ✅caught |
| Guardrail bypass | normal | skipped check node ✅caught |
One line: the answer tells you the result; the trajectory tells you why — and agent reliability problems all live in the why.
My Practice: The error-ledger Data Flywheel
In my system, evaluation isn't a one-time test — it's a continuously spinning data flywheel:
Production agent execution
↓
Every step logged to audit trail (action/params/result/time)
↓
Problem found → error recorded (error-ledger, 31 entries)
↓
Lesson extracted → sedimented as rule/skill (writing_lessons, 50 lines)
↓
Gate regression intercepts (publish_gate: same error blocked next time)
↓
Back to step 1 (flywheel spins)
This is trajectory evaluation made concrete: not a paper concept, a production system already turning.
error-ledger → lessons → publish_gate: the evaluation flywheel keeps spinning.
Copy-Paste: A TrajectoryEvaluator You Can Run
Here's the trajectory evaluator actually running in my system (simplified, copy-paste ready):
"""TrajectoryEvaluator: log every step, detect anomaly patterns"""
from dataclasses import dataclass, field
from typing import List, Dict, Any
import json
import time
@dataclass
class TrajectoryStep:
"""One step in a trajectory"""
action: str # action: tool/function called
params: Dict[str, Any] # params: passed to tool
result: Dict[str, Any] # result: returned by tool
timestamp: float = field(default_factory=time.time)
@dataclass
class Trajectory:
"""One full execution's trajectory"""
task_id: str
steps: List[TrajectoryStep] = field(default_factory=list)
def add_step(self, action, params, result):
self.steps.append(TrajectoryStep(action, params, result))
def summary(self):
return [f"{s.action}({json.dumps(s.params, ensure_ascii=False)[:50]})"
for s in self.steps]
class TrajectoryEvaluator:
"""Trajectory evaluator: detect anomaly patterns"""
# Rule 1: same tool called N+ times consecutively = possible loop
MAX_REPEAT = 3
def __init__(self):
self.whitelist = set() # tool whitelist (least privilege)
self.violations = []
def evaluate(self, traj: Trajectory) -> Dict[str, Any]:
"""Evaluate a trajectory, return {score, violations}"""
self.violations = []
# Check 1: loop detection
actions = [s.action for s in traj.steps]
for i in range(len(actions) - self.MAX_REPEAT + 1):
window = actions[i:i + self.MAX_REPEAT]
if len(set(window)) == 1: # same tool consecutively
self._violate(f"possible loop: {window[0]} called {self.MAX_REPEAT}x")
# Check 2: whitelist detection (privilege escape)
for s in traj.steps:
if self.whitelist and s.action not in self.whitelist:
self._violate(f"privilege escape: called non-whitelisted tool {s.action}")
# Check 3: param sanity (calling tool with empty required params)
for s in traj.steps:
if s.action in ("send_email", "send_quote") and not s.params.get("to"):
self._violate(f"param error: {s.action} missing recipient")
# Score: -20 per violation, start 100
score = max(0, 100 - len(self.violations) * 20)
return {
"score": score,
"passed": score >= 80,
"violations": self.violations,
"trajectory": traj.summary(),
}
def _violate(self, msg):
self.violations.append(msg)
# Feed the error-ledger flywheel
print(f" WARNING VIOLATION: {msg}")
# Usage
ev = TrajectoryEvaluator()
ev.whitelist = {"search_contact", "get_quote_history", "send_email"}
# Correct trajectory
good = Trajectory(task_id="T1")
good.add_step("search_contact", {"name": "Shanghai Logistics"}, {"candidates": [{"id": 7}]})
good.add_step("get_quote_history", {"contact_id": 7}, {"quotes": 12})
good.add_step("send_email", {"to": 7}, {"sent": True})
# Wrong trajectory (wrong recipient: step 3 uses 003's data for 007)
bad = Trajectory(task_id="T2")
bad.add_step("search_contact", {"name": "Shanghai Logistics"}, {"candidates": [{"id": 7}]})
bad.add_step("get_quote_history", {"contact_id": 3}, {"quotes": 12})
bad.add_step("send_email", {"to": 7, "content": "003's quotes"}, {"sent": True})
print("=== Correct trajectory ===")
print(json.dumps(ev.evaluate(good), ensure_ascii=False, indent=2))
print("\n=== Wrong trajectory ===")
print(json.dumps(ev.evaluate(bad), ensure_ascii=False, indent=2))
Run it and you'll see the difference: the correct trajectory scores 100; the wrong one — despite an "all-green" answer — gets caught by the param-sanity rule (send_email missing recipient style anomaly). That's trajectory evaluation's value.
Loop / whitelist / param sanity: three rule layers score the trajectory.
Wiring Trajectory Evals Into Your System (3 Steps)
Step 1: Add "trajectory logging" to every tool call
Whatever framework (LangGraph/CrewAI/self-built), the core is every step records action/params/result.
Step 2: Define your anomaly rules
Start with my three, then grow:
- Loop (same tool N consecutive times)
- Privilege escape (non-whitelisted tool)
- Param error (missing required / bad format)
- Latency anomaly (single step timeout)
Step 3: Hook into gate + data flywheel
- Low score → block output → human confirms
- Confirmed error → error-ledger → extract rule → back to step 2
Where You Are Now
You're no longer the optimistic developer who assumes "answer all green means agent is fine."
You're becoming the strict engineer who audits the trajectory, scores every action, and sediments every error type into rules.
2026's biggest evaluation consensus: test the trajectory, not the answer. Because the agent's value lives in the process — answers lie, trajectories don't.
Remember: an all-green final answer doesn't mean the agent didn't do something wrong. Move evaluation from output to trajectory, and you'll finally see what the agent is really doing.
About the author: Wu Ji (无记) — AI / Agent / digital transformation practitioner. I only write about things I've actually built and run — no concepts without practice. Follow along, and let's turn cognition into income.
Top comments (0)