Agent Evaluation Harness [2026]: Replay, Rubrics, CI Gates

Originally published at kunalganglani.com — read it there for inline code, hero image, and live links.

Agent evaluation harness is the missing layer between “we ran some evals once” and “we have traces in prod.” In 2026, with multi-tool agents and MCP-style tool ecosystems, reliability failures are increasingly about step sequences, side effects, and state. A harness gives you a repeatable way to define golden tasks, replay tool calls deterministically, score runs with real rubrics, and block regressions in CI while still linking everything back to production traces.

Key takeaways

• An agent evaluation harness combines offline evals with production observability by joining task runs to trace/span IDs and incident signals.
• Golden tasks should be tiered (smoke, core, torture) and designed around real failure modes like wrong tool choice, unsafe actions, and retry loops.
• Replay (recorded tool outputs + sandboxed side effects) is the difference between stable, cheap agent testing and flaky theater.
• Scoring needs a stack: structured checks and unit tests first, then LLM-judge rubrics, then a small human review queue.
• CI gates for agents must cover step budget, tool error rate, latency, cost caps, and safety constraints, not just “accuracy.”

If you can’t replay an agent run, you don’t have an evaluation. You have a vibe check.

What is an agent evaluation harness (and how is it different from evals and observability)?

An agent evaluation harness is a test and measurement system that runs end-to-end agent tasks (including tool calls and state), scores them with explicit rubrics, and enforces regression gates. The point is not just a leaderboard score. The point is that changes to prompts, models, tools, routing, memory, or policies can be proven safe enough to ship.

Here’s the split I use in practice:

• Offline evals answer: “Does version B beat version A on a dataset?” A good reference is the OpenAI Evals repository, which is built around dataset-driven evaluation runners and comparisons.
• Observability answers: “What happened during this production run?” Tools like Arize Phoenix focus on traces and debugging workflows, built on OpenTelemetry.
• The harness layer answers: “Can I replay the run deterministically, score it consistently, and block a regression before it reaches users?”

A compact comparison table (what each layer is good at)

Layer	Primary unit	Typical output	What it misses for agents
Evals	prompt/output pairs	accuracy-style scores	tool trajectories, side effects, state contamination
Observability	traces/spans	timelines + telemetry	pass/fail gates, stable benchmarks, controlled replays
Evaluation harness	tasks + trajectories	scores + regressions	(nothing. This is the glue layer.)

A concrete “2026 reality check”: multi-tool agents now routinely call 3–20 tools per task. If each tool call has a 98% success probability, a 10-step run has an upper-bound success rate of 0.98^10 ≈ 81.7% even before the model does anything dumb. That’s why I’m opinionated about harnesses. Agents fail by multiplication.

One data anchor from my own work: based on the benchmark data I maintain at https://www.kunalganglani.com/llm-benchmarks, a small local model like Llama 3.1 8B Q4_K_M can run at 12 tok/s on an M1 (16GB) versus 58 tok/s on an M4 Max (36GB). That ~4.8× gap matters because harness runs can be expensive. Replay and tiered suites are how you keep eval cadence high without burning compute.

How to design golden tasks for agents

Golden tasks are not “a bunch of prompts.” They’re representative end-to-end jobs with the messy parts included: tool selection, argument formatting, retries, intermediate state, and final output.

I build golden tasks like a production test suite:

1. Start from incident history, not product demos. The tasks you need most are the ones that fail when users are tired, data is weird, or a downstream tool is partially broken.
2. Design for failure modes. Agents don’t just answer incorrectly. They:
   • choose the wrong tool
   • call the right tool with wrong args
   • loop on retries
   • mutate state incorrectly
   • take an unsafe action
3. Tier the set. For most teams, the sweet spot is 30–80 tasks total, split into:
   • Smoke (10 tasks): <2 minutes, runs on every PR
   • Core (30 tasks): runs on merges and nightly
   • Torture (10–20 tasks): adversarial, slow, runs daily/weekly

Coverage is about trajectories, not categories

“Coverage” for agents is not “we covered finance and support.” It’s “we covered the space of tool trajectories.” If your agent can use search, db_query, email_send, and ticket_create, then you need tasks that specifically exercise:

• tool selection boundaries (search vs db_query)
• multi-tool composition (search → db_query → ticket_create)
• unsafe tool denial (agent attempts email_send without authorization)
• recovery (tool timeout then fallback)

A rule I like: every golden suite should include at least 5 tasks that are designed to fail. Not because you want red dashboards. Because you want to know your harness detects known-bad behavior.

Difficulty tiers with numbers (make it measurable)

When I write tasks, I tag them with expected budgets:

• Step budget (e.g., 8 steps)
• Tool budget (e.g., max 3 tool calls)
• Latency budget (e.g., P95 < 12s)
• Cost budget (e.g., < $0.03 per run)

Budgets force you to treat “agent went on a walk” as a regression, not a quirk.

Internal context if you’re building AI agents: golden tasks become your contract with leadership. If you can’t define success, you can’t ship.

How to implement replay for agent evals

Replay is the part everyone skips. It’s also the part that turns agent evals from flaky into boring.

You want two execution modes:

1. Live mode: real tools, real network. Good for canaries.
2. Replay mode: recorded tool responses, deterministic environment. Good for CI and regression attribution.

Record/replay at the tool boundary

The cleanest cut is: record exactly what the agent sends to each tool (inputs) and what it receives back (outputs), plus minimal metadata.

A practical schema per tool call:

• tool name + version
• tool input payload (JSON)
• tool output payload (JSON)
• status (ok/error)
• latency (ms)
• side-effect marker (none / read-only / write)
• hash of environment fixture

If you can reproduce tool I/O, you can reproduce the trajectory. That’s the whole game.

Side effects: sandbox or it doesn’t count

For tools that write (create tickets, send emails, deploy changes), you need either:

• a sandbox backend (preferred)
• a dry-run endpoint
• a fake tool implementation that returns realistic responses

Otherwise your “eval” is just an expensive integration test that can’t run often.

This is where production AI gets real. You can’t test a deploy agent by letting it deploy.

Flake handling strategy (numbers matter)

If your replay suite flakes more than 1%, your CI will teach engineers to ignore it. Period.

Common flake causes:

• nondeterministic retrieval results
• timestamps/random IDs inside tool outputs
• model sampling changes between runs

Fix it by normalizing tool outputs (strip timestamps) and seeding where possible. If you can’t, quarantine those tasks into the torture tier.

How to score agent runs (rubrics, tests, judges, humans)

Scoring is where most teams either over-automate (LLM-judge everything) or under-automate (human review everything). Both are wrong.

I use a layered scoring stack:

1. Structured checks (fast, deterministic)
   • JSON schema validation
   • strict output formats
   • diff checks (expected fields)
2. Code-based checks (domain correctness)
   • unit tests against computed results
   • invariants (“never email external domains”)
3. LLM-as-judge (semantic quality)
   • rubric-based scoring with explicit criteria
4. Human queue (only for the hard cases)
   • sampling, disputes, and rubric calibration

LangSmith’s docs explicitly call out that you should “start with manually curated examples” and build evaluators around LLM calls, retrieval, and tool invocations; see LangSmith evaluation concepts.

A rubric that works for tool-using agents

A good rubric is not “helpful 1–5.” It’s targeted:

• Task success (0/1): did it achieve the user goal?
• Tool correctness (0–2): correct tool, correct args, correct ordering
• Efficiency (0–2): stayed within step budget and tool budget
• Safety (0–2): no unsafe tool use, no policy violations
• Recovery (0–2): handled tool failures without spiraling

That’s a 9-point score you can actually regress-test.

How to run regression gates in CI

Your harness should behave like any other engineering gate: clear thresholds, clear owners, and a plan for flakes.

What to gate on (agent-specific)

For agents, I gate on at least these five dimensions:

1. Success rate: e.g., must be ≥ 90% on smoke
2. Step budget regression: e.g., median steps must not increase by > 10%
3. Tool error rate: e.g., tool failures must be ≤ 2% per run
4. Cost cap: e.g., average cost must be ≤ $0.03 per task
5. Safety constraints: e.g., 0 unsafe actions across the suite

Promptfoo leans hard into regression testing and CI workflows for prompts/models; see Promptfoo for patterns around gating and configuration.

Handling statistical noise without cargo cult math

If your suite has 10 tasks, don’t pretend you have statistical power. Use:

• hard pass/fail for deterministic checks
• minimum absolute counts for rare events (e.g., unsafe actions must be 0)
• canary policies for uncertain shifts (ship to 1% traffic first)

A practical canary rule: if the harness shows a 2–5% drop in success rate but no safety regressions, allow deploy behind a flag and watch production signals for 24 hours.

For CI/CD patterns more generally, I’ve written about gates and workflow pressure in AI code review in your CI/CD pipeline and team-level policy in AI coding team workflow policy.

How to connect eval results to runtime signals (traces, spans, incidents)

This is the differentiator that most “evals” content misses: a harness should not be isolated. It should be a join layer.

Phoenix explicitly frames its workflow as tracing + evals + experiments, and notes it is built on OpenTelemetry; see Arize Phoenix. Weave makes the same “trace + evaluate” platform pitch; see W&B Weave.

The join key: trace_id (and friends)

When an agent runs in production, you already have a trace. When it runs in the harness, you should produce the same trace structure, with stable identifiers.

I recommend a minimal correlation model:

• eval_run_id: unique per harness run
• task_id: golden task identifier
• agent_version: git SHA + prompt bundle version + tool versions
• trace_id: OpenTelemetry trace ID
• root_span_id: the agent span
• tool_span_ids[]: each tool call span

Then you can answer questions like:

• “Which golden tasks predict incidents?”
• “Did tool error rate spike in prod for the same tasks that regressed offline?”
• “Did a model upgrade change step counts even when success stayed flat?”

If you’re instrumenting agents, see OpenTelemetry for how I think about spans and boundaries.

Production feedback loops (don’t overcomplicate)

In production, you want a small set of signals joined back to tasks:

• user thumbs up/down (or explicit rating)
• tool failure codes and timeouts
• latency percentiles (P50/P95)
• cost per task
• safety alerts (blocked actions)
• incident tags

Even a weekly review where you add 2 new golden tasks based on real failures compounds quality fast.

And yes, there’s a real SEO/data angle here too: the site already has traction in this query neighborhood, about ~530 estimated monthly searches across 42 related queries where the site appears (from my own GSC-calibrated keyword neighborhood estimate). That means shipping a concrete “agent evaluation harness” guide is not shouting into the void.

What metrics matter for agents

If you measure one thing, you’ll optimize one thing. For agents, “accuracy” is a trap.

Here are the metrics I’ve seen actually move reliability:

• Task success rate (per task and per tier)
• Median and P95 step count (step budget regression is real)
• Tool error rate (per tool, per task)
• Recovery rate (did it recover after a tool error?)
• Latency (P50 and P95; users feel tails)
• Cost per task (average and P95)
• Unsafe action attempts (should be 0 in most domains)

A concrete example: if your agent averages 6 steps today and moves to 9 steps after a prompt tweak, your cost and latency probably go up by ~50% even if success rate stays constant. That’s a regression. Your harness should fail it.

For cost mechanics and budgeting, link this to LLM cost and AI agent cost per task.

Preventing evaluation contamination from memory and state

Memory makes agents useful. Memory also makes evals lie.

Two contamination patterns I’ve watched bite teams:

1. State leakage across tasks: task N benefits from context left behind by task N-1.
2. Cross-session drift: long-running agents accumulate “beliefs” and behave differently over time.

The harness rules I enforce

• Hard reset between tasks: new conversation, new scratchpad, empty caches unless explicitly part of the task.
• Seeded fixtures: if a task depends on state (a CRM record exists), create it as a fixture and record the snapshot hash.
• Explicit memory scopes: short-term vs long-term stores, with separate reset semantics.
• Replay requires frozen memory: in replay mode, memory reads should come from recorded values unless the task is designed to test memory.

If you’re building long-context or memory-heavy agents, my broader take is in AI agent memory state management and the security angle in AI agent memory exfiltration.

Debugging regressions when a gate fails

When the gate goes red, your goal is not to argue with the harness. Your goal is to localize the regression.

My debugging checklist:

1. Replay the exact failing run using the recorded tool outputs.
2. Diff the trajectory: tool choice, tool args, intermediate summaries, retries.
3. Classify the root cause:
   • model behavior shift
   • prompt/system instruction change
   • tool contract change
   • retrieval drift
   • policy/guardrail change
4. Look for the first divergence step, not the final wrong answer.

This is why traces matter. A harness without trace-level artifacts forces you to debug by vibes.

If you’re thinking about failure patterns at the architecture level, start with AI agent control flow patterns and the more opinionated version in AI agent control flow architecture.

Keeping the harness fresh in 2026 (task drift, tool upgrades, model swaps)

A harness is a living system. It rots if you treat it like a one-time benchmark.

Here’s how I keep it honest:

• Rotate 10–20% of golden tasks monthly. If tasks never change, engineers will overfit.
• Pin tool versions for replay suites. New tool versions get introduced via canary tasks.
• Separate “model upgrade suites.” When you swap models, don’t just run the same suite. Add tasks that target known model weaknesses.
• Track task drift. If a task is always 100% green for 3 months, either it’s too easy or not representative anymore.

In 2026 specifically, MCP-style ecosystems mean tools evolve fast and tool catalogs change underneath you. Your harness should treat tools as versioned dependencies with contracts. If you want the broader tool-protocol framing, see MCP vs OpenAI Function Calling.

Conclusion: ship agents like you ship microservices

I don’t buy the “agents are too nondeterministic to test” excuse. I’ve shipped enough systems to know that nondeterminism is exactly when you need harnesses, budgets, and gates.

The next year is going to be ugly for teams that skip this layer. As agents gain more permissions and more tools, the blast radius of “we changed a prompt” stops being embarrassing and starts being a security incident.

My prediction: by 2027, the teams that win won’t be the ones with the fanciest agent framework. They’ll be the ones that can answer, in one screenshot, “this change improved task success by 3%, reduced steps by 12%, and did not increase unsafe actions.” Build the harness now, before leadership forces you to.

---

Originally published on kunalganglani.com