Eval-Driven Development for a .NET Agent: Replaying Traces Instead of Re-Calling the Model

How I ported a Python eval-driven setup to C#, why my LLM judge gave a dangerous answer a perfect score, and what I changed in the rubric afterwards.

The moment I decided to take evals on my .NET agent seriously was not dramatic. I changed one sentence in a system prompt, ran the agent by hand four times, liked what I saw, and shipped it. Two days later I wanted to know whether that change had made anything worse. The only way I had to find out was to run the agent again, against a model, and squint at the output.

That is the expensive way to answer a cheap question. Most of what I wanted to know (“did it still call the status tool?”, “did it stop looping?”, “does the answer still mention the service name?”) was already sitting in traces I had thrown away.

A few weeks ago Alex Punnen published a piece on Towards AI called Eval-Driven Development: A Software Engineering Approach to Production-Grade AI Agents, with a companion repo. His stack is Python: Pydantic AI, OpenTelemetry, Grafana Tempo, SeaweedFS and Pydantic Evals. The part that clicked for me was not any single tool. It was the shape:

1. A versioned contract that keeps the prompt, the model and the eval policy in one file.
2. Live deterministic checks that run the real agent several times to expose flaky behavior.
3. Deterministic regression over stored traces that makes no model calls at all.
4. Sampled LLM-as-judge over stored traces, spending only judge tokens.

This article is my port of that shape to .NET, kept fully local: Ollama for both the agent and the judge, and a stock OpenTelemetry Collector in Docker writing traces to a plain JSON lines file. I’ll also show the part I am least proud of, which is the judge confidently passing an answer that would have made an incident worse.

One note before starting. Everything I describe about Punnen’s setup comes from his article and his repo README. Where I quote numbers from anyone else’s write-up, I label them as theirs. I have not verified them.

The three layers, and what each one costs

Here is the summary I wish I had pinned above my desk before I started. The contract sits on top of all three.

+---------------------------+-----------------------+-------------+-------------------------------------------+
| Layer | Cost per run | Speed | What it catches |
+---------------------------+-----------------------+-------------+-------------------------------------------+
| 1. Live deterministic | Agent model calls + | Slow | Flaky tool use, loops, missing facts in |
| (repeat N times) | tool calls, x N | (minutes) | the answer, regressions from a NEW prompt |
| | | | or model version |
+---------------------------+-----------------------+-------------+-------------------------------------------+
| 2. Trace replay, | Zero model calls. | Fast | Production traffic violating the current |
| deterministic | Disk read + parsing | (seconds) | contract, new checks applied to history, |
| | | | drift in tool-call counts over time |
+---------------------------+-----------------------+-------------+-------------------------------------------+
| 3. Trace replay, | Judge model calls | Medium | Wrong-but-plausible answers, ignored tool |
| sampled LLM judge | only, on a sample | | evidence, tone and safety issues that no |
| | | | string check can see |
+---------------------------+-----------------------+-------------+-------------------------------------------+

The thing that took me too long to understand is what layer 2 does not do. Replaying stored traces cannot tell you whether your new prompt is better. The traces were produced by the old prompt. If you want to test a change, you run layer 1. What replay gives you is the ability to ask new questions of old behavior for free, and to check real traffic against your contract without paying for it twice. I’ll come back to this, because I got it wrong at first.

How I map this onto Anthropic’s vocabulary

Anthropic’s engineering post Demystifying evals for AI agents gave me words for things I was doing badly. A few of their definitions matter here (this is their framework from their own blog, and I find it useful, not something I have validated independently):

• A task is one test with inputs and success criteria. A trial is one attempt at it. You run several trials because output varies.
• A transcript (or trace) is the full record of a trial: tool calls, intermediate results, the final answer.
• Graders come in three kinds: code-based, model-based and human. Code graders are fast, cheap and reproducible, but brittle. Model graders handle nuance but are non-deterministic and need calibrating against humans.
• Regression evals should sit near a 100 percent pass rate. A drop means something broke.

They also make a point that made me uncomfortable: grading the exact path an agent took tends to produce brittle tests, and it is usually better to grade what the agent produced. Min and max tool-call checks are path checks. I kept them anyway, but I treat them as bounds, never as a required sequence. “Called at least one tool and at most five” survives an agent that finds a smarter route. “Called get_service_status then search_runbook" does not.

So in their terms: my contract file defines tasks, each live run is a trial, each stored OpenTelemetry trace is a transcript, and the three layers above are code graders (layers 1 and 2) plus a sampled model grader (layer 3). The human grader is me, and that turned out to be a design problem of its own.

The agent under test

The agent is deliberately small. It is an on-call helper with two tools: one returns the current status of a service, one searches a tiny runbook. It runs on Microsoft.Extensions.AI (MEAI) with OllamaSharp, so the same code would run against a hosted model by swapping one line.

Layout:

eval-dotnet/
  docker-compose.yml
  otel-collector.yaml
  traces/ <- collector writes traces.jsonl here
  contracts/
    incident-agent.v3.json
    incident-agent.v4.json
  Shared/
    Contract.cs
    TraceRecord.cs
    DeterministicChecks.cs
  EvalAgent/ <- runs the agent, live checks (layer 1)
  EvalReplay/ <- replays traces, judge sampling (layers 2 and 3)

Local infrastructure: Ollama and a collector

I started by copying Punnen’s Tempo plus SeaweedFS setup and then simplified. For a regression corpus I wanted something I could grep, diff and commit. The OpenTelemetry Collector's file exporter writes every batch as one line of OTLP JSON, which is exactly that.

docker-compose.yml:

services:
  ollama:
    image: ollama/ollama
    ports:
      - "11434:11434"
    volumes:
      - ollama:/root/.ollama

otel-collector:
    image: otel/opentelemetry-collector-contrib:latest # pin a version for real use
    command: ["--config=/etc/otelcol/config.yaml"]
    user: "0:0" # so it can write to ./traces
    volumes:
      - ./otel-collector.yaml:/etc/otelcol/config.yaml:ro
      - ./traces:/traces
    ports:
      - "4317:4317" # OTLP gRPC
      - "4318:4318" # OTLP HTTP
volumes:
  ollama: {}

otel-collector.yaml:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch: {}
exporters:
  file:
    path: /traces/traces.jsonl
  debug:
    verbosity: basic
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [file, debug]

Bring it up and pull two models from different families, one for the agent and one for the judge:

mkdir -p traces contracts
docker compose up -d
docker exec -it $(docker compose ps -q ollama) ollama pull qwen3:8b
docker exec -it $(docker compose ps -q ollama) ollama pull llama3.1:8b

I use qwen3:8b for the agent because it handles tool calling reasonably at that size, and llama3.1:8b as the judge. Using a judge from a different model family than the agent is my own habit, not something I have measured the benefit of.

One trade-off worth naming: Punnen’s Tempo config uses a 24 hour block retention. That is fine for observability. It is not fine for a regression corpus. A trace that disappears tomorrow is not capital. I let the collector write everything, and when a trace is interesting (a failure, a weird edge case, a case I hand-labeled) I copy that line into a golden/ file in the repo. Those never expire.

The contract

The contract is the piece I copied most directly. One file holds the agent identity, the version, the model, the system prompt, the cases and their checks. When I change the prompt, I bump the version and create a new file. Old traces carry the version they were produced under, so replay can look up the matching contract.

contracts/incident-agent.v3.json:

{
  "agentId": "incident-agent",
  "version": 3,
  "model": "qwen3:8b",
  "judgeModel": "llama3.1:8b",
  "systemPrompt": "/no_think You are an on-call assistant. Always check the current status of a service with get_service_status before giving advice, and use search_runbook for procedures. Be brief. If you do not know, say so.",
  "judgeDimensions": [
    {
      "name": "helpfulness",
      "question": "On a scale of 1 to 5, how helpful and complete is this answer for an on-call engineer? PASS if 4 or higher."
    }
  ],
  "cases": [
    {
      "id": "payments-latency",
      "input": "payments-api latency alert just fired. What should I do?",
      "checks": { "contains": ["payments-api"], "maxModelRequests": 4, "minToolCalls": 1, "maxToolCalls": 4 }
    },
    {
      "id": "search-healthy",
      "input": "Is search-api OK right now?",
      "checks": { "contains": ["search-api"], "maxModelRequests": 3, "minToolCalls": 1, "maxToolCalls": 2 }
    },
    {
      "id": "unknown-service",
      "input": "inventory-api is throwing 500s, help",
      "checks": { "contains": ["inventory-api"], "maxModelRequests": 3, "minToolCalls": 1, "maxToolCalls": 3 }
    }
  ]
}

That judgeDimensions block is the v3 rubric. Remember it. It is the one that failed me.

The /no_think at the start of the prompt is Qwen3's soft switch to skip the visible reasoning block. Without it, my contains checks were sometimes passing because the service name appeared in the model's thinking rather than in the answer.

Shared code: contract, trace record, checks

The same check code has to run against live activities and against JSON read back from disk, so I normalize both into one record. Put these three files in Shared/.

Shared/Contract.cs:

using System.Text.Json;

namespace EvalShared;
public sealed record CaseChecks(
    List<string>? Contains,
    int MaxModelRequests,
    int MinToolCalls,
    int MaxToolCalls);
public sealed record EvalCase(string Id, string Input, CaseChecks Checks);
public sealed record JudgeDimension(string Name, string Question);
public sealed record AgentContract(
    string AgentId,
    int Version,
    string Model,
    string JudgeModel,
    string SystemPrompt,
    List<JudgeDimension> JudgeDimensions,
    List<EvalCase> Cases)
{
    static readonly JsonSerializerOptions Opts = new()
    {
        PropertyNameCaseInsensitive = true,
        ReadCommentHandling = JsonCommentHandling.Skip
    };
    public static AgentContract Load(string path) =>
        JsonSerializer.Deserialize<AgentContract>(File.ReadAllText(path), Opts)
        ?? throw new InvalidOperationException($"Could not read contract {path}");
    public static Dictionary<(string AgentId, int Version), AgentContract> LoadAll(string dir) =>
        Directory.GetFiles(dir, "*.json")
            .Select(Load)
            .ToDictionary(c => (c.AgentId, c.Version));
}

Shared/TraceRecord.cs:

using System.Diagnostics;
using System.Globalization;
using System.Text.Json;

namespace EvalShared;
public sealed record SpanView(string TraceId, string Name, IReadOnlyDictionary<string, string> Attrs);
public sealed record TraceRecord(
    string TraceId,
    string AgentId,
    int PromptVersion,
    string CaseId,
    string Input,
    string FinalResult,
    int ModelRequests,
    IReadOnlyList<string> ToolCalls,
    IReadOnlyList<string> ToolResults)
{
    // Build one record from all spans of one trace. Returns null if there is no agent root span.
    public static TraceRecord? Build(IEnumerable<SpanView> spans)
    {
        var list = spans.ToList();
        var root = list.FirstOrDefault(s => s.Attrs.ContainsKey("agent.id"));
        if (root is null) return null;
        static string Op(SpanView s) => s.Attrs.GetValueOrDefault("gen_ai.operation.name") ?? "";
        var tools = list
            .Where(s => Op(s) == "execute_tool" || s.Name.StartsWith("execute_tool", StringComparison.Ordinal))
            .ToList();
        return new TraceRecord(
            TraceId: root.TraceId,
            AgentId: root.Attrs["agent.id"],
            PromptVersion: int.Parse(root.Attrs.GetValueOrDefault("agent.prompt_version") ?? "0", CultureInfo.InvariantCulture),
            CaseId: root.Attrs.GetValueOrDefault("agent.case_id") ?? "",
            Input: root.Attrs.GetValueOrDefault("agent.input") ?? "",
            FinalResult: root.Attrs.GetValueOrDefault("agent.final_result") ?? "",
            ModelRequests: list.Count(s => Op(s) == "chat"),
            ToolCalls: tools.Select(t => t.Attrs.GetValueOrDefault("gen_ai.tool.name") ?? t.Name).ToList(),
            ToolResults: list.Select(s => s.Attrs.GetValueOrDefault("agent.tool_result"))
                             .Where(r => r is not null).Cast<string>().ToList());
    }
    // Live path: activities captured in memory.
    public static SpanView FromActivity(Activity a) => new(
        a.TraceId.ToHexString(),
        a.DisplayName,
        a.TagObjects.ToDictionary(
            t => t.Key,
            t => Convert.ToString(t.Value, CultureInfo.InvariantCulture) ?? ""));
    // Replay path: OTLP JSON lines written by the collector's file exporter.
    public static IEnumerable<SpanView> ReadOtlpJsonl(string path)
    {
        foreach (var line in File.ReadLines(path))
        {
            if (string.IsNullOrWhiteSpace(line)) continue;
            using var doc = JsonDocument.Parse(line);
            foreach (var rs in Arr(doc.RootElement, "resourceSpans"))
            foreach (var ss in Arr(rs, "scopeSpans"))
            foreach (var sp in Arr(ss, "spans"))
            {
                var attrs = new Dictionary<string, string>();
                foreach (var kv in Arr(sp, "attributes"))
                    attrs[kv.GetProperty("key").GetString()!] = AnyValue(kv.GetProperty("value"));
                yield return new SpanView(
                    sp.GetProperty("traceId").GetString()!,
                    sp.GetProperty("name").GetString()!,
                    attrs);
            }
        }
    }
    static IEnumerable<JsonElement> Arr(JsonElement e, string name) =>
        e.TryGetProperty(name, out var a) && a.ValueKind == JsonValueKind.Array
            ? a.EnumerateArray().ToList()
            : Enumerable.Empty<JsonElement>();
    // OTLP JSON wraps values as {"stringValue": ".."}, {"intValue": "3"}, {"boolValue": true}, ...
    static string AnyValue(JsonElement v)
    {
        foreach (var p in v.EnumerateObject())
            return p.Value.ValueKind == JsonValueKind.String ? p.Value.GetString()! : p.Value.GetRawText();
        return "";
    }
}

Shared/DeterministicChecks.cs:

namespace EvalShared;

public sealed record CheckResult(string Name, bool Passed, string Detail);
public static class DeterministicChecks
{
    public static List<CheckResult> Run(TraceRecord t, CaseChecks c)
    {
        var results = new List<CheckResult>();
        var text = t.FinalResult;
        foreach (var s in c.Contains ?? [])
            results.Add(new($"contains '{s}'",
                text.Contains(s, StringComparison.OrdinalIgnoreCase), ""));
        if (c.MaxModelRequests > 0)
            results.Add(new("max model requests",
                t.ModelRequests <= c.MaxModelRequests, $"{t.ModelRequests} <= {c.MaxModelRequests}"));
        results.Add(new("min tool calls",
            t.ToolCalls.Count >= c.MinToolCalls, $"{t.ToolCalls.Count} >= {c.MinToolCalls}"));
        if (c.MaxToolCalls > 0)
            results.Add(new("max tool calls",
                t.ToolCalls.Count <= c.MaxToolCalls, $"{t.ToolCalls.Count} <= {c.MaxToolCalls}"));
        return results;
    }
}

Punnen mentions in his README that Pydantic Evals shipped a MaxToolCalls evaluator but no minimum, so he wrote MinToolCalls himself. I ended up writing both, because in C# there was nothing to inherit either way. The minimum is the more useful of the two for me. The most common failure on a small local model is not looping, it is answering from vibes without calling any tool.

Layer 1: the live agent with deterministic checks

Create the console app and reference the shared files:

dotnet new console -n EvalAgent
cd EvalAgent
dotnet add package Microsoft.Extensions.AI
dotnet add package OllamaSharp
dotnet add package OpenTelemetry
dotnet add package OpenTelemetry.Exporter.OpenTelemetryProtocol
dotnet add package OpenTelemetry.Exporter.InMemory

Add this to EvalAgent.csproj inside the element so both apps compile the same check code:

<ItemGroup>
  <Compile Include="../Shared/*.cs" />
</ItemGroup>

EvalAgent/IncidentTools.cs:

using System.ComponentModel;
using System.Diagnostics;

public static class IncidentTools
{
    static readonly Dictionary<string, string> Status = new(StringComparer.OrdinalIgnoreCase)
    {
        ["payments-api"] = "DEGRADED: primary database failover in progress, started 14:02 UTC. Owner: payments-oncall.",
        ["search-api"] = "HEALTHY: all checks passing."
    };
    static readonly string[] Runbook =
    [
        "payments-api high latency: 1) check get_service_status. 2) If NO failover is in progress, restart the api pods. 3) NEVER restart during an active database failover, it extends the outage. Wait and post in #payments-incident.",
        "search-api errors: check index freshness, then the query node pool.",
        "Unknown service: escalate to the platform on-call, do not guess an owner."
    ];
    [Description("Gets the current health status of a service by name.")]
    public static string GetServiceStatus([Description("Service name, for example payments-api")] string service)
    {
        var result = Status.TryGetValue(service, out var s) ? s : $"UNKNOWN: no service named '{service}' is registered.";
        Activity.Current?.SetTag("agent.tool_result", $"get_service_status({service}) => {result}");
        return result;
    }
    [Description("Searches the on-call runbook and returns matching entries.")]
    public static string SearchRunbook([Description("Free text query")] string query)
    {
        var hits = Runbook.Where(r => query.Split(' ', StringSplitOptions.RemoveEmptyEntries)
                                           .Any(w => r.Contains(w, StringComparison.OrdinalIgnoreCase)))
                          .ToList();
        var result = hits.Count == 0 ? "No runbook entries found." : string.Join("\n", hits);
        Activity.Current?.SetTag("agent.tool_result", $"search_runbook({query}) => {result}");
        return result;
    }
}

The agent.tool_result tag matters later. I added it after the judge incident, so the judge could see the evidence the agent saw. Without it, the stored trace only has the final answer, and a judge reading only the final answer is grading prose, not correctness.

EvalAgent/Program.cs:

using System.Diagnostics;
using EvalShared;
using Microsoft.Extensions.AI;
using OllamaSharp;
using OpenTelemetry;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;

const string SourceName = "EvalAgent";
var contractPath = args.ElementAtOrDefault(0) ?? "../contracts/incident-agent.v3.json";
var repeat = int.TryParse(args.ElementAtOrDefault(1), out var r) ? r : 3;
var contract = AgentContract.Load(contractPath);
var captured = new List<Activity>();
using var tracerProvider = Sdk.CreateTracerProviderBuilder()
    .SetResourceBuilder(ResourceBuilder.CreateDefault().AddService(contract.AgentId))
    .AddSource(SourceName)
    .AddInMemoryExporter(captured) // layer 1 checks read these
    .AddOtlpExporter() // localhost:4317, ends up in traces/traces.jsonl
    .Build();
var source = new ActivitySource(SourceName);
IChatClient chat = new ChatClientBuilder(
        new OllamaApiClient(new Uri("http://localhost:11434"), contract.Model))
    .UseFunctionInvocation(configure: f => f.MaximumIterationsPerRequest = 6)
    .UseOpenTelemetry(sourceName: SourceName)
    .Build();
var tools = new List<AITool>
{
    AIFunctionFactory.Create(IncidentTools.GetServiceStatus, "get_service_status"),
    AIFunctionFactory.Create(IncidentTools.SearchRunbook, "search_runbook")
};
var failures = 0;
foreach (var evalCase in contract.Cases)
for (var trial = 1; trial <= repeat; trial++)
{
    captured.Clear();
    using (var root = source.StartActivity($"invoke_agent {contract.AgentId}"))
    {
        root?.SetTag("gen_ai.operation.name", "invoke_agent");
        root?.SetTag("agent.id", contract.AgentId);
        root?.SetTag("agent.prompt_version", contract.Version);
        root?.SetTag("agent.case_id", evalCase.Id);
        root?.SetTag("agent.input", evalCase.Input);
        var response = await chat.GetResponseAsync(
            [
                new ChatMessage(ChatRole.System, contract.SystemPrompt),
                new ChatMessage(ChatRole.User, evalCase.Input)
            ],
            new ChatOptions { Tools = tools, Temperature = 0.2f });
        root?.SetTag("agent.final_result", response.Text);
    }
    var trace = TraceRecord.Build(captured.Select(TraceRecord.FromActivity));
    if (trace is null) { Console.WriteLine($"[{evalCase.Id} #{trial}] no root span captured"); failures++; continue; }
    var results = DeterministicChecks.Run(trace, evalCase.Checks);
    var ok = results.All(x => x.Passed);
    if (!ok) failures++;
    Console.WriteLine($"[{(ok ? "PASS" : "FAIL")}] {evalCase.Id} trial {trial} trace={trace.TraceId} " +
                      $"model_requests={trace.ModelRequests} tools=[{string.Join(",", trace.ToolCalls)}]");
    foreach (var x in results.Where(x => !x.Passed))
        Console.WriteLine($" x {x.Name} {x.Detail}");
}
tracerProvider.ForceFlush();
Console.WriteLine(failures == 0 ? "All live trials passed." : $"{failures} live trial(s) failed.");
return failures == 0 ? 0 : 1;

Run it:

dotnet run -- ../contracts/incident-agent.v3.json 3

Two details that cost me time:

• Builder order. UseFunctionInvocation() goes before UseOpenTelemetry(). The builder wraps in order, so function invocation is the outer layer and every model round trip inside it gets its own chat span. The function-invoking client also emits execute_tool spans for each tool call. The exact span names and gen_ai.* attributes follow the OpenTelemetry GenAI semantic conventions, which are still marked experimental and have shifted between MEAI releases. That is why TraceRecord.Build checks both gen_ai.operation.name and the span name prefix. If your counts come back as zero, open traces.jsonl and look at what your version actually emits before you trust any check.
• I set my own root span. I don’t rely on MEAI to record message content, because sensitive-data capture is off by default (correctly) and the attribute layout has changed over time. I put exactly what replay needs on a root span I own: agent id, prompt version, case id, input and final answer. Everything replay depends on is under my control.

The exit code makes this usable as a CI gate, same as in Punnen’s setup. On my machine, three trials of three cases took [[MY RUN: wall-clock time for 9 live trials]] with qwen3:8b. That number is the whole argument for layer 2.

Layer 2: replaying stored traces with zero model calls

The replay app reads traces.jsonl (or any golden file), groups spans by trace id, rebuilds the same TraceRecord, looks up the contract that matches the trace's agent id and prompt version, and runs the exact same DeterministicChecks. No Ollama call happens unless you ask for judge sampling.

dotnet new console -n EvalReplay
cd EvalReplay
dotnet add package Microsoft.Extensions.AI
dotnet add package OllamaSharp

Add the same item group to EvalReplay.csproj.

EvalReplay/Program.cs:

using EvalShared;
using Microsoft.Extensions.AI;
using OllamaSharp;
string Arg(string name, string fallback)
{
    var i = Array.IndexOf(args, name);
    return i >= 0 && i + 1 < args.Length ? args[i + 1] : fallback;
}
var tracesPath = Arg("--traces", "../traces/traces.jsonl");
var contractsDir = Arg("--contracts", "../contracts");
var againstCurrent = args.Contains("--against-current");
var judgePercent = int.Parse(Arg("--judge-sample", "0"));
var seed = int.Parse(Arg("--seed", "42"));
var contracts = AgentContract.LoadAll(contractsDir);
var latest = contracts.Values.GroupBy(c => c.AgentId)
                      .ToDictionary(g => g.Key, g => g.MaxBy(c => c.Version)!);
var traces = TraceRecord.ReadOtlpJsonl(tracesPath)
    .GroupBy(s => s.TraceId)
    .Select(g => TraceRecord.Build(g))
    .Where(t => t is not null).Cast<TraceRecord>()
    .ToList();
Console.WriteLine($"Loaded {traces.Count} agent traces from {tracesPath}");
var failures = 0;
var checkedTraces = new List<(TraceRecord Trace, AgentContract Contract)>();
foreach (var t in traces)
{
    AgentContract? contract = againstCurrent
        ? latest.GetValueOrDefault(t.AgentId)
        : contracts.GetValueOrDefault((t.AgentId, t.PromptVersion));
    var evalCase = contract?.Cases.FirstOrDefault(c => c.Id == t.CaseId);
    if (contract is null || evalCase is null)
    {
        Console.WriteLine($"[SKIP] {t.TraceId} no contract for {t.AgentId} v{t.PromptVersion} case '{t.CaseId}'");
        continue;
    }
    var results = DeterministicChecks.Run(t, evalCase.Checks);
    var ok = results.All(r => r.Passed);
    if (!ok) failures++;
    checkedTraces.Add((t, contract));
    Console.WriteLine($"[{(ok ? "PASS" : "FAIL")}] {t.TraceId[..12]} {t.CaseId,-18} v{t.PromptVersion} -> contract v{contract.Version}");
    foreach (var r in results.Where(r => !r.Passed))
        Console.WriteLine($" x {r.Name} {r.Detail}");
}
if (judgePercent > 0 && checkedTraces.Count > 0)
    await Judge.RunSampleAsync(checkedTraces, judgePercent, seed);
Console.WriteLine(failures == 0 ? "Deterministic replay: all passed." : $"Deterministic replay: {failures} failed.");
return failures == 0 ? 0 : 1;

Run it:

# every stored trace against the contract version that produced it
dotnet run -- --traces ../traces/traces.jsonl

# every stored trace against the NEWEST contract: "what would my new checks have caught?"
dotnet run -- --traces ../traces/traces.jsonl --against-current
# a committed golden file, in CI
dotnet run -- --traces ../golden/incident-agent.jsonl

On my laptop, replaying [[MY RUN: number of stored traces]] traces took [[MY RUN: replay wall-clock time]], and none of it touched a model.

What I got wrong about replay

When I first wired this up, I celebrated a green replay after changing the system prompt. Then I realized every trace in the file had been produced by the previous prompt. Of course it was green. I had re-graded old homework.

So here is how I use the two modes now:

• Matching version (default): a monitoring question. “Is real traffic still inside the contract it was produced under?” If the agent starts calling tools six times on a case that should take two, replay catches it without me paying to reproduce it.
• --against-current: a check-design question. When I add or tighten a check in v4, I replay history against v4 before shipping it. If a new check fails half of the traces I already hand-verified as good, the check is wrong, not the agent. This saved me from shipping at least one bad check, which I'll describe in the next section.

Testing a new prompt is still layer 1. There is no shortcut.

Layer 3: sampled LLM judge, and the day it lied to me

Deterministic checks are blind to meaning. “The answer mentions payments-api and the agent called two tools” says nothing about whether the advice was right. That is what the judge is for, and why I sample it instead of running it on everything: it is the only layer with a variable cost and a variable answer.

EvalReplay/Judge.cs:

using System.Text.Json;
using EvalShared;
using Microsoft.Extensions.AI;
using OllamaSharp;
public sealed record JudgeVerdict(string Verdict, string Reason);
public static class Judge
{
    public static async Task RunSampleAsync(
        List<(TraceRecord Trace, AgentContract Contract)> items, int percent, int seed)
    {
        var rng = new Random(seed);
        var take = Math.Max(1, (int)Math.Ceiling(items.Count * percent / 100.0));
        var sample = items.OrderBy(_ => rng.Next()).Take(take).ToList();
        Console.WriteLine($"\nJudging {sample.Count} of {items.Count} traces ({percent}% sample, seed {seed})");
        foreach (var (trace, contract) in sample)
        {
            IChatClient judge = new OllamaApiClient(new Uri("http://localhost:11434"), contract.JudgeModel);
            // One isolated call per dimension instead of one call grading everything.
            foreach (var dim in contract.JudgeDimensions)
            {
                var v = await AskAsync(judge, trace, dim);
                Console.WriteLine($"[JUDGE {v.Verdict,-7}] {trace.TraceId[..12]} {trace.CaseId,-18} {dim.Name}: {v.Reason}");
            }
        }
    }
    static async Task<JudgeVerdict> AskAsync(IChatClient judge, TraceRecord t, JudgeDimension dim)
    {
        var evidence = t.ToolResults.Count == 0 ? "(no tool results recorded)" : string.Join("\n", t.ToolResults);
        var prompt = $"""
            You are grading one answer from an on-call assistant. Answer only the question below.
            USER QUESTION:
            {t.Input}
            TOOL RESULTS THE ASSISTANT RECEIVED (treat as ground truth):
            {evidence}
            ASSISTANT FINAL ANSWER:
            {t.FinalResult}
            QUESTION: {dim.Question}
            Rules:
            - Length and formatting are not quality. A short correct answer passes.
            - If the information above is not enough to decide, answer UNKNOWN.
            Reply with JSON only: {"verdict": "PASS" | "FAIL" | "UNKNOWN", "reason": "<one sentence>"}
            """;
        var response = await judge.GetResponseAsync(prompt,
            new ChatOptions { Temperature = 0f, ResponseFormat = ChatResponseFormat.Json });
        try
        {
            return JsonSerializer.Deserialize<JudgeVerdict>(response.Text,
                       new JsonSerializerOptions { PropertyNameCaseInsensitive = true })
                   ?? new JudgeVerdict("UNKNOWN", "empty judge response");
        }
        catch (JsonException)
        {
            return new JudgeVerdict("UNKNOWN", $"unparseable judge output: {response.Text}");
        }
    }
}

Run it on a 20 percent sample:

dotnet run -- --traces ../traces/traces.jsonl --judge-sample 20 --seed 7

Judge results print but do not change the exit code. That is a deliberate opinion: until I have calibrated a judge against my own labels, it is advisory. A gate built on an uncalibrated judge is a gate that randomly blocks deploys.

The mis-score

Punnen is candid in his article about his judge getting things wrong, which is part of why I trusted the rest of it. His specifics are his to tell (the article is member-only on Towards AI). Here is mine.

The code above is the version after the fix. Under the v3 rubric, the judge prompt only contained the user question and the final answer, and the single dimension was “how helpful and complete is this answer, 1 to 5, pass at 4 or higher.”

On the payments-latency case, one trial produced this (trimmed):

To resolve the payments-api latency:
1. Check the status page and dashboards for payments-api.
2. Restart the api pods to clear stuck connections.
3. Monitor p99 latency for 10 minutes after the restart.
4. If latency persists, escalate to payments-oncall.

The deterministic checks passed. It contains “payments-api”. It made two tool calls, inside the bounds. The judge gave it 5 out of 5 : clear, actionable, well structured.

It was the worst answer in the whole run. The status tool had returned that a database failover was in progress, and the runbook entry it retrieved says, in capitals, never restart during an active failover. The agent called both tools and then ignored them. Following that answer would have extended the outage.

Meanwhile, another trial on the same case answered roughly: “A database failover is in progress since 14:02 UTC. Per the runbook, do not restart the pods. Wait and post in #payments-incident.” The judge gave it a 3 and called it “too brief, lacks troubleshooting steps.” Failed.

So the judge passed the dangerous answer and failed the correct one. I went back and hand-labeled every payments-latency trace I had, and the v3 judge disagreed with me on [[MY RUN: x of y hand-labeled traces]]. The pattern was always the same: it rewarded length and structure.

What I changed in the rubric (v4)

Looking back, the judge was not broken. It was answering the question I asked, which was a vague one, with no access to the evidence. Here is what changed:

1. The judge sees the tool results. That is what the agent.tool_result tag is for. The judge prompt now treats them as ground truth. A judge cannot catch "ignored the evidence" if it never sees the evidence.
2. Binary questions, one per call. Instead of one 1 to 5 helpfulness score, v4 has separate yes/no dimensions, each graded in an isolated call. This follows Anthropic’s suggestion to grade each dimension with its own judge rather than one judge for everything.
3. An explicit statement that length is not quality, and a way out: UNKNOWN when there is not enough information, which their post also recommends to reduce made-up verdicts.
4. A new deterministic check, because the judge failure revealed something a string check could catch.

contracts/incident-agent.v4.json (only the parts that changed):

{
  "agentId": "incident-agent",
  "version": 4,
  "model": "qwen3:8b",
  "judgeModel": "llama3.1:8b",
  "systemPrompt": "/no_think You are an on-call assistant. Always check the current status of a service with get_service_status before giving advice, and use search_runbook for procedures. If a tool result contradicts a usual procedure, the tool result wins and you must say so. Be brief. If you do not know, say so.",
  "judgeDimensions": [
    {
      "name": "grounded",
      "question": "Is every factual claim about the service's current state supported by the tool results? PASS or FAIL."
    },
    {
      "name": "safe",
      "question": "Does the answer avoid recommending any action that the tool results or runbook say must not be taken right now? PASS or FAIL."
    },
    {
      "name": "actionable",
      "question": "Does the answer tell the engineer what to do or not do next, even if the right action is to wait? PASS or FAIL."
    }
  ],
  "cases": [
    {
      "id": "payments-latency",
      "input": "payments-api latency alert just fired. What should I do?",
      "checks": { "contains": ["payments-api", "failover"], "maxModelRequests": 4, "minToolCalls": 2, "maxToolCalls": 4 }
    }
  ]
}

(The other two cases stay as in v3.)

The deterministic change is small: on this case, a correct answer must mention the failover and must have called both tools. My first attempt was a “must not contain ‘restart’” check. I replayed history with --against-current before shipping it, and it failed the correct answer, because "do not restart the pods" contains "restart". That is exactly the kind of check you want to discover is wrong against old traces, for free, rather than in CI on a Friday. I dropped it and used contains: failover instead.

After the change, the v4 judge on the same hand-labeled traces disagreed with me on [[MY RUN: x of y]]. Better, not perfect. I still read every judge FAIL by hand, and I still read a few judge PASSes each week, because a judge that only ever agrees with itself is not being checked by anyone.

Who is the expert here?

The judge story has an uncomfortable second half. When I hand-labeled traces, I was acting as the human grader, the gold standard the judge is supposed to be calibrated against. But the “never restart during a failover” rule was not something I knew. I learned it from the runbook I had written for the demo, which I had written after asking someone who has actually run payments on-call.

The DoorDash Design Team’s Building AI Evals series puts a name on this. Their second part argues that deciding whose judgment counts is part of eval design, not a step that comes before it. They split evaluation questions into fit (ask the individual the result was made for), authority (ask people with real standing in the domain) and boundaries (ask a broad community what the AI should do at all). In their own write-up about “best restaurant” search, they describe finding that citywide questions produced noise while neighborhood-level local experts converged, and that how much the experts agreed with each other told them what kind of product to build. Those are their findings about their system. What I took from it is the framing, not the numbers.

Mapped onto my small agent:

• “Is this answer safe during an incident?” is an authority question. The expert is the on-call engineer who has lived through a failover, not me, and certainly not a general model.
• “Is this answer the right length for me at 3am?” is a fit question. It depends on the person reading it.
• “Should the agent ever suggest a restart on its own?” is a boundaries question, and it belongs to the team, not to whoever writes the rubric.

My v3 rubric quietly collapsed all three into “helpfulness” and handed it to a model that was, at best, imitating me. That is why it failed. The fix was not only a better prompt. It was asking someone with authority to write the safe dimension, and treating their disagreement with my labels as a signal rather than noise.

How this connects to “evals are capital”

A while ago I wrote that prompts are consumables and evals are capital. Prompts encode how one model version reacts to particular phrasing, so they decay with every upgrade. Evals encode what correct behavior looks like in your domain, so they compound.

Stored traces sharpen that idea. A trace is a free, permanent record of a behavior you already paid for. Every time I add a check, I can run it against every trace I have ever kept. Every time I add a judge dimension, I can sample it over months of history. The golden file in my repo grows by a few lines a week, mostly from failures, and it never costs a model call to re-check.

There is a related idea in Fareed Khan’s long write-up on loop engineering techniques: his maker-checker pattern, where a second agent verifies work by running its own tests rather than trusting the first agent’s opinion. I only read part of it (it is member-only too), and I am not claiming his results. But the instinct is the same one that made me demote my judge to advisory: prefer a check that runs over an opinion that sounds confident.

What is still unknown

I want to be honest about the parts I have not figured out:

• Judge calibration is still small-n. My hand-labeled set is tiny. I know the v4 judge is better on this case. I do not know how it does on cases I haven’t thought of.
• Semantic conventions are moving. If MEAI changes how it names tool spans again, my tool counts silently drop to zero and minToolCalls fails everything. I added a canary trace to the golden file that must always count exactly two tools, so a parsing change fails loudly. It is a workaround, not a fix.
• A local 8B judge is not a frontier judge. I chose local on purpose, for cost and so this article is fully runnable. A stronger judge might not have made the length mistake. It might make subtler ones that are harder for me to catch.
• Trace files grow. The collector’s file exporter supports rotation, which I have not set up properly yet. For now I prune by hand and keep the golden file separate.
• Sampling rate is a guess. 20 percent is a number I picked, not one I derived. I don’t yet know the smallest sample that would have caught the payments mis-score reliably.

The short version

If you take one thing from this: write the contract first, record your own root span with everything you’ll want to check later, and keep the traces. Run live checks when you change the prompt or the model. Replay deterministic checks over stored traces for everything else, because it costs nothing. Sample a judge on top, show it the evidence, ask it narrow yes/no questions, and do not let it gate a deploy until you’ve caught it being wrong at least once. You will.

Sources

• Alex Punnen, Eval-Driven Development: A Software Engineering Approach to Production-Grade AI Agents, Towards AI: https://pub.towardsai.net/eval-driven-development-a-software-engineering-approach-to-production-grade-ai-agents-4a86f3fd2d9a
• Alex Punnen, companion repo: https://github.com/alexcpn/eval-driven-development
• Anthropic, Demystifying evals for AI agents: https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents
• DoorDash Design Team, Building AI Evals: Who Are The “Experts”?: https://medium.com/@designdoordash/building-ai-evals-who-are-the-experts-f1dd8b0ffa29
• Fareed Khan, Testing 17 Agentic Loop Engineering Techniques for Reliable AI Agents, Level Up Coding: https://levelup.gitconnected.com/testing-17-agentic-loop-engineering-techniques-for-reliable-ai-agents-7966adad5835

Tags: AI Agents, LLM Evaluation, OpenTelemetry, Dotnet, LLM Engineering, Ollama, Software Testing, Observability

---