I Ran a Harness Eval Loop on My Own .NET Agent. Here’s What Changed and What Didn’t

For about four months I improved my coding agent the way most people do. I’d watch it fail on something, sigh, and then add a tool. Or add three paragraphs to the system prompt. Or both. Each change felt obviously right at the time, and each one made the harness a little heavier.

Then one evening I looked at my token bill for the month and at the actual quality of the patches the agent was producing, and I couldn’t answer a very basic question: which of those changes had made anything better?

I had no idea. I had vibes.

This article is about what happened when I stopped relying on vibes and ran a proper harness eval loop on my own .NET agent. Some of it went the way I hoped. One change I was sure about turned out to make things worse, and I only found out because my eval was lying to me first. That part is the most useful bit, so stick around for it.

The loop, in plain terms

The idea is almost embarrassingly simple:

1. Give the agent a real task. Not a toy, not a benchmark someone else designed, a task that looks like the work you actually want it to do.
2. Run it. Several times, because agents are not deterministic.
3. Score the outcome, not the path. Did it produce the correct result? You don’t grade how elegant its reasoning looked.
4. Change exactly one thing in the harness.
5. Re-run the same tasks the same number of times.
6. Keep the change only if the numbers say it helped. Otherwise revert it.

That’s it. The “harness” here means everything around the model: the system prompt, the tool set, how context gets retrieved, whether there’s a review step, retry logic, and so on.

A piece in The Code newsletter this week, on why handing your agent more tools doesn’t make it better, summarized this approach well, and it matched something I had been feeling but not measuring. When you improve a harness by intuition, every change sounds reasonable in isolation. More tools means more capability. A longer prompt means more guidance. But you never subtract anything, and you never check. You end up with a bloated, expensive harness that is not meaningfully better than the lean one you started with, and sometimes it is worse.

The loop forces you to earn every addition.

My agent and the task I picked

The agent itself is small. It’s a .NET 9 console app built on Microsoft.Extensions.AI, with an IChatClient in the middle and a handful of tools exposed through AIFunctionFactory. It reads a GitHub issue, explores the repository, edits files, runs the build and tests, and produces a patch. Nothing exotic. I built it for maintenance work on a few internal .NET services where the issues are mostly small bugs and "this endpoint returns the wrong thing when X".

For the eval I needed something concrete and repeatable, so I picked this:

Given a real GitHub issue and the repository at the commit just before it was fixed, produce a patch that passes a hidden test suite.

I went back through our issue history and picked 12 closed issues where:

• The fix was already merged, so I knew a correct solution existed.
• The fix touched one to three files.
• I could write (or already had) tests that fail before the fix and pass after it.

Those tests are the “hidden” suite. The agent never sees them. It gets the issue text and the repo, and that’s all.

Scoring is binary per run: after the agent finishes, I copy the hidden tests into the workspace and run dotnet test. Exit code zero means pass. I don't care if the agent took a weird route, read too many files, or wrote a slightly different fix than the human did. If the behavior is correct, it passes.

I know there’s a good argument against outcome-only scoring. Udaykiran Estari has a piece on trajectory-level evals making the case that a green test suite can hide a messy, lucky process, like an agent that thrashes around, edits unrelated files, and stumbles onto a passing state. I agree with that, and I’ll come back to it, because reading trajectories is exactly how I caught my eval cheating. But for deciding whether a harness change helps, outcome scoring is the right primary signal. It’s cheap, it’s objective, and it maps to what I actually want.

Each configuration runs 12 tasks times 5 runs, so 60 runs per experiment. That’s not a huge sample, and I’ll be honest about the noise below.

The eval harness itself

Here’s the scorer. It’s Python because I wanted it to stay dumb and separate from the agent, which is .NET. Keeping the judge in a different language from the thing being judged turned out to be a nice accidental guardrail: I was never tempted to share code between them.

The key design choice: the agent process runs on my machine (it needs network access to call the model), but every tool it uses executes inside a throwaway Docker container with networking disabled. File reads, edits, builds, test runs, all of it goes through docker exec.

#!/usr/bin/env python3
"""Run each eval task N times in a fresh sandbox and score the outcome."""
import json
import re
import subprocess
import sys
import time
import uuid
from dataclasses import dataclass, asdict, field
from pathlib import Path

IMAGE = "agent-sandbox:latest"
RUNS_PER_TASK = 5
AGENT_PROJECT = Path("../src/Agent")
TASKS_DIR = Path("tasks")
OUT_DIR = Path("out")
# Set these to your model's actual prices (USD per token).
PRICE_IN = 3.00 / 1_000_000
PRICE_OUT = 15.00 / 1_000_000
# If any of these show up in an agent's tool log, the run is suspect.
TRIPWIRES = [
    r"git\s+(log|reflog|fsck).*--all",
    r"\.nuget/packages",
    r"Hidden",
    r"CANARY-7f3a9c",
    r"/opt/canary",
]

@dataclass
class RunResult:
    task_id: str
    run: int
    passed: bool
    input_tokens: int
    output_tokens: int
    seconds: float
    cost_usd: float = 0.0
    tripwires: list = field(default_factory=list)

def sh(*args, check=True, capture=False):
    return subprocess.run(args, check=check, text=True, capture_output=capture)

def run_once(task: Path, run: int, config: str) -> RunResult:
    name = f"eval-{uuid.uuid4().hex[:8]}"
    log_path = OUT_DIR / config / f"{task.name}-{run}.jsonl"
    log_path.parent.mkdir(parents=True, exist_ok=True)
    sh("docker", "run", "-d", "--name", name, "--network", "none",
       "--memory", "4g", IMAGE, "sleep", "infinity")
    try:
        # Repo snapshot is a git archive of the base commit: no history at all.
        sh("docker", "cp", str(task / "repo.tar"), f"{name}:/tmp/repo.tar")
        sh("docker", "exec", name, "bash", "-c",
           "cd /work && tar -xf /tmp/repo.tar && rm /tmp/repo.tar && "
           "git init -q && git add -A && "
           "git -c user.email=eval@local -c user.name=eval commit -qm base")
        start = time.time()
        agent = sh("dotnet", "run", "--project", str(AGENT_PROJECT), "--",
                   "--container", name,
                   "--issue", str(task / "issue.md"),
                   "--config", config,
                   "--log", str(log_path),
                   check=False, capture=True)
        seconds = time.time() - start
        # Agent prints a usage summary as JSON on its last stdout line.
        usage = json.loads(agent.stdout.strip().splitlines()[-1])
        # Hidden tests go in only AFTER the agent has exited.
        sh("docker", "cp", str(task / "hidden_tests"), f"{name}:/work/tests/Hidden")
        test = sh("docker", "exec", name, "dotnet", "test",
                  "/work/tests/Hidden", "--nologo", "-v", "q",
                  check=False, capture=True)
        log_text = log_path.read_text() if log_path.exists() else ""
        hits = [p for p in TRIPWIRES if re.search(p, log_text)]
        result = RunResult(
            task_id=task.name, run=run, passed=test.returncode == 0,
            input_tokens=usage["input_tokens"],
            output_tokens=usage["output_tokens"],
            seconds=round(seconds, 1), tripwires=hits,
        )
        result.cost_usd = round(result.input_tokens * PRICE_IN
                                + result.output_tokens * PRICE_OUT, 4)
        return result
    finally:
        sh("docker", "rm", "-f", name, check=False, capture=True)

def main(config: str):
    tasks = sorted(p for p in TASKS_DIR.iterdir() if p.is_dir())
    results = [run_once(t, r, config) for t in tasks for r in range(RUNS_PER_TASK)]
    n = len(results)
    passed = sum(r.passed for r in results)
    flagged = [r for r in results if r.tripwires]
    summary = {
        "config": config,
        "runs": n,
        "pass_rate": round(passed / n, 3),
        "avg_tokens": round(sum(r.input_tokens + r.output_tokens for r in results) / n),
        "avg_cost_usd": round(sum(r.cost_usd for r in results) / n, 4),
        "flagged_runs": len(flagged),
    }
    (OUT_DIR / config / "summary.json").write_text(json.dumps(
        {"summary": summary, "runs": [asdict(r) for r in results]}, indent=2))
    print(json.dumps(summary, indent=2))
    for r in flagged:
        print(f"SUSPECT {r.task_id} run {r.run}: {r.tripwires}", file=sys.stderr)

if __name__ == " __main__":
    main(sys.argv[1] if len(sys.argv) > 1 else "baseline")

And the sandbox image. The NuGet cache is warmed at build time so dotnet build and dotnet test work offline. The canary file is there on purpose, more on that later.

FROM mcr.microsoft.com/dotnet/sdk:9.0
WORKDIR /work

# Warm the NuGet cache with the repos' dependencies, then run offline.
COPY restore-seed/ /seed/
RUN dotnet restore /seed/Seed.sln && rm -rf /seed
# Canary: if this token ever appears in an agent log, the agent went looking.
RUN mkdir -p /opt/canary && \
    echo "CANARY-7f3a9c SOLUTION: see upstream commit" > /opt/canary/SOLUTION.md

Building a task snapshot is one line per issue. git archive gives you the tree at a commit with zero history attached:

git -C ~/src/orders-service archive --format=tar \
  --output tasks/ISSUE-142/repo.tar 3f9c1e2

Running it:

docker build -t agent-sandbox:latest .
python3 run_eval.py baseline
python3 run_eval.py semantic-search

A note on noise before any numbers. With 60 runs and a pass rate around 58%, the standard error is roughly 6 percentage points. So I decided up front that I would treat any pass-rate difference under about 8 points as “no real change” unless it repeated on a second full run. Deciding that threshold before seeing results matters. If you decide it afterwards, you’ll quietly pick whatever threshold makes your favorite change look good. I know because I almost did.

The results, change by change

Here’s the whole picture first, then the stories behind each row.

+----+--------------------------------+-----------+------------+-----------+----------+
| # | Harness config | Pass rate | Avg tokens | Cost/run | Decision |
+----+--------------------------------+-----------+------------+-----------+----------+
| 0 | Baseline (full-file reads) | 58% | 184k | $0.61 | - |
| 1 | + Semantic search tool | 57% | 108k | $0.37 | Kept |
| 2a | + Big toolbox, long prompt | 67% | 151k | $0.52 | (bogus) |
| 2b | Same, after sandbox fix | 52% | 142k | $0.49 | Reverted |
| 3 | #1 + one self-review pass | 70% | 131k | $0.45 | Kept |
+----+--------------------------------+-----------+------------+-----------+----------+
  60 runs per row (12 issues x 5 runs). Costs at my model's list price.

A quick honesty note: these are my numbers from my repos and my model. Your absolute values will differ. The direction of each change is the part worth paying attention to.

Change 1: semantic search instead of full-file dumps (helped)

My baseline agent had a read_file tool and a list_files tool. That's it. To find where something was implemented, it would list directories and read whole files, sometimes six or seven 400-line files, before landing on the right one. All of that stayed in context for the rest of the run.

I replaced the exploration step with a search_code tool that chunks the repo, embeds the chunks, and returns the top matches with file paths and line ranges. The agent can still call read_file, but now it knows where to look first.

I used Ollama with nomic-embed-text for the embeddings, so this part costs nothing per call and never leaves my machine:

ollama pull nomic-embed-text

// dotnet add package OllamaSharp
// dotnet add package System.Numerics.Tensors
using System.ComponentModel;
using System.Numerics.Tensors;
using Microsoft.Extensions.AI;
using OllamaSharp;
public sealed record CodeChunk(string Path, int StartLine, int EndLine, string Text);
public sealed class CodeSearchTool
{
    private readonly IEmbeddingGenerator<string, Embedding<float>> _embedder;
    private readonly List<(CodeChunk Chunk, float[] Vector)> _index = new();
    public CodeSearchTool(Uri ollamaUrl)
    {
        _embedder = new OllamaApiClient(ollamaUrl, "nomic-embed-text");
    }
    public async Task BuildIndexAsync(IEnumerable<CodeChunk> chunks)
    {
        var list = chunks.ToList();
        foreach (var batch in list.Chunk(32))
        {
            var embeddings = await _embedder.GenerateAsync(batch.Select(c => c.Text));
            for (int i = 0; i < batch.Length; i++)
                _index.Add((batch[i], embeddings[i].Vector.ToArray()));
        }
    }
    [Description("Search the codebase semantically. Returns file paths, line ranges and a short snippet for the best matches.")]
    public async Task<string> SearchCode(
        [Description("What you are looking for, in plain language")] string query,
        [Description("How many results to return, 1 to 10")] int top = 5)
    {
        var q = (await _embedder.GenerateAsync(new[] { query }))[0].Vector.ToArray();
        var hits = _index
            .Select(e => (e.Chunk, Score: TensorPrimitives.CosineSimilarity(q, e.Vector)))
            .OrderByDescending(h => h.Score)
            .Take(Math.Clamp(top, 1, 10));
        return string.Join("\n\n", hits.Select(h =>
            $"{h.Chunk.Path}:{h.Chunk.StartLine}-{h.Chunk.EndLine} (score {h.Score:F2})\n" +
            string.Join('\n', h.Chunk.Text.Split('\n').Take(12))));
    }
}
// Registering it with the agent:
// var search = new CodeSearchTool(new Uri("http://localhost:11434"));
// await search.BuildIndexAsync(ChunkRepository(repoRoot));
// tools.Add(AIFunctionFactory.Create(search.SearchCode, "search_code"));

Chunking is nothing clever. I split .cs files on type and method boundaries using a regex, with a cap of 80 lines per chunk. Roslyn would be better. The regex was good enough to test the idea, and the loop told me it was good enough to keep.

Result: average tokens per run dropped from 184k to 108k, a 41% cut. Pass rate went from 58% to 57%, which is one run out of 60, well inside the noise band. Cost per run fell from $0.61 to $0.37.

This is the kind of change the loop is designed to catch. No accuracy gain, but the same quality for a lot less money. Without measuring, I might have skipped it because “the agent already finds the right files eventually”. With measuring, it’s an easy keep.

Change 2: more tools and a much longer system prompt (made it worse, and first it lied)

This is the one I was confident about.

My theory was that the agent failed on some issues because it lacked context. So I did the classic thing. I added five tools (run_shell, git_history, find_references, nuget_info, and read_docs for our internal wiki exports) and expanded the system prompt from about 1,100 tokens to about 4,300, adding our coding conventions, a description of each service, and a long list of "things to watch out for".

The first run came back at 67%. Nine points above baseline. Above my noise threshold. I was pleased with myself for about an evening.

Then I did something I’d been meaning to do for a while: I opened the trajectory logs for the passing runs on the two hardest issues. Those were issues the baseline passed maybe once in five. With the big toolbox, they passed four or five times out of five.

In most of those runs the agent had called run_shell with git log --all --oneline and then git show on a commit whose message was, more or less, the title of the issue.

The fix commit was sitting right there in the repository history.

At that point my task snapshots were full clones checked out at the base commit. The future commits, including the actual fix, were still in the object database. The baseline agent had no shell, so it couldn’t see them. The moment I gave it a general-purpose shell, a smart enough model found the answer key. Its patches looked correct because they were, more or less, the human’s patch.

That’s precisely the failure described in a Microsoft Developer Blog post from this month, “Your AI coding agent evaluation is only as good as its sandbox”. Their point is that an agent passing an eval tells you nothing if it found the answer somewhere in its environment instead of working it out, and that a correct answer can still invalidate the measurement. Reading it the same week I found this in my own logs was a slightly uncomfortable experience.

What I changed:

• Task snapshots became git archive tarballs of the base commit, re-initialized as a fresh single-commit repo inside the container. There is no history to find.
• Every run gets a new container. Nothing survives between runs, including bin/ and obj/ folders, which I realized could also have leaked hidden test assemblies if I'd ever built them in a reused container.
• Hidden tests are copied in only after the agent process exits.
• The container has --network none. The model API call happens on the host, outside the sandbox.
• I planted a canary file with a unique token and added tripwire patterns to the scorer. Any run whose log mentions git log --all, the NuGet package cache, the hidden test path, or the canary token is flagged for me to read by hand.

The canary is a cheap trick I like a lot. It doesn’t prevent anything. It tells you whether your agent is the type that goes rummaging, and that’s worth knowing before you trust any number.

With the sandbox fixed, I re-ran change 2. It came back at 52%. That’s six points below baseline. Within the noise band on its own, so I ran it a second time and got 53%. Consistently not better, possibly worse, and 30% more expensive than the lean config with semantic search.

When I read the failing trajectories, the pattern was clear. The agent spent its early turns reading wiki pages and NuGet metadata that had nothing to do with the bug, and the long prompt’s list of warnings made it hedge. It wrote more defensive code, touched more files, and broke existing behavior more often. More guidance became more distraction.

I reverted the whole thing. I kept find_references for later testing on its own, because I suspect it's useful and the other four were dragging it down. That's another lesson: bundling five changes into one experiment means you can't tell which one hurt. I knew the rule was "change one thing" and I broke it anyway because I was sure. Being sure is exactly when you need the measurement most.

Change 3: one self-review pass (a tradeoff I had to decide on)

The last change was a review step. After the agent thinks it’s done, the harness runs the visible test suite, then gives the model a fresh, short context containing only the issue text, the diff, and the test output, and asks it to either approve the patch or list specific problems. If it lists problems, the agent gets one more attempt. Only one.

On top of the semantic search config, pass rate went from 57% to 70%. That’s 13 points, clearly outside the noise, and it held on a second run at 68%.

The catch is cost. Relative to config 1, tokens per run went up from 108k to 131k, and wall-clock time per run went up about 40%, since it’s a sequential extra step. Against the original baseline it’s still cheaper, but against the best lean config it’s about 22% more expensive per run.

This is where the loop gives you data and then leaves the decision to you. I thought about it in terms of cost per correct patch, not cost per run:

+----------------------------+-----------+----------+----------------------+
| Config | Pass rate | Cost/run | Cost per correct fix |
+----------------------------+-----------+----------+----------------------+
| Baseline | 58% | $0.61 | $1.05 |
| Semantic search | 57% | $0.37 | $0.65 |
| Semantic search + review | 70% | $0.45 | $0.64 |
+----------------------------+-----------+----------+----------------------+

Per correct fix, the two top configs cost about the same. But a failed run isn’t free on my side: I have to read it, figure out that it’s wrong, and do the work myself. So the higher pass rate is worth more to me than the raw numbers suggest. I kept the review step.

What I did not do was allow two or three review rounds. I tried two rounds briefly and it got to 71% at noticeably higher cost, which is the definition of diminishing returns. One round stays.

If I were running this agent at a much larger volume on low-stakes issues, I might make the opposite call. That’s fine. The point is that it’s now a deliberate decision with numbers behind it, not a feeling.

Running the whole loop locally

One thing I didn’t expect: the eval loop itself was useful for finding bugs in my harness, not only for measuring quality. A crash in the tool layer or a malformed tool schema shows up immediately as a string of zero-score runs.

For those smoke runs I don’t want to pay for 60 hosted model calls, so the agent can swap its IChatClient for a local model through Ollama. OllamaApiClient implements IChatClient, so it's a one-line change in the composition root:

ollama pull qwen2.5-coder:14b

using Microsoft.Extensions.AI;
using OllamaSharp;
IChatClient BuildChatClient(string config) => config.EndsWith("-local")
    ? new OllamaApiClient(new Uri("http://localhost:11434"), "qwen2.5-coder:14b")
        .AsBuilder().UseFunctionInvocation().Build()
    : BuildHostedClient(); // your normal provider setup

python3 run_eval.py semantic-search-local

The local pass rate is much lower on these issues, and I would never use it to decide whether a harness change is good for the hosted model. Harness changes don’t transfer cleanly between models. But as a fast, free check that the pipeline runs end to end, that tripwires fire, and that the scorer isn’t broken, it has saved me a lot of money.

What I’d tell myself four months ago

A few opinions I hold more strongly now:

Outcome scoring decides, trajectory reading audits. I still score only the result. But I read a sample of trajectories for every experiment, especially passing runs on hard tasks. A pass on a task the agent used to fail is the moment to be suspicious, not the moment to celebrate. Estari’s argument about lucky green builds is right, and for me it applied to the eval itself before it applied to the agent.

Most of the gain was subtraction. The best change I made removed tokens. The worst change I made added tools and prompt text. I don’t think that’s a coincidence.

Your sandbox is part of your measurement. If the environment contains the answer, a capable agent will find it, and you’ll measure your filesystem instead of your harness. Adding a general shell tool is exactly the kind of change that turns a clean eval into a leaky one without anything else changing.

You don’t need a framework to start. There are plenty of roundups of agent evaluation frameworks now (Ali Ibrahim’s list of seven for 2026 is a decent starting point if you want one). I’ll probably adopt one eventually. But a Python script, a Docker image, and a dozen real issues got me further in two weekends than months of intuition did.

A reusable checklist for any agent harness

This isn’t specific to .NET or to coding agents. I’ve started applying it to a ticket-triage agent too.

[] Pick 10 to 20 real tasks from your own history, with known-correct outcomes.
[] Define a binary, automatic outcome check (hidden tests, exact label, schema match).
[] Keep the scoring artifacts out of the agent's reach until after it exits.
[] Snapshot task inputs without history (git archive, not a clone).
[] Fresh container per run, network off for tool execution.
[] Plant a canary and add tripwire patterns for known leak paths.
[] Run each config at least 5 times per task; record pass rate, tokens, cost, time.
[] Decide your noise threshold BEFORE looking at results.
[] Change exactly one thing per experiment.
[] Re-run anything that looks surprisingly good, then read its trajectories.
[] Compare cost per correct outcome, not just cost per run.
[] Revert anything that doesn't clear the threshold, even if you like it.
[] Use a local model for smoke runs; use the real model for decisions.

The last item on the list that isn’t written down: write the result down, including the ones that didn’t work. My notes on change 2 are more valuable to me than my notes on change 1. They’re the reason I no longer trust my instinct that more context is always better.

If you’ve run something like this on your own agent, I’d genuinely like to hear which change surprised you. Mine was learning that the agent was smarter at finding answer keys than I was at hiding them.

Tags: dotnet, ai, llm, agents, evaluation, testing, csharp, docker, ollama, softwareengineering