I spent most of last year treating my CLAUDE.md file like it was source code. Every time an agent did something dumb, I patched the prompt. Every time I found a phrasing that made the model behave, I bolted it on. By month eight the file was 400 lines long, full of instructions like “never use raw SQL here” and “always check the pgvector index before writing,” and I was genuinely proud of it. It felt like an asset. It felt like something I was building.
It wasn’t. It was a lease, and I didn’t know the lease had an end date until the rent came due.
Here’s the thesis I want to argue in this piece, as plainly as I can state it: CLAUDE.md files, system prompts, and skill libraries are consumable. They decay with every model version, sometimes visibly and sometimes in ways that produce no error at all, just a quieter kind of wrong. Test suites and evals are the opposite. They are durable capital. They compound. And most of the teams I talk to, including the version of me from a year ago, invest their engineering time exactly backwards, polishing the thing that expires and treating the thing that compounds as an afterthought.
The Skill That Stopped Working and Nobody Noticed
Here’s what actually happened to me. Featpilot, the multi-repo ticket generator I’ve been building, uses Claude to turn a plain-language feature description into structured tickets with acceptance criteria and per-repo sizing. Getting the tone and structure right took a lot of prompt tuning: how much context to pull from each repo’s README, how strict to be about the “Why / What / Acceptance Criteria” format, how to stop the model from inventing acceptance criteria that sounded plausible but didn’t map to anything in the codebase.
I got it to a place I was happy with. Tickets came out clean. Sizing was reasonable. I stopped touching the prompt and moved on to other parts of the system.
Then, at some point after a model version bump, the tickets got worse. Not broken. Worse. The acceptance criteria started drifting toward generic boilerplate. Sizing got a little more optimistic than it should have been. Nothing threw an exception. Nothing failed a build. The FastAPI endpoint returned 200 every single time, because from the API’s point of view, nothing was wrong. A string came back. The string just wasn’t as good.
It took me close to three weeks to notice, and I only noticed because I happened to be re-reading old generated tickets side by side with new ones while writing documentation. If I hadn’t done that side-by-side comparison out of sheer coincidence, I don’t know how long it would have kept degrading before I caught it. That’s the part that unsettled me. I had no alarm that could have told me. I had no test that encoded “this is what a good ticket looks like” in a way a machine could check. I had a prompt, and prompts don’t fail loudly. They fail by drifting.
I’m not the only one who’s run into this, and it’s not just anecdotal. A benchmark called SWE-Skills-Bench tested 49 publicly available agent skills across roughly 565 real task instances, and found that 39 of the 49 produced zero measurable pass-rate improvement. Zero. Not “modest,” not “hard to tell,” literally flat or worse. The average gain across all 49 skills was about 1.2 percent, and three of them actively made things worse by up to 10 percentage points because their guidance had gone stale relative to the project context they were being applied to. Only seven skills produced real gains, and the ones that did tended to be narrow and specific rather than broad and aspirational.
That number stopped me. I’d been assuming that a skill file, once it worked, kept working. The data says the opposite: most of the value people believe they’re getting from their carefully hand-tuned skills and prompt libraries is imaginary, or at best, temporary.
And it’s not only third-party skills rotting on the vine. Anthropic’s own postmortem on Claude Code quality regressions this spring is a case study in exactly this failure mode happening inside the company that makes the model. Between March and April, three separate changes each quietly degraded behavior that had previously worked fine: a default reasoning-effort change from “high” to “medium” that made the model “feel less intelligent” without any single obvious failure, a caching bug that cleared reasoning context on every turn instead of once per hour and made the agent appear forgetful, and a new system-prompt instruction capping responses at 25 words between tool calls, which testing later showed caused a measured 3 percent drop in performance for both Opus 4.6 and 4.7. None of these threw errors. All of them shipped, ran in production, and degraded real user workflows before anyone traced the cause. If the team that owns the model can lose weeks diagnosing a quiet regression in their own harness, the rest of us maintaining CLAUDE.md files on top of a model we don’t control should assume we’re even more exposed.
Why Prompts Decay and Tests Don’t
The asymmetry here isn’t an accident, it’s structural. A prompt encodes assumptions about how a specific model, at a specific version, responds to specific phrasing. “Think step by step before answering” mattered enormously for earlier reasoning models and is close to a no-op, or even mildly harmful, for models with native extended thinking. “Always double check your file paths” was compensating for a class of mistake a given model made constantly, right up until a training update made that mistake rare, at which point the instruction is just dead weight sitting in your context window, quietly taxing every single request.
A test doesn’t encode assumptions about the model. It encodes assumptions about correct behavior in your domain, which changes far more slowly. “GetShippingAddress should not throw when Order.Customer is null, it should return null” is true regardless of which model, which version, or which vendor is writing the fix. That fact was true before agents existed and it’ll be true after whatever comes after Opus and GPT and Gemini current-generation. The test is capital because its value doesn’t depend on the thing you’re testing staying the same. The prompt is a consumable because its entire value depends on the thing it’s steering staying exactly as unpredictable as it was when you wrote it.
Boris Cherny, who built Claude Code, has apparently told people internally to delete their CLAUDE.md every six months just to see what the current model can do without the accumulated scaffolding. Anthropic itself deleted roughly 80 percent of Claude Code’s own system prompt at one point and measured no regression. Read that twice. Eighty percent of a carefully hand-tuned system prompt, gone, no measurable loss. If that’s true of the people who write the model, it should terrify anyone who’s spent six months polishing 400 lines of instructions for a model they don’t control.
Here’s the honest comparison I wish someone had shown me a year ago.
+----------------------+---------------------------+----------------------------+
| Property | Prompts / Skills | Test Suites / Evals |
+----------------------+---------------------------+----------------------------+
| Encodes | Model-specific workarounds| Domain-specific correctness|
| Value over time | Decays, often silently | Compounds |
| Failure signal | None (output just drifts) | Explicit pass/fail |
| Survives model swap | Rarely, needs rework | Usually, with no changes |
| Cost to maintain | Recurring, unbounded | Upfront, then amortized |
| What breaks it | A model update | A real requirement change |
+----------------------+---------------------------+----------------------------+
The Case for Treating Evals as Capital
The clearest number I’ve found for what a good test actually buys you comes out of Google’s work on automated program repair. Their Passerine system generates a bug reproduction test before attempting a fix. When researchers measured the probability of Passerine producing a plausible fix conditional on whether that reproduction test was actually used in the repair loop, the number without it was 2 percent. With it, 33 percent. Same model, same bug set, same repair system. The only variable was whether the agent had a test that could tell it, unambiguously, whether its fix actually worked. That’s not a marginal improvement from better prompting. That’s the difference between an agent that’s mostly guessing and one that’s mostly verifying.
Anthropic’s own writing on evals for agents makes a related point with a different case. On CORE-Bench, Opus 4.5 initially scored 42 percent. That looked like a capability ceiling. It wasn’t. Researchers found the eval itself had bugs: overly rigid grading criteria and ambiguous task specs. Once the eval was fixed, the same model scored 95 percent on the same tasks. The model hadn’t gotten smarter. The measurement had gotten honest. That’s the capital argument in one sentence: a bad eval doesn’t just fail to help you, it actively lies to you about where you stand, and a good eval is the only thing standing between “I think this works” and “I know this works.”
Then there’s the number that should make every team doing “AI-assisted development” a little uncomfortable. METR ran a randomized controlled trial with 16 experienced open-source developers completing 246 real coding tasks, each about two hours, randomly assigned to be done with or without AI tool access. Developers using AI tools were 19 percent slower. Not faster. Slower. And here’s the part I find genuinely striking: after finishing, those same developers estimated they’d been about 20 percent faster with AI. The gap between how fast people feel and how fast they actually are was close to 40 percentage points in the wrong direction, and it was consistent across nearly everyone in the study.
I want to be fair about what happened next, because the real story is more interesting than “the results got buried.” METR didn’t quietly shelve the finding. They published it, then tried to run the same design again later in 2025 to track whether newer tools closed the gap. It fell apart, not because anyone hid the data, but because the world had changed underneath the experiment. By mid-2025, a growing share of developers refused to participate in the no-AI condition at all, even at fifty dollars an hour for tasks of their own choosing. Between 30 and 50 percent of eligible developers declined to submit tasks they suspected would be painfully slow without AI assistance. The people willing to do a task without AI were, by that point, a self-selected group whose baseline productivity without AI was no longer representative of anyone. The study wasn’t discontinued because the finding was inconvenient. It was discontinued because you can no longer recruit a clean control group once the intervention has become the default working condition for the population you’re trying to measure. That’s a fascinating failure mode in its own right, but the practical upshot for the rest of us is the same: nobody has a reliable, current number for how much AI-assisted coding actually helps, and the one clean measurement we do have says “less than developers think, and possibly negative.”
None of this means agents are useless. It means the confidence people have in their own agent workflows is not evidence the workflows work. The only thing that is evidence is a test that runs the same task the same way every time and tells you, mechanically, whether the output was correct. That’s the eval. That’s the capital.
A Minimal Eval Harness for .NET Agents
Here’s what I actually built once I stopped trying to fix this with better prompt phrasing. Most of my own agent tooling is Python and TypeScript, but a good chunk of my week involves a .NET services team, and this is the shape of harness I ended up putting in front of their CI pipeline. The goal is deliberately narrow: run a fixed, small set of representative tasks against your agent every time the prompt, the skill files, or the underlying model changes, score outcomes mechanically, and fail the build if the pass rate drops below a threshold you chose in advance, not one you back into after the fact.
The core pieces are an agent runner, a catalog of eval cases with outcome-based judges, a result sink, and a gate test that reads the accumulated results and fails if you’ve regressed.
// AgentEvalHarness.cs
using System.Diagnostics;
namespace Agent.Evals;
/// <summary>
/// One fixed, representative task the agent has to complete end to end.
/// The judge only looks at outcomes: did the code compile, did the tests
/// pass, did the specific behavior we care about actually change. It never
/// scores whether the agent "sounded right."
/// </summary>
public sealed record EvalCase(
string Id,
string Prompt,
string WorkspacePath,
Func<EvalRunResult, Task<bool>> Judge);
public sealed record EvalRunResult(
int ExitCode,
string StdOut,
string StdErr,
TimeSpan Duration);
public interface IAgentHarness
{
Task<EvalRunResult> RunAsync(string prompt, string workspacePath, CancellationToken ct);
}
/// <summary>
/// Shells out to whatever CLI runs your agent. Swap this for a fake in local
/// dev if you don't want to spend tokens on every test run.
/// </summary>
public sealed class CliAgentHarness : IAgentHarness
{
private readonly string _executablePath;
public CliAgentHarness(string executablePath) => _executablePath = executablePath;
public async Task<EvalRunResult> RunAsync(string prompt, string workspacePath, CancellationToken ct)
{
var psi = new ProcessStartInfo
{
FileName = _executablePath,
WorkingDirectory = workspacePath,
RedirectStandardOutput = true,
RedirectStandardError = true,
RedirectStandardInput = true,
UseShellExecute = false
};
var sw = Stopwatch.StartNew();
using var process = Process.Start(psi)!;
await process.StandardInput.WriteLineAsync(prompt);
process.StandardInput.Close();
var stdOut = await process.StandardOutput.ReadToEndAsync(ct);
var stdErr = await process.StandardError.ReadToEndAsync(ct);
await process.WaitForExitAsync(ct);
sw.Stop();
return new EvalRunResult(process.ExitCode, stdOut, stdErr, sw.Elapsed);
}
}
The catalog is where the actual eval lives. This is a real example, trimmed down from one of the regression tests I run against a ticket-triage agent. Notice the judge never asks the model, or a second model, whether the fix “looks right.” It runs dotnet test and checks that a specific regression test actually exists and passes.
// EvalCatalog.cs
using System.Diagnostics;
namespace Agent.Evals;
public static class EvalCatalog
{
public static EvalCase Load(string taskId, string workspaceRoot) => taskId switch
{
"null-check-fix" => new EvalCase(
Id: "null-check-fix",
Prompt: "In OrderService.cs, GetShippingAddress throws a " +
"NullReferenceException when Order.Customer is null. " +
"Fix it so the method returns null instead of throwing, " +
"and add a regression test that proves it.",
WorkspacePath: Path.Combine(workspaceRoot, "null-check-fix"),
Judge: async result =>
{
if (result.ExitCode != 0) return false;
var testRun = await RunDotnetTest(Path.Combine(workspaceRoot, "null-check-fix"));
return testRun.ExitCode == 0
&& testRun.StdOut.Contains("GetShippingAddress_NullCustomer");
}),
_ => throw new ArgumentException($"Unknown eval task '{taskId}'")
};
private static async Task<EvalRunResult> RunDotnetTest(string workspacePath)
{
var psi = new ProcessStartInfo("dotnet", "test --filter Category=Regression")
{
WorkingDirectory = workspacePath,
RedirectStandardOutput = true,
RedirectStandardError = true,
UseShellExecute = false
};
var sw = Stopwatch.StartNew();
using var process = Process.Start(psi)!;
var stdOut = await process.StandardOutput.ReadToEndAsync();
var stdErr = await process.StandardError.ReadToEndAsync();
await process.WaitForExitAsync();
sw.Stop();
return new EvalRunResult(process.ExitCode, stdOut, stdErr, sw.Elapsed);
}
}
Now the actual xUnit tests. One theory test per representative task, plus a separate gate test that reads the recorded results and fails the build if the pass rate drops below your threshold.
// AgentEvalSuite.cs
using Xunit;
using Xunit.Abstractions;
namespace Agent.Evals;
public class AgentEvalSuite
{
private readonly IAgentHarness _agent;
private readonly string _workspaceRoot;
private readonly ITestOutputHelper _output;
public AgentEvalSuite(ITestOutputHelper output)
{
var cliPath = Environment.GetEnvironmentVariable("AGENT_CLI_PATH") ?? "claude";
_agent = new CliAgentHarness(cliPath);
_workspaceRoot = Environment.GetEnvironmentVariable("EVAL_WORKSPACE_ROOT")
?? Path.Combine(Path.GetTempPath(), "agent-evals");
_output = output;
}
public static IEnumerable<object[]> RepresentativeTasks()
{
// Keep this list small and honest: 10 to 20 tasks pulled from real
// incidents or real PRs, not synthetic toy problems. If a task
// hasn't happened in production, it doesn't belong here.
yield return new object[] { "null-check-fix" };
yield return new object[] { "add-idempotency-key" };
yield return new object[] { "extract-repository-interface" };
}
[Theory]
[MemberData(nameof(RepresentativeTasks))]
public async Task Agent_completes_representative_task(string taskId)
{
var testCase = EvalCatalog.Load(taskId, _workspaceRoot);
var result = await _agent.RunAsync(testCase.Prompt, testCase.WorkspacePath, CancellationToken.None);
var passed = await testCase.Judge(result);
EvalResultSink.Record(taskId, passed, result.Duration);
_output.WriteLine($"[{taskId}] passed={passed} duration={result.Duration}");
Assert.True(passed, $"Eval '{taskId}' failed. Exit code {result.ExitCode}. Stderr: {result.StdErr}");
}
}
public class EvalGate
{
private const double MinimumPassRate = 0.80;
[Fact]
public void Pass_rate_must_not_regress_below_threshold()
{
var results = EvalResultSink.LoadAll();
Assert.NotEmpty(results);
var passRate = results.Count(r => r.Passed) / (double)results.Count;
Assert.True(
passRate >= MinimumPassRate,
$"Agent eval pass rate is {passRate:P0}, below the {MinimumPassRate:P0} floor. " +
"Do not ship this prompt, skill, or model change.");
}
}
And the result sink, a plain append-only JSON-lines file, deliberately boring so it survives being read by a second process in CI:
// EvalResultSink.cs
using System.Text.Json;
namespace Agent.Evals;
public static class EvalResultSink
{
private static readonly string OutputPath =
Environment.GetEnvironmentVariable("EVAL_RESULTS_PATH") ?? "eval-results.jsonl";
private static readonly object Lock = new();
public static void Record(string taskId, bool passed, TimeSpan duration)
{
var line = JsonSerializer.Serialize(new
{
taskId,
passed,
durationMs = duration.TotalMilliseconds,
timestamp = DateTimeOffset.UtcNow
});
lock (Lock)
{
File.AppendAllLines(OutputPath, new[] { line });
}
}
public static IReadOnlyList<(string TaskId, bool Passed)> LoadAll()
{
if (!File.Exists(OutputPath)) return Array.Empty<(string, bool)>();
return File.ReadAllLines(OutputPath)
.Select(line => JsonDocument.Parse(line).RootElement)
.Select(el => (el.GetProperty("taskId").GetString()!, el.GetProperty("passed").GetBoolean()))
.ToList();
}
}
In CI, run the two test classes as two separate steps so the gate always reads a complete result file:
name: agent-evals
on:
pull_request:
workflow_dispatch:
jobs:
eval-suite:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-dotnet@v4
with:
dotnet-version: '9.0.x'
- name: Run representative agent tasks
run: dotnet test --filter FullyQualifiedName~AgentEvalSuite
env:
AGENT_CLI_PATH: /usr/local/bin/claude
EVAL_RESULTS_PATH: ${{ github.workspace }}/eval-results.jsonl
- name: Gate on pass rate
run: dotnet test --filter FullyQualifiedName~EvalGate
env:
EVAL_RESULTS_PATH: ${{ github.workspace }}/eval-results.jsonl
One more thing worth building if you don’t have a Claude or OpenAI subscription to burn through on every CI run: swap CliAgentHarness for something that talks to a local model over Ollama, so the eval suite costs nothing to run repeatedly while you're developing it.
// OllamaAgentHarness.cs
using System.Diagnostics;
using System.Net.Http.Json;
namespace Agent.Evals;
public sealed class OllamaAgentHarness : IAgentHarness
{
private readonly HttpClient _http;
private readonly string _model;
public OllamaAgentHarness(HttpClient http, string model = "qwen2.5-coder:14b")
{
_http = http;
_model = model;
}
public async Task<EvalRunResult> RunAsync(string prompt, string workspacePath, CancellationToken ct)
{
var sw = Stopwatch.StartNew();
var payload = new
{
model = _model,
prompt = $"You are a coding agent working in {workspacePath}. Task: {prompt}. " +
"Return a unified diff only, nothing else.",
stream = false
};
using var response = await _http.PostAsJsonAsync("http://localhost:11434/api/generate", payload, ct);
response.EnsureSuccessStatusCode();
var body = await response.Content.ReadFromJsonAsync<OllamaResponse>(cancellationToken: ct);
sw.Stop();
// A production version would write body.Response to a .patch file,
// run `git apply`, then `dotnet build`, and return that exit code.
// Kept as a stub here because the interesting part is the harness
// contract, not the patch-application plumbing.
return new EvalRunResult(ExitCode: 0, StdOut: body!.Response, StdErr: string.Empty, Duration: sw.Elapsed);
}
private sealed record OllamaResponse(string Response);
}
Run ollama pull qwen2.5-coder:14b once, point IAgentHarness at OllamaAgentHarness in your test setup for local iteration, and switch back to CliAgentHarness for the CI runs that actually gate a real model or prompt change. You get the full harness loop for free while you're still figuring out which tasks belong in your representative set.
The Time-Budget Rule I Made Myself Follow
Here’s the rule of thumb I’ve settled on, and I’ll admit it felt wrong to write down the first time: spend at least as much engineering time on your eval suite as you spend on your prompts and skill files combined. Not less. At least as much, and if you’re maintaining an agent that touches production code, probably more.
This feels backwards to almost everyone coming from traditional software engineering, myself included, because in traditional software, tests are overhead. You write the feature, then you write tests to protect the feature you already built. The feature is the product. The tests are insurance.
With agents, that relationship inverts. The prompt is not the product. The prompt is a fragile, temporary configuration of a system you don’t control and can’t fully predict, and it will be partially or fully invalidated by the next model release whether you touch it or not. The eval suite is the actual asset, because it’s the only artifact you own that keeps telling the truth about whether the system works, across every single model swap, every skill update, and every prompt rewrite you’ll ever make. When Anthropic ships the next version and your CLAUDE.md instructions might now be redundant, or actively counterproductive the way that 25-word instruction turned out to be, your eval suite is the only thing that can tell you which parts of your prompt survived and which parts are now dead weight. Without it, you’re back to vibes, re-reading old tickets side by side and hoping you notice the drift in time.
I don’t hit my own rule every week. Some weeks I still catch myself tweaking a prompt for twenty minutes because it’s fast and satisfying and the payoff is immediate. But I track the ratio now, roughly, and when it drifts too far toward prompt tweaking I stop and write another eval case instead, even a small one. The eval case is the thing that’s still going to be true in six months.
Revisiting My Own CLAUDE.md With This Lens
I went back through my own prompt files after writing the first draft of this piece, mostly to keep myself honest. About a third of what’s in there is genuinely durable: project-specific facts like “tickets must include per-repo sizing” or “all generated code comments must be in English,” things that describe what correct output looks like rather than how to coax a specific model into producing it. I’ll keep maintaining that third indefinitely, because it’s really closer to a spec than a prompt.
The rest I’m treating as disposable starting now. The phrasing tricks, the “please think carefully” scaffolding, the workarounds for quirks a specific model version had that a newer version might not have anymore, all of it goes on a six-month clock. I’m not going to keep polishing instructions that exist to patch around a model’s current weaknesses, because those weaknesses are the least stable thing about this entire system. Instead, every one of those workarounds gets a matching eval case first. If the workaround is actually load-bearing, the eval will tell me when a new model makes it unnecessary, or when it starts failing again for a different reason. If it isn’t load-bearing, I’ll find that out too, and I’ll stop maintaining something that was never doing anything in the first place.
That’s the whole shift, really. Stop treating the thing that expires like it’s the product, and start treating the thing that tells you the truth like it’s the product. The prompt gets you through this model version. The eval gets you through all of them.
Tags: AI Agents, Software Testing, Claude Code, DevOps, DotNet, LLM Engineering, CI/CD
Top comments (0)