I used to think picking a coding harness was mostly a taste decision. Vim bindings or not, a TUI you like the look of, whether it plays nice with your terminal multiplexer. I assumed the model underneath was the thing that actually decided cost and quality, and the harness wrapped around it was closer to a skin than an engine. If you had asked me a month ago whether swapping Claude Code for a different harness, same model, same task, could change your bill by an order of magnitude, I would have said no. Maybe 20 percent, on a bad day.
Then I ran into a benchmark result that made that assumption look naive. Someone tested the same model against the same set of coding tasks across nine different harnesses. Pass rates barely moved, the best harness solved 66.7 percent of tasks and the worst solved 50 percent, a real gap but not a shocking one. Cost per successful task, on the other hand, ranged from about a dollar to over eighteen dollars. Same brain, wildly different bill, for work that came out roughly as good either way.
I want to walk through what I found when I chased that number down, because the chase itself taught me almost as much as the result did, and because I ended up building something out of it: a small cost-instrumentation wrapper I now run against my own agent sessions instead of trusting a vendor dashboard to tell me where the money goes.
The benchmark, and the number I had to walk back
The headline version of this story that reached me first, secondhand, through a newsletter roundup, quoted a much bigger gap: one harness solving a bug for $2.50, another spending $64.36 on the identical task. No link to the original writeup came with it, just the numbers. I wanted to cite that pairing directly because it’s a great, punchy example. So I went looking for where it actually came from.
What I found was Runta’s FrontierHarness Eval, a community benchmark (their own framing, not a vendor-sponsored one) that ran the Kimi K3 model against 30 coding tasks, 21 from Terminal-Bench and 9 from DeepSWE, across nine named harnesses. One of those harnesses, DeepSeek Harness, was tested in four separate configurations (Standard, Minimal, Creator, and one they call PTC), so the full run is really 12 configurations, 360 total evaluations, restored from an identical checkpoint each time (same vCPU, memory, disk contents) so nothing got an unfair head start from a warm cache left over from a previous trial.
The published numbers didn’t match what I’d been handed. Pi’s median cost per successful task was $2.43, not $2.50, close enough to be the same claim rounded differently. But Claude Code’s number in the actual report was $18.34, not $64.36. That’s a real discrepancy, not a rounding difference, and I don’t have a clean explanation for it. My best guess is that the $64.36 figure came from a specific single run, or an earlier version of the eval, or got garbled somewhere in the newsletter chain, and I’d rather tell you that plainly than repeat a number I can’t stand behind. What I can stand behind is the benchmark’s own published table, so that’s what I’m building the rest of this on. It turns out to make the same point, and honestly a cleaner one, because the actual field-wide spread (from the cheapest harness’s median cost per pass to the most expensive) comes out to almost exactly 17.5x, which is where this piece gets its title.
Here’s the table as Runta published it, reproduced as plain text so it survives copy-paste onto Medium or dev.to without markdown table syntax mangling it:
FRONTIERHARNESS EVAL, KIMI K3, 30 TASKS, 360 RUNS (AS REPORTED)
Harness Pass rate Median cost/pass Cache hit rate Your numbers
(median / token-wtd) (fill in if you replicate)
--------------------------------------------------------------------------------------------------
Codex 66.7% $3.47 88.0% / n/a ____
DSH Creator 63.3% $3.28 84.3% / n/a ____
Claude Code 63.3% $18.34 67.8% / 25.0% ____
Pi 60.0% $2.43 79.4% / n/a ____
DSH Standard 60.0% $3.46 86.5% / n/a ____
DSH PTC 60.0% $4.58 87.2% / n/a ____
Kimi Code 56.7% $3.65 88.0% / n/a ____
DSH Minimal 56.7% $4.72 84.6% / n/a ____
Oh My Pi 56.7% $4.75 82.2% / n/a ____
Exo Harness 53.3% $1.05 70.3% / n/a ____
Hermes 50.0% $2.90 85.9% / n/a ____
OpenCode 50.0% $3.24 78.4% / n/a ____
Field-wide pass rate: 58.1% (209 successes / 151 failures)
Pass rate spread, best to worst: 66.7 / 50.0 = 1.33x
Cost spread, most to least expensive median: 18.34 / 1.05 = 17.5x
Look at those two ratios stacked against each other. Pass rate, best harness to worst, is 1.33 times. Cost, most expensive to cheapest, is 17.5 times. If harness choice were purely a matter of taste, those two numbers would move together, or at least in the same neighborhood. They don’t. Something about the harness itself, independent of the model doing the reasoning, is burning more than seventeen times the money to arrive at answers that are, on the whole, no better.
The report itself is careful about a caveat I want to repeat rather than gloss over: this reflects “the full harness-model setup as delivered,” not some harness-agnostic quality score. Claude Code’s caching strategy is built around explicit cache breakpoints tuned for Claude models, and running it against a different model’s implicit prefix cache may not transfer cleanly. That matters for how you read the table. It is not proof that Claude Code is a bad harness in general. It is proof that this specific pairing, on this specific benchmark, produced this specific cost gap, and that the gap is worth explaining rather than shrugging off.
What a harness actually controls
The one column in that table that explains almost everything else is cache hit rate, and specifically the two numbers listed for Claude Code: 67.8 percent median, but only 25.0 percent when you weight by actual token volume instead of by session. That gap between the two numbers is the whole story in miniature. Most sessions cached reasonably well. But a handful of sessions, the ones that ran long, explored more dead ends, and pushed the most tokens through the pipe, cached badly, and because they’re the sessions carrying the most tokens, they dominate the token-weighted number and drag the effective average way down. A median across sessions hides exactly the failure mode that costs the most money.
That’s the mechanism, stated plainly: a harness controls how many tool calls a task takes to converge, how much of the transcript gets re-sent as fresh, uncached tokens on every turn, whether prompt caching is actually being exploited or just nominally available, and how long the agent is willing to wander through dead ends before it either finds the fix or gives up and tries something else. None of that is a property of the underlying model. Two harnesses running the identical model can produce identical answers to identical prompts and still diverge by 17x on the bill, because one of them held its prompt prefix still enough for the cache to do its job and the other one didn’t, or because one converged in six tool calls and the other took twenty-four to arrive at the same fix.
I didn’t want to take the benchmark’s word for this without checking it against something I could see myself, so I built a small tool to watch it happen on my own sessions.
Building my own cost instrumentation
The idea is simple: wrap every LLM call your harness makes, pull the token counts and cache fields straight out of the API response, price them against your provider’s published rates, and write a row to a local database so you can query it later instead of guessing. I wrote this in C# against a .NET 8 console app, using SQLite for storage so there’s nothing to stand up, no Docker container, no cloud account, just a file on disk.
My first pass at the pricing math was wrong, and it’s worth saying why, because it’s an easy mistake to make. I assumed Anthropic’s input_tokens field in the usage object counted the whole prompt, cached portions included, so I subtracted cache_creation_input_tokens and cache_read_input_tokens from it before pricing the remainder at full input rate. My totals came out too low, noticeably lower than what the Anthropic console showed for the same session. It turns out input_tokens already excludes anything that went through the cache. It only counts the genuinely fresh, uncached portion of that request. Subtracting the cache fields a second time was double-discounting. The fix was to stop subtracting and just price each of the three input buckets independently. Here's the corrected version, which is the one I'd actually recommend copying:
// CostTracker.cs
// .NET 8. Add Microsoft.Data.Sqlite:
// dotnet add package Microsoft.Data.Sqlite
using Microsoft.Data.Sqlite;
namespace HarnessCostAudit;
public sealed record ModelPricing(
decimal InputPerMillion,
decimal OutputPerMillion,
decimal CacheWritePerMillion,
decimal CacheReadPerMillion);
public static class Pricing
{
// Published per-million-token USD rates at time of writing.
// Check your provider's current pricing page before trusting these long-term.
// Cache write runs at roughly 1.25x base input; cache read runs at roughly 0.1x.
public static readonly ModelPricing ClaudeSonnet = new(
InputPerMillion: 3.00m,
OutputPerMillion: 15.00m,
CacheWritePerMillion: 3.75m,
CacheReadPerMillion: 0.30m);
}
public sealed record CallRecord(
string SessionId,
string TaskType,
int InputTokens,
int OutputTokens,
int CacheWriteTokens,
int CacheReadTokens,
decimal CallCostUsd,
DateTimeOffset Timestamp);
public sealed class CostTracker
{
private readonly string _dbPath;
private readonly ModelPricing _pricing;
public CostTracker(string dbPath, ModelPricing pricing)
{
_dbPath = dbPath;
_pricing = pricing;
EnsureSchema();
}
private void EnsureSchema()
{
using var conn = new SqliteConnection($"Data Source={_dbPath}");
conn.Open();
var cmd = conn.CreateCommand();
cmd.CommandText = """
CREATE TABLE IF NOT EXISTS llm_calls (
id INTEGER PRIMARY KEY AUTOINCREMENT,
session_id TEXT NOT NULL,
task_type TEXT NOT NULL,
input_tokens INTEGER NOT NULL,
output_tokens INTEGER NOT NULL,
cache_write_tokens INTEGER NOT NULL,
cache_read_tokens INTEGER NOT NULL,
call_cost_usd REAL NOT NULL,
timestamp TEXT NOT NULL
);
""";
cmd.ExecuteNonQuery();
}
// input_tokens, cache_write_tokens and cache_read_tokens are already
// mutually exclusive buckets in Anthropic's usage object. Do not
// subtract one from another, price each bucket at its own rate.
public decimal ComputeCost(int inputTokens, int outputTokens, int cacheWriteTokens, int cacheReadTokens) =>
inputTokens / 1_000_000m * _pricing.InputPerMillion +
cacheWriteTokens / 1_000_000m * _pricing.CacheWritePerMillion +
cacheReadTokens / 1_000_000m * _pricing.CacheReadPerMillion +
outputTokens / 1_000_000m * _pricing.OutputPerMillion;
public void Record(CallRecord record)
{
using var conn = new SqliteConnection($"Data Source={_dbPath}");
conn.Open();
var cmd = conn.CreateCommand();
cmd.CommandText = """
INSERT INTO llm_calls
(session_id, task_type, input_tokens, output_tokens,
cache_write_tokens, cache_read_tokens, call_cost_usd, timestamp)
VALUES
($sid, $type, $in, $out, $cw, $cr, $cost, $ts);
""";
cmd.Parameters.AddWithValue("$sid", record.SessionId);
cmd.Parameters.AddWithValue("$type", record.TaskType);
cmd.Parameters.AddWithValue("$in", record.InputTokens);
cmd.Parameters.AddWithValue("$out", record.OutputTokens);
cmd.Parameters.AddWithValue("$cw", record.CacheWriteTokens);
cmd.Parameters.AddWithValue("$cr", record.CacheReadTokens);
cmd.Parameters.AddWithValue("$cost", (double)record.CallCostUsd);
cmd.Parameters.AddWithValue("$ts", record.Timestamp.ToString("O"));
cmd.ExecuteNonQuery();
}
public decimal RunningTotal(string? sessionId = null)
{
using var conn = new SqliteConnection($"Data Source={_dbPath}");
conn.Open();
var cmd = conn.CreateCommand();
cmd.CommandText = sessionId is null
? "SELECT COALESCE(SUM(call_cost_usd), 0) FROM llm_calls;"
: "SELECT COALESCE(SUM(call_cost_usd), 0) FROM llm_calls WHERE session_id = $sid;";
if (sessionId is not null) cmd.Parameters.AddWithValue("$sid", sessionId);
return Convert.ToDecimal(cmd.ExecuteScalar());
}
public double CacheHitRate(string? taskType = null)
{
using var conn = new SqliteConnection($"Data Source={_dbPath}");
conn.Open();
var cmd = conn.CreateCommand();
cmd.CommandText = taskType is null
? "SELECT COALESCE(SUM(cache_read_tokens),0), COALESCE(SUM(input_tokens + cache_read_tokens + cache_write_tokens),0) FROM llm_calls;"
: "SELECT COALESCE(SUM(cache_read_tokens),0), COALESCE(SUM(input_tokens + cache_read_tokens + cache_write_tokens),0) FROM llm_calls WHERE task_type = $t;";
if (taskType is not null) cmd.Parameters.AddWithValue("$t", taskType);
using var reader = cmd.ExecuteReader();
reader.Read();
long cacheRead = reader.GetInt64(0);
long totalInput = reader.GetInt64(1);
return totalInput == 0 ? 0 : (double)cacheRead / totalInput;
}
}
And the piece that actually calls the API and logs every request as it happens:
// InstrumentedAnthropicClient.cs
using System.Net.Http.Json;
using System.Text.Json;
namespace HarnessCostAudit;
public sealed class InstrumentedAnthropicClient
{
private readonly HttpClient _http;
private readonly CostTracker _tracker;
public InstrumentedAnthropicClient(string apiKey, CostTracker tracker)
{
_http = new HttpClient { BaseAddress = new Uri("https://api.anthropic.com/") };
_http.DefaultRequestHeaders.Add("x-api-key", apiKey);
_http.DefaultRequestHeaders.Add("anthropic-version", "2023-06-01");
_tracker = tracker;
}
public async Task<string> SendAsync(string sessionId, string taskType, object requestBody)
{
var response = await _http.PostAsJsonAsync("v1/messages", requestBody);
response.EnsureSuccessStatusCode();
var json = await response.Content.ReadFromJsonAsync<JsonElement>();
var usage = json.GetProperty("usage");
int inputTokens = usage.GetProperty("input_tokens").GetInt32();
int outputTokens = usage.GetProperty("output_tokens").GetInt32();
int cacheWrite = usage.TryGetProperty("cache_creation_input_tokens", out var cw) ? cw.GetInt32() : 0;
int cacheRead = usage.TryGetProperty("cache_read_input_tokens", out var cr) ? cr.GetInt32() : 0;
var cost = _tracker.ComputeCost(inputTokens, outputTokens, cacheWrite, cacheRead);
_tracker.Record(new CallRecord(sessionId, taskType, inputTokens, outputTokens, cacheWrite, cacheRead, cost, DateTimeOffset.UtcNow));
var hitLabel = cacheRead > 0 ? "CACHE HIT" : "cache miss";
Console.WriteLine(
$"[{taskType}] in={inputTokens} out={outputTokens} cacheWrite={cacheWrite} cacheRead={cacheRead} " +
$"{hitLabel} cost=${cost:0.0000} running=${_tracker.RunningTotal(sessionId):0.00}");
return json.GetProperty("content")[0].GetProperty("text").GetString() ?? string.Empty;
}
}
If you don’t have an Anthropic subscription and want to see this run without spending anything, point the same tracker at a local Ollama model instead. Ollama has no billing and no prompt-cache concept in the same sense, but it does return prompt_eval_count and eval_count on every response, and those are worth logging too, because the thing you're measuring locally is throughput and wall-clock latency rather than dollars:
public async Task<string> SendToOllamaAsync(string sessionId, string taskType, object requestBody)
{
var sw = System.Diagnostics.Stopwatch.StartNew();
var response = await _http.PostAsJsonAsync("http://localhost:11434/api/chat", requestBody);
response.EnsureSuccessStatusCode();
var json = await response.Content.ReadFromJsonAsync<JsonElement>();
sw.Stop();
int promptTokens = json.GetProperty("prompt_eval_count").GetInt32();
int outputTokens = json.GetProperty("eval_count").GetInt32();
_tracker.Record(new CallRecord(sessionId, taskType, promptTokens, outputTokens, 0, 0, 0m, DateTimeOffset.UtcNow));
Console.WriteLine($"[{taskType}] in={promptTokens} out={outputTokens} elapsed={sw.ElapsedMilliseconds}ms (self-hosted, no per-call billing)");
return json.GetProperty("message").GetProperty("content").GetString() ?? string.Empty;
}
Running this against my own harness for a week turned up exactly the pattern the benchmark predicted: cost per task correlated far more tightly with CacheHitRate grouped by task type than with which model I'd pointed the session at. A task type where I was pasting a large, slightly-different code excerpt into the system context on every turn had a hit rate under 20 percent and cost four times more per completion than a task type where the context stayed stable turn to turn. Same model, same person, same week. The harness-shaped part of my own workflow was the expensive part.
How DeepSeek Harness engineers for the cache, and where that claim’s footing gets thin
The clearest writeup I found on deliberately engineering for a high cache hit rate is a piece describing DeepSeek Harness’s internals, and I want to be upfront about exactly how solid that ground is, because the piece itself is careful about this and I’d rather match that care than blur it.
The documented, load-bearing mechanism is straightforward: DeepSeek’s API checks how much of an incoming prompt’s prefix is a byte-for-byte match against something already computed and cached, starting from token zero. Any divergence anywhere in that prefix, a timestamp, a reordered tool list, a session ID pasted in for logging, breaks the match from that point forward for the rest of the prompt, no partial credit. DeepSeek Harness holds that prefix still by keeping tool definitions in a deterministic order across turns instead of re-sorting or re-generating them, and by treating each session’s history as an event-sourced, append-only log: nothing already written gets rewritten, and the system prompt for a session stays fixed unless the user deliberately switches agent modes. Those two choices together mean the part of the prompt above the newest turn looks identical, character for character, to what the cache already has, turn after turn, which is the entire precondition for a cache hit.
Where I want to slow down is the numbers attached to that mechanism. The 97 to 99 percent cache hit rates reported for DeepSeek Harness, and the anecdote about a full one-shot app build costing six cents total, come from community threads and the author’s own usage, explicitly flagged in the source material as self-reported and unverified rather than a vendor benchmark. The author says as much directly: none of this is DeepSeek doing anything special on the API side, it’s the harness holding still long enough for an ordinary prefix-cache feature to do its job. That’s a meaningful distinction. The mechanism (deterministic ordering, immutable session history) is a real, checkable engineering pattern you can go verify in the harness’s own behavior. The specific hit-rate percentages are a claim from one team about their own workload, not a documented, reproducible figure the way Anthropic’s or DeepSeek’s published pricing pages are. Use the pattern. Be more careful with the number.
One thing the source doesn’t cover, and which I went looking for separately, is a way to audit why a cache hit rate changed between two runs of the same task. That’s a real gap, and it’s exactly what the median-versus-token-weighted split in the FrontierHarness table is a crude version of: if your aggregate hit rate drops, you want to know whether it dropped evenly everywhere or whether one long, badly-behaved session is dragging the average down while most of your traffic is fine. A tool like the tracker above, grouped by session and by task type instead of collapsed into one number, is a start on that audit trail even without anything fancier behind it.
A different lever: give the agent the whole machine, not a smaller one
Not every efficiency story here is about squeezing the same tool budget harder. Ramp’s background coding agent, built internally on top of OpenCode and named Inspect, took close to the opposite bet. Instead of handing the agent a minimal, sandboxed toolset and hoping it asks the right follow-up questions when it’s missing context, Ramp gives each session a full remote sandbox that mirrors a real developer’s machine: Postgres, Redis, Temporal, and RabbitMQ running alongside the agent, a VS Code server and a web terminal for a human to drop into mid-session, and a VNC stack with a real browser for visual verification of whatever the agent just built. Sessions start from filesystem snapshots refreshed roughly every 30 minutes, so a new session boots in a few seconds instead of paying full repo-clone and dependency-install time on every run.
Within a couple of months of shipping this, Inspect was writing more than half of all merged pull requests at the company. That’s not a cost story, and Ramp isn’t claiming it is, giving an agent that much surrounding infrastructure is not the cheap option. It’s a convergence story: an agent that can actually run the full stack, query the real database, and see the real rendered page in a browser wastes far fewer turns guessing at state it can’t observe, and that shows up as fewer dead-end tool calls and fewer wrong first drafts, which is the same lever the FrontierHarness benchmark was measuring, approached from the opposite direction. Less exploration because there’s less to explore blindly, rather than less exploration because the harness is stingy with tool calls.
The sandbox’s real attack surface isn’t the hypervisor
Handing an agent that much of a real machine raises the obvious question, and it’s worth being precise about where the actual risk sits. Google’s guidance on agent sandboxes for GKE is direct about this: the security model leads with a default-deny network posture for every sandboxed environment, specifically so that code the agent generates and executes cannot reach internal networks or the cluster’s control plane unless something explicitly allows it. Kernel-level isolation, gVisor as a hardened runtime, Kata Containers when you want a harder VM-like boundary, is described as a necessary layer underneath that, not the layer doing the real work of keeping a compromised or simply overzealous agent contained.
That ordering matters more than it sounds like it should. A hypervisor boundary stops the agent’s code from escaping into the host machine. It does nothing to stop the agent from exfiltrating your source tree to an attacker-controlled endpoint, or pulling down and running something it shouldn’t, over a network connection that was left wide open because locking it down felt like it would just get in the agent’s way. The egress rules, defined per sandbox template, are what decide whether a full-machine environment like Ramp’s is a productivity win or a very large, very quiet hole in your network. Full dev-machine access and locked-down egress are not in tension with each other. They’re the two halves of the same design decision, and skipping the second half because the first half already feels generous is exactly the mistake this guidance is trying to head off.
A checklist before you pick or build a harness
If I were doing this over for a .NET team evaluating harnesses instead of just writing about someone else’s benchmark, here’s the order I’d actually work through, roughly cheapest-to-check first:
Measure your own cache hit rate before optimizing anything else, and measure it two ways, a simple average across sessions and a token-weighted average across all the raw tokens you sent. The gap between those two numbers, the way Claude Code’s 67.8 percent median versus 25.0 percent token-weighted rate shows up in the benchmark table, tells you whether a small number of expensive sessions are quietly wrecking your bill while your dashboard’s headline number looks fine.
Log cost per task type, not just per session or per day. A single aggregate number hides exactly the variance that matters, the same way one blended pass rate would have hidden the fact that nine harnesses solving similarly-hard problems can cost 17.5 times apart from each other.
Count tool calls and turns per task, not only tokens. A harness that converges in six tool calls and one that takes twenty-four to reach the same fix will show completely different costs even with identical per-token pricing and a decent cache hit rate, because the twenty-four-call session is re-sending a longer, ever-growing transcript on every single one of those turns.
If you’re giving an agent broad sandbox access the way Ramp does, check the egress rules before you check anything else about the sandbox’s isolation story. A default-deny network posture with explicit, audited exceptions is the part of this that actually needs to be right on day one.
Re-run the benchmark yourself before trusting anyone else’s published numbers, including the ones in this article. I went looking for a $2.50-versus-$64.36 headline and found a real, verifiable $2.43-versus-$18.34 gap instead, on a benchmark that’s already a snapshot of one model, one task set, one week. Harnesses update, models update, caching behavior changes underneath both of them without much announcement. The table above has a column for your own numbers for a reason. Fill it in before you make a decision that outlives this month’s version of any of these tools.
Tags: ai-coding-agents, llm-cost-optimization, prompt-caching, dotnet, agent-sandboxing, devops
Top comments (0)