There is a problem with most benchmarks for coding agents: they are not software development.
They are useful, of course. Give an agent an issue, run a test suite, check whether the patch passes. SWE-bench and similar evaluations give us a standardized way of comparing models.
But this is not how I actually use agents.
I don't want an agent to fix a single isolated issue and stop. I want it to spend hours inside a codebase, understand the architecture, write requirements, implement things, run tests, benchmark alternatives, discover that its first idea was wrong, revise it, document what happened, and then use that new knowledge in the next task.
So, during the last few days, I accidentally created another kind of benchmark.
I gave DeepSeek Harness (DSH) + DeepSeek V4 Pro a real project and let it work.
The result was:
| Metric | Result |
|---|---|
| Visible development window | Aug 31 → Sep 3 |
| Git commits | 45 |
| Files changed | 193 |
| API requests | 3,266 |
| Tokens processed | 612,923,681 |
| Cache-hit input | 600,885,248 |
| Cache-miss input | 6,458,427 |
| Output | 5,580,006 |
| Input cache-hit rate | 98.94% |
| API cost | $28.35 |
Yes.
612 million tokens. Twenty-eight dollars.
And before anyone starts typing "tokens are not productivity" into the comments: I agree.
That is exactly why the interesting part of this experiment is not the token count.
It is what happened to the repository.
The Project
The project is LovelaceSharp, my attempt to build an arbitrary-precision mathematical environment in C#.
It contains arbitrary-precision natural numbers, integers and real numbers, a scripting language, vectors and N-dimensional arrays, linear algebra, a web IDE, benchmarking tools and a Lean project for formally verifying the underlying arithmetic model.
This is a useful agentic workload because it is not a CRUD application.
There are plenty of opportunities for code that looks correct but isn't.
Arithmetic has edge cases. Numeric algorithms have crossover points. Performance changes can destroy correctness. Array views introduce aliasing and stride semantics. Precision can leak across sessions. Formal proofs can prove something subtly different from what the optimized production implementation actually does.
In other words:
bullshit has somewhere to hide.
That makes it a much more interesting test than asking an agent to add another REST endpoint.
Establishing the Boundary
I also got lucky with the Git history.
The old work on LovelaceSharp stopped on March 17 at commit 9b26f05. There was then no activity until August 31, when the new DSH-driven development started.
The comparison from that old baseline to the end of the visible burst contains:
45 commits across 193 changed files.
You can inspect the history and comparison directly:
So I am not counting the previous implementation as work performed by DSH.
That matters.
The benchmark isn't:
"Look at this entire repository an AI supposedly made."
It is:
"Here is the repository before this run, here is the repository after it, and here is the API bill during the days those changes landed."
My complete DeepSeek export spans Aug 30 through Sep 5 and contains roughly:
- 805.4M tokens
- 3,931 requests
- $36.68 of spending
For the benchmark, however, I only attribute Aug 31 through Sep 3, because those are the dates represented by the visible development burst on main.
That leaves:
612.9M tokens and $28.35
associated with the 45-commit comparison.
So what did those $28 actually buy?
First: Replace the Core Number Representation
One of the largest changes was rewriting Natural, the arbitrary-precision unsigned integer implementation.
The previous representation was based around decimal BCD storage.
The new implementation moved the actual number representation to little-endian base-2^64 limbs.
That is already a reasonably dangerous refactor because essentially every numerical type above it depends on Natural.
The change included:
- native carry/borrow arithmetic;
- schoolbook multiplication;
- Karatsuba multiplication;
- Knuth Algorithm D division;
- short division;
- divide-and-conquer conversion between decimal and binary representations.
But the important part is what came with it.
DSH added tests specifically around limb boundaries and randomized differential testing against System.Numerics.BigInteger.
The differential suite covers arithmetic across multiple operand sizes and adversarial values around boundaries such as 2^64 and 2^128.
Relevant commit:
Rewrite Natural storage around 64-bit limbs
That is much more interesting to me than "the model knew Knuth division."
LLMs know algorithms.
The useful behavior is:
change representation
↓
identify new failure boundaries
↓
construct an independent oracle
↓
cross-check the implementation
That is engineering.
Then It Made Multiplication More Complicated — for a Reason
After moving to binary limbs, the agent added an exact two-prime Number Theoretic Transform multiplication path for very large numbers.
This wasn't simply:
NTT is asymptotically fast, therefore NTT everywhere.
It benchmarked the implementation against Karatsuba and introduced a dispatch threshold around 100,000 combined limbs, roughly the region where the NTT path actually started winning.
It also cross-checked multiplication against BigInteger at extremely large operand sizes.
Relevant commit:
Add NTT multiplication for huge Natural values
This distinction matters enormously when evaluating agents.
A coding model can produce complicated code all day long.
A useful engineering agent must understand that an asymptotically superior algorithm can still be the wrong implementation for almost every practical input.
And that became even more obvious with division.
Then It Implemented an Algorithm and Discovered It Should Barely Use It
DSH implemented Newton-reciprocal division with asymptotically better behavior for huge operands.
Then it benchmarked it.
And Knuth division won.
Not forever, but for a surprisingly long time.
The measured crossover put Newton at roughly the multi-million-decimal-digit range, so the production dispatcher retained Knuth below that region and only selected Newton for enormous inputs.
Relevant commit:
Add Newton reciprocal division and benchmark its crossover
This may be my favorite part of the experiment.
Because the workflow was essentially:
hypothesis
↓
implementation
↓
measurement
↓
"well... that didn't work as expected"
↓
revised implementation
I want more agent benchmarks to measure this.
Not whether the model knew Newton iteration.
Whether the agent was willing to prove itself wrong.
The Work Wasn't Limited to Big Integers
During the same development burst, Lovelace gained a typed-array abstraction with concepts such as:
ArrayValueDenseArray<T>- dtype metadata
- precision metadata
- slices
- strided views
- a plugin-oriented kernel contract
The abstraction arrived with tests covering array layout and view behavior.
Relevant commit:
Introduce typed array abstractions
Then the scripting engine itself was migrated onto the typed representation, adding things such as:
- promotion;
- narrowing;
- broadcasting;
- slicing;
- views;
- empty-dimension behavior.
Relevant commit:
Migrate Suite to the typed array representation
Around the same period, the project also acquired a web IDE, Lovelace.Studio, over the common scripting engine.
The initial Studio work extracted the scripting language into a shared engine and built an ASP.NET Core + browser IDE around it.
Relevant commit:
A later change added isolated sessions, per-session precision, incremental computation, asynchronous progress and CodeMirror autocomplete.
Relevant commit:
Add isolated sessions and incremental evaluation
The point is not LOC.
The point is that the workload moved through:
numerical algorithms, language implementation, web tooling, concurrency, arrays, benchmarks and formal methods
without changing the overall agentic workflow or treating each category as a fresh synthetic benchmark.
That is much closer to the way I want an engineering agent to behave.
It Also Started Proving Things
Another part of the burst introduced Lovelace.Proofs, a Lean 4 project formalizing base-b arithmetic.
It covers representation, addition, subtraction, multiplication and division using core Lean.
Relevant commit:
Add Lean proofs for core arithmetic
There is an important caveat here.
The Lean proofs describe the digit-by-digit reference arithmetic.
The production Natural implementation was subsequently changed to binary limbs and gained optimized Knuth, NTT and Newton paths.
So saying:
"Lean proves the optimized implementation is correct"
would be wrong.
What we actually have is closer to:
a formally verified arithmetic reference model plus optimized production implementations checked through unit tests, boundary tests and differential testing.
I think that distinction is worth mentioning because this is another place where agentic development can become dangerous.
An agent can add formal verification and make a project look dramatically safer while proving something adjacent to the thing actually running.
The human still needs to understand the correspondence.
Now Let's Talk About the 612 Million Tokens in the Room
At first glance, this workload looks horrifyingly inefficient.
612,923,681 tokens in four days.
That is around 188K processed tokens per API request.
If your mental model of agent economics is:
tokens × normal input price = bill
this should have been expensive.
But that is not what happened.
Of the input sent during the benchmark window:
cache hit: 600,885,248
cache miss: 6,458,427
The resulting input cache-hit rate was:
98.94%.
For DeepSeek V4 Pro alone during the same period, it was even higher.
And this wasn't one anomalous gigantic request dominating the average. Across the complete development days, the provider-side daily numbers remained extremely cache-heavy.
DeepSeek supports automatic context caching for repeated prompt prefixes:
DeepSeek Context Caching documentation
That matters a lot for a persistent harness.
Long-running agent sessions naturally contain large stable prefixes:
- previous prompts;
- repository discoveries;
- tool results;
- architectural decisions;
- earlier failures;
- requirements;
- test observations;
- existing conversation state.
If the harness keeps those prefixes stable enough, the provider does not have to charge them like entirely novel input every time.
So the raw token count is enormous.
The novel expensive prefix is not.
Cache Hits Change the Economics Completely
The provider export records different prices for:
- cache-hit input;
- cache-miss input;
- output.
For V4 Pro, cache-hit input in the export was priced at 1/30th of cache-miss input.
That is the part that makes the economics weird.
Using the actual pricing tiers recorded in my export, the DSH development window cost:
$28.35.
Now take exactly those same requests and make one artificial change:
pretend every cached input token had instead been charged as a cache miss, preserving the same observed price bands.
The cost would have been on the order of hundreds of dollars rather than tens of dollars.
For the complete Aug 30 → Sep 5 export, the observed bill was $36.68.
Using the same thought experiment, the equivalent bill without cache reuse is roughly $567.
So caching reduced the total bill by approximately:
93.5%
or around:
15.4×
for that complete export.
This is where I think the usual discussion about "token efficiency" becomes a little misleading.
DSH did not optimize this workflow by minimizing how many tokens it processed.
It processed a ridiculous number of tokens.
What made the economics work was that almost all of the input became cheap to revisit.
There is a fundamental difference between:
minimize context
and:
maximize reusable context.
For an agent doing sustained engineering work, the second strategy can be surprisingly attractive.
Context Can Become an Asset Instead of a Tax
Traditional context optimization tends to look like this:
large context
↓
summarize
↓
discard details
↓
keep prompt small
That makes sense when every input token is similarly expensive.
With strong prefix caching, another strategy becomes possible:
accumulate useful context
↓
keep the stable portion stable
↓
reuse it repeatedly
↓
pay mostly cache-read pricing
This does not mean "send everything forever."
Cached tokens still cost money.
Irrelevant context can still hurt:
- attention;
- latency;
- model quality;
- tool-selection quality;
- reasoning focus.
A 99% hit rate can also become a vanity metric if the harness is repeatedly hauling 500K tokens it never needed.
So I would not propose:
cache hit rate = agent quality.
But I would absolutely propose:
cache locality is a first-class agent-runtime metric.
Because once an agent starts working for hours or days instead of answering isolated prompts, context reuse becomes an economic property of the architecture.
And This Is Why "$ per Million Tokens" Is the Wrong Agent Benchmark
My effective blended price during this development window was approximately:
$0.046 per million processed tokens.
That number sounds absurd because it includes hundreds of millions of cache reads.
But it also demonstrates why comparing agents using raw token consumption is increasingly questionable.
Suppose Agent A uses 10 million tokens and forgets half the repository every few turns.
Agent B processes 100 million tokens but 99 million of them come from stable cached context containing decisions, tests, observations, architectural constraints and previous failures.
Which one is more efficient?
You cannot answer that from token count.
Even cost alone does not answer it.
What actually matters is something closer to:
validated engineering work
──────────────────────────
inference cost
And eventually:
validated engineering work
──────────────────────────
human review time + inference cost
That second denominator is probably where things get uncomfortable.
The New Bottleneck Is Me
This experiment cost $28.35 in attributable inference.
The repository received 45 commits across 193 files.
The agent was able to move between:
- formal proofs;
- big-integer arithmetic;
- array semantics;
- benchmarking;
- performance dispatch;
- Native AOT work;
- scripting-language behavior;
- a browser IDE;
- concurrency and session isolation.
At this point, shaving another five dollars from inference is almost irrelevant.
My problem is reviewing it.
Did an NTT edge case escape the differential tests?
Are array aliases correct under every non-contiguous view?
Does per-session precision really eliminate all shared-state races?
Do the Lean theorems correspond closely enough to the optimized code paths for the documentation to describe them accurately?
Those questions are now more expensive than generating another implementation.
And I think this is an important transition in agentic software development.
For a long time, generating code was the expensive part.
Then generating good code became the challenge.
With cheap models, aggressive caching and persistent harnesses, we may be moving toward another bottleneck:
How much machine-generated engineering can a human responsibly validate?
That is a very different problem.
So Is This a Benchmark?
Not in the SWE-bench sense.
There is no controlled comparison against Claude Code, Codex, Cursor or another harness.
There is no randomized task set.
I cannot derive:
"DSH is X times better than Claude Code"
from this experiment.
Anyone doing that from these numbers would be abusing the data.
It is better understood as a real-work agentic case benchmark.
It has a reproducible Git boundary:
9b26f05 → current DSH-era main
It has a public work product:
45 commits, 193 changed files.
It has provider-side usage data:
612.9M processed tokens, 3,266 requests.
It has provider-side cache accounting:
98.94% input cache hit.
And it has an actual bill:
$28.35.
Most importantly, the output is available for anyone to inspect.
You don't have to believe me that the work is good.
The code is the benchmark artifact.
Repository:
https://github.com/jjackbauer/LovelaceSharp
What I Would Measure Next
After seeing this run, I am much less interested in "tokens per task."
For sustained agentic engineering, I think the useful metrics are going to look more like:
- cost per validated change;
- percentage of agent changes surviving human review;
- regression rate after integration;
- human review minutes per accepted change;
- cache-hit distribution;
- uncached input per accepted change;
- wall-clock time to accepted change;
- benchmark improvements that survive independent reproduction.
Because a harness can cheat almost every superficial metric.
It can produce fewer tokens by forgetting things.
It can produce more commits by making tiny commits.
It can produce more code by generating garbage.
It can get a spectacular cache-hit rate by repeatedly sending irrelevant history.
It is much harder to cheat:
Did the software get materially better, did the changes survive verification, and how much did that cost?
That is the benchmark I care about.
Conclusion
I started looking at the usage dashboard because 800 million tokens for about $36 looked ridiculous.
It is ridiculous.
But the interesting result wasn't simply that DeepSeek V4 Pro is cheap.
The interesting result was understanding why this workload stayed cheap while maintaining enormous context.
DSH + DeepSeek V4 Pro effectively turned repeated context from one of the largest costs of agentic development into a heavily amortized resource.
During the cleanest observable development window, that meant:
45 commits.
193 files.
612.9 million processed tokens.
98.94% cache-hit input.
$28.35.
There are still plenty of things I don't trust without human verification.
There should be.
An agent writing formal mathematics and arbitrary-precision arithmetic should not earn trust because a dashboard has a nice green number.
But if we want to discuss the economics of agents seriously, I think we need to stop benchmarking them as expensive autocomplete.
Persistent agents behave differently.
Their context behaves differently.
And, apparently, their bills can behave very differently too.
Top comments (0)