We spent the past few months on a question that sounds simple: what does it actually take to move a coding workload off a frontier API and onto open-weight models you run yourself?
We picked coding agents deliberately. They are long-horizon, they are messy, and they are the workload most organisations would move first, because that is where the token bill is growing fastest. We ran a fixed set of real software engineering tasks across a range of open-weight models, each paired with hardware capable of serving it properly, and varied the agent harness on top. Then we measured what came out: tasks resolved, time taken, users served in parallel.
Three things surprised us.
Capability is priced in steps, it’s not linear
The intuition going in was that better models cost proportionally more to run. They don’t. Quality improves gradually across the open-weight field, but the hardware underneath moves in jumps, because what matters is whether the weights fit in the memory of the box you already have. Cross that line and you are not paying for a few billion more parameters, you are paying for the next machine.
So some models are badly priced for what they give you and others are a bargain, and which is which has little to do with where they sit on a leaderboard. One mid-sized model in our set cost 4 times more to serve than a smaller one while barely improving the application performance with 2%. A much larger one cost only marginally more than the tier below while solving substantially more. The useful question is not which model is best, but which model lands just inside a hardware tier, measured per resolved task rather than per token.
Concurrency is the variable nobody budgets for
Behind an API, concurrency is someone else’s problem. You get a reasonably predictable response time and you stop thinking about it.
If you manage your own stack, it becomes the binding constraint. Every user brings a context, every context has to fit in memory, and latency stops being acceptable long before the hardware stops working. How many developers a box can comfortably carry varied enormously across our setups, and not in the direction the spec sheets suggest. The largest, most capable configuration served the fewest people at once.
This is where most self-hosting business cases quietly fall apart. The break-even utilisation for the weaker configurations we tested sits far above what enterprises actually achieve. Reported utilisation in real deployments is low, and well-run estates are not dramatically better. If your case needs near-continuous saturation, it does not work.
The harness changes the bill and only marginally influences application performance
We compared three widely used coding harnesses on identical hardware with identical inference settings, expecting the orchestration layer to matter a great deal. For resolve rate, it barely moved the needle. The spread across harnesses was a couple of tasks either way.
For cost, it mattered a lot. Same model, same hardware, same tasks resolved, and wall time differing by a factor of two depending on the pairing. When you are renting GPUs, wall time is what really defines your cost.
And when you settle on a harness, evals are key as things can easily break. In our experiments, one harness was the clear winner, until the next software release, made our performance collapse. The culprit was a small change in how prefix caching was handling. Suddenly, the caching mechanism broke entirely and the GPU rental cost skyrocketed with a factor 5. The key takeaway is: pick the harness your developers want to use, then instrument it, because a point release can move your costs more than a model swap.
What we would tell you to do with this
The question is not whether self-hosting is cheaper. Asked that way it has no answer, because it depends on your workload, your peaks and your tolerance for slower agents.
The better question is what your real utilisation looks like, and what you cannot put on someone else’s API at any price. Data that is not allowed to leave the building, a stack nobody can rate-limit, prices that do not move under you mid-quarter. Those reasons survive contact with the spreadsheet. Cost alone usually does not.
Top comments (2)
The concurrency math gets even worse once you price in peak-to-trough developer activity.
Internal engineering clusters face severe intraday seasonality. To prevent queueing during the core four-hour overlap window, infrastructure teams have to provision for the local peak, while evening and weekend utilization drops near single digits. A frontier API pools uncorrelated enterprise demand across global time zones, absorbing that variance into a shared base load. On an internal cluster, you pay the full capital depreciation on idle VRAM across twenty-four hours.
The second hidden drag is context holding. Long-horizon coding agents accumulate large rolling contexts. Even when a developer is reading a diff or thinking, their allocated KV cache sits locked in high-bandwidth memory unless aggressively offloaded. Once you blend a twenty percent realized utilization rate with memory reservation for idle sessions, the effective amortization per resolved task easily outstrips paying uncached frontier API rates.
“Cost per resolved task” is the right unit because concurrency changes the capacity picture. I would publish a saturation curve too: resolved tasks, p95 latency, and failure rate as simultaneous agents increase. That makes it possible to see whether a cheaper model only looks economical while the queue is empty.