Choosing the Optimal Hardware for Self-Hosted Coding Agents in 2026
If you want to run a coding agent on hardware you own instead of paying for an API, you have to pick a machine. Almost every guide ranks machines by tokens per second of generation. That turns out to be roughly the right number, but for a reason the guides rarely give, and it comes with one exception that will cost you real time.
I started this guide expecting the opposite. An agent sends an enormous prompt on every turn: the system prompt, your file tree, the files it just read, and the whole conversation so far, often 40,000 tokens or more. Against that, a 300-token reply looks like a rounding error. So reading the prompt should dominate, and you should buy for prompt-processing speed.
Two things make that wrong. Prompt caching means an agent does not re-read those 40,000 tokens on a normal turn, only the couple of thousand that changed. And reasoning models spend most of a turn writing thinking tokens, which is generation, not reading.
Where an agent turn actually spends its time
Running a model locally has two stages, and they are limited by different parts of the machine. Prefill is the model reading your prompt. It processes the whole prompt in one batch of matrix multiplications, so it is limited by raw compute. Decode is the model writing its reply, one token at a time. Each token requires reading the model’s active weights out of memory once, so it is limited by memory bandwidth. This is the tokens-per-second figure everyone quotes.
Prompt caching removes most of the prefill
Every serving stack worth using keeps the KV cache from the previous turn and processes only the part of the prompt that changed. vLLM documentation is explicit that prefix caching lets the new query skip the computation of the shared part. llama.cpp does the same with cache_prompt, on by default: the common prefix does not have to be re-processed, only the suffix that differs between the requests. SGLang’s RadixAttention is on by default too.
An agent appends to its transcript rather than rewriting it, so a normal turn prefills a few thousand new tokens instead of forty thousand. The effect is not marginal. Anthropic’s sample session reads 1.2k input, 5.3k output, 940.0k cache read, 50.0k cache write. Of roughly 991,000 tokens on the input side, 94.8% came from cache and were never processed again.
The four numbers that decide a build
- Capacity: Decides what you can load. Budget roughly 0.5 to 0.6 GB per billion total parameters at 4-bit.
- Bandwidth: Sets generation speed. Divide bandwidth by the bytes read per token, then take 60-80% of that for a realistic figure.
- Prefill compute: Sets how fast the model reads an uncached prompt.
- Concurrency: Decides how many agents run at once.
The comparison of hardware
| Machine | Price | Memory | Bandwidth | Verdict |
|---|---|---|---|---|
| Radeon AI PRO R9700 | $1,799 | 32 GB | 640 GB/s | Best value |
| RTX 5090 | $4,300 | 32 GB | ~1,792 GB/s | Fastest single card |
| Mac Studio M5 Max | $5,099 | 128 GB | 614 GB/s | Best balance |
Reach your model from anywhere with Pinggy
There are two cases where a machine on your desk needs a public URL. One is an editor like Cursor that will only talk to a publicly reachable endpoint. The other is you on a laptop, away from the workstation running the model. Pinggy gives you one over SSH, with no firewall rules, port forwarding or static IP:
ssh -p 443 -R0:localhost:8080 free.pinggy.io
This returns an HTTPS URL forwarding to localhost:8080, which you paste into the harness as its base URL.
Technical Considerations and Troubleshooting
When optimizing for coding agents, remember that the serving stack matters as much as the box. Ollama no longer has its own inference engine, serving via the upstream llama-server subprocess. For professional workflows, ensure you are using parameters like --cache-reuse to avoid accidental cold-cache performance hits.
Tool-call formats differ significantly per model family. If the server’s parser does not match the model, the harness receives raw XML as message text and the agent fails on its first tool call. Use --jinja, which is now the default in llama.cpp, or set --tool-call-parser explicitly.
Context truncation is a common silent failure. In several discussions, tool calling was broken across multiple providers because the server defaulted to a 4096-token window even though the models advertised much larger ones. Ensure you explicitly set environment variables or configuration flags like OLLAMA_CONTEXT_LENGTH=128000 to prevent early truncation of the system prompt and tool definitions.
Does it pay for itself?
Usually not in pure dollars. The good reasons to self-host are not about money. Electricity is cheap for this. At the US residential average of 18.34 cents per kWh, a single RTX 5090 tower under load eight hours a day costs about $32.56 a month. A Mac Studio is around $12.85, and roughly $1.20 a month if you leave it idling with a model resident, because Apple’s idle figure is 9W.
The hardware is the expensive part, and the right thing to compare it against is a subscription, not API list prices. Against a $200/month plan, a $5,000 machine takes about 30 months to break even, which is most of its useful life. The argument that does hold up follows from the caching section. Anthropic’s own sample session shows 940,000 of roughly 991,000 input-side tokens served from cache, and cache reads bill at 0.1x the input rate. Most of an agent's bill is paying to re-read context you already sent. On a machine you own, that re-reading is free, because the KV cache is already sitting in memory.
The other reasons are simpler: your code never leaves your network, and nobody changes your rate limits. If you are building high-volume automation, the cost savings of avoiding context re-processing at the API level can be significant, but for most individuals, self-hosting is about privacy, latency, and avoiding vendor lock-in.
Conclusion
Buy in this order: enough memory to hold the model, then generation speed, then prompt-processing speed. Prompt caching keeps reading off the critical path on every turn except the cold ones, and reasoning models spend most of a turn writing. Before you spend anything, check that prompt caching is working in your stack. It is worth more than the difference between most of the machines on this list.
Top comments (0)