DEV Community: DHPP

Saving 50-70% on Coding Agent Bills: Protecting Prompt Cache Affinity with VMR

DHPP — Tue, 18 Aug 2026 16:19:17 +0000

TL;DR — With per-token pricing now the default everywhere, the biggest line item in a coding-agent bill is usually not "smart model" but repetition bought at full price. Modern providers (Anthropic, DeepSeek, OpenAI) charge 10-20% for a cache hit on a repeated input prefix. The winning move is not making your prompts smaller — it's making sure your multi-turn session never leaves the upstream endpoint where its cache is warm. In our long-session load tests (150 req/s), pinning sessions with sticky routing took cache-hit rate from "luck" to 80-95% and cut the bill 50-70% (self-observed, not third-party audited).

The problem: your router is the reason your cache is cold

Here's the counterintuitive part. If you run a coding agent (Claude Code, Cursor, OpenClaw — anything with long multi-turn context), and you configured multiple API keys or providers for resilience, your gateway is very likely killing your cache on every round.

Prompt cache is isolated per upstream endpoint — per account, per physical node. When your load balancer round-robins request #1 to account A, #2 to account B, #3 back to A, every round looks like a brand-new session to every account. Nothing ever hits. Everything is billed at full price, including the 90% of tokens that were already computed in earlier rounds.

Worse: the classic gateway scheduling algorithms — smooth weighted round-robin, greedy quota balancing — are designed to spread traffic evenly. Cache locality wants the exact opposite: traffic pinned. In long sessions, "fair" and "cheap" are mutually exclusive.

The insight: cache economics beats token compression

Two routes dominate the "make agents cheaper" space, and they rest on opposite assumptions about how upstream billing works:

Local compression (e.g. OpenProxy's RTK): assumes upstream is stateless and bills everything full-price, so shrink the payload — re-encode tables, inject terse-output instructions, prune tool schemas. Claims 20-40% savings.
Cache-affinity protection (VMR): assumes upstream is stateful — that 90% of long-session cost is determined by prompt cache. So touch nothing, keep the byte prefix identical, and pin the session to one endpoint so the repeated prefix keeps hitting the discounted tier.

The conflict is structural: every compression "help" breaks the cache. Dynamic re-encoding, injected instructions, schema pruning — each change invalidates the prefix hash from token zero. In a 30-round session, compressing 30% of tokens doesn't compensate for re-paying the full price of 29 rounds of accumulated context. The mechanism-level conclusion: in cache-enabled 2026, compression can make you more expensive.

(To be fair: OpenProxy is a genuinely broader tool — OAuth subscription pooling for ChatGPT/Codex/Gemini, a real web UI, hedging and fusion scheduling, MCP/multimodal extensions. The critique above is specifically about RTK compression in long sessions, not the project as a whole.)

The fix: session-sticky routing, in two layers

VMR's sticky registry (internal/sticky/sticky.go) is an in-memory map from a session fingerprint to the endpoint that last served it. The fingerprint is a hash of system prompt + first user message — deliberately not the client session ID, because clients (especially agent frameworks) regenerate IDs on restart while the cache prefix is determined by content.

Routing has two layers, highest priority first:

Sticky Pin — if the fingerprint is in the registry and the endpoint is healthy, force the request back to it. This outranks all quota scheduling. Even if that account's quota is exhausted, we keep the session pinned and eat the overage, because switching mid-session zeroes the cache and the recompute cost of tens of rounds is an order of magnitude larger than a short overage.
New-session spread — only sessions with no sticky binding participate in scheduling. After priority sorting ties, assign greedily by quota slack. Notably, we do not use SWRR: it needs a persistent accumulator, and spreading traffic is itself anti-cache-locality. A fresh session picks an endpoint once; layer 1 holds it there.

Two engineering details worth stealing:

TTL is tiered. Global sticky_ttl defaults to 10m (fine for Anthropic/OpenAI memory caches), hard-capped at 24h by a memory-eviction backstop so the registry can't grow unbounded. Disk-cached providers like DeepSeek keep caches for hours-to-days — override per-endpoint (sticky_ttl: 2h) to match cache lifetime.
Expiry sweep is opportunistic, not a ticker goroutine — event-triggered and throttled, avoiding a global lock walk on every call.

What it looks like

models:
  coding:
    capabilities: [text, tools]
    max_context_tokens: 256000
    endpoints:
      - protocol: openai
        provider: deepseek
        models: [deepseek-v4-flash]
        sticky_ttl: 2h   # disk cache lives hours-to-days; match it
      - protocol: anthropic
        provider: anthropic
        models: [claude-sonnet-5]
        # sticky_ttl: 10m (default)

Client points at http://localhost:8080/v1 (or /v1/messages for the Anthropic protocol), model name = your virtual model. Same session, same endpoint, every round.

Same-session trace before → after (illustrative micro-numbers within realistic ranges):

before  req#0127  endpoint=deepseek-01   cache_hit=0    input=185K  cost=$0.41
before  req#0128  endpoint=anthropic-02  cache_hit=0    input=199K  cost=$1.86
before  req#0129  endpoint=deepseek-01   cache_hit=0    input=213K  cost=$0.47

after   req#0127  endpoint=deepseek-01   cache_hit=182K  input=185K  cost=$0.03
after   req#0128  endpoint=deepseek-01   cache_hit=195K  input=199K  cost=$0.04
after   req#0129  endpoint=deepseek-01   cache_hit=208K  input=213K  cost=$0.04

Pin the session, and the accumulated context stops being re-bought at full price.

Trade-offs we accepted

No global quota-optimal scheduling. A perfect scheduler that preserves cache and shuffles quota between sessions is possible in theory, but needs persistent state and prediction. For an individual developer, the complexity-to-bug ratio isn't worth it. Session-local lock + inter-session greedy is the pragmatic optimum.
No hedging. Firing the same request at two providers and taking the faster one adds latency insurance but doubles full-price consumption and breaks the cache. Negative expected value for personal use.
TTL granularity is coarse. 10m is conservative for very long sessions; we ship per-endpoint overrides instead of adaptive TTLs. Known future work.

A 3-step checklist for your setup

Check your cache-hit rate first. Look at cache_read vs fresh in your billing/status output. Below ~50% on long sessions means your config is leaking money — fix that before any other optimization.
Stop spreading sessions across keys. If your tool supports session affinity, turn it on. If not, assign one key per long session manually and let failover handle only real outages.
Keep the system prompt byte-stable. Dynamic injection (timestamps, random vars) invalidates the prefix on every request. And if you use a "compressing" proxy, know that it's trading cache hits for shorter payloads — usually a bad trade in long sessions.

Self-hosted, zero-database, one static binary (~15MB, Go). If that sounds like your kind of tool, VMR is open source at https://github.com/bigfatsea/vmr.

Data caveat: the 50-70% figure is from our own long-session load tests and project records, not third-party audited. Actual savings depend on provider cache pricing — DeepSeek-class disk caches give the biggest wins.

Why Your Coding Agent Doesn't Need an Electron App: A Minimalist LLM Proxy in Go

DHPP — Tue, 18 Aug 2026 08:21:35 +0000

TL;DR (the bottom line, up front)

Your coding agent (Claude Code, Cursor, Codex, ...) doesn't need a dashboard. It needs a transparent, self-healing, low-overhead pipe between its SDK and a pool of LLM providers — because at 3 AM, when the agent is running unattended, nobody is there to click a button.

That's why we built VMR: a local, single-binary LLM router in Go.

~31K lines of Go, 4 direct dependencies, ships as one static binary (~12–15MB)
Zero database, zero web UI, zero Node/Python runtime
Byte-faithful pass-through: requests arrive upstream byte-identical to a direct connection
Session-sticky cache affinity: keeps long conversations on the same endpoint, protecting upstream Prompt Cache (we observe 85–95% hit rates, ~50–70% cost reduction on cache discounts — self-measured, not third-party audited)
From our benchmark (Apple M1): routing overhead +0.3–0.9ms p50; RSS 15MB; cold start ~60ms

This post is the design reasoning behind the minimalist bet. No hype, just the decision chain.

1. The problem: agent traffic is server traffic, not desktop traffic

Two nights pushed us here.

Night one was the bill. After wiring Claude Code / Cursor to multiple providers (Anthropic, DeepSeek, MiniMax, OpenRouter...), I started with the naive approach — hardcode different Base URLs per provider. It fell apart fast. Agent sessions are long-context, high-frequency round-trips. The moment a subagent switch slightly perturbs the request body, the provider's Prompt Cache key misses, and the cost climbs visibly. I was bouncing between "one provider's bill" and "another provider's queue time."

Night two was the outage. A batch refactor was supposed to run for ~an hour unattended. An upstream key hit 429, the proxy wedged, and by morning the task had died with progress lost.

Both nights pointed the same direction: the proxy layer has to be designed for unattended operation — self-healing, self-retrying, and always leaving evidence — not for human dashboard-watching.

2. What we looked at (and why we walked away)

I'm not going to pretend we didn't consider "fuller" solutions. We did, and they're genuinely good at what they do:

LiteLLM-class translation gateways: huge coverage, but also huge complexity. The "all protocols → internal format → target protocol" translation layer is something you maintain forever as upstream event formats evolve. For a solo maintainer, that's a real tax.
Electron/desktop control planes (e.g. Claude Code Router): deep, well-made — agent profiles, tool markets, Web UI. For our use case it was a drinking fountain built to get a glass of water. We wanted the invisible, non-invasive layer only.
Hardcoded Base URLs + manual switching: simplest, but no failover, no health probing, no audit trail. At 3 AM nobody is switching anything manually.

They serve different audiences. The question we kept asking, which became VMR's standing review rule:

Every new feature, ask: does this add capability, or does it add complexity?

By that test, dashboards, user management, billing, prompt management, plugin systems, and MCP frameworks all got rejected at the door. Useful? Yes. Complexity-adders for the "be a transparent base for coding agents" mission? Also yes.

3. Three constraints that pushed us to Go + a single binary

Not "Go for the sake of Go." Three constraints stacked:

1. Unattended = server-traffic semantics. High concurrency, long sessions, zero supervision. The proxy must be low-latency, high-throughput, and self-healing. This workload is closer to a reverse proxy / load balancer than a desktop GUI. Go's goroutine model, its server-grade GC behavior, and single-file cross-compilation make it the cheapest fit.

2. Byte-level fidelity keeps you compatible with upstream. LLM providers ship new params and event types fast. If your proxy does JSON.parse → mutate → stringify on every request, any unknown field is theoretically at risk of being reordered, escaped, or dropped. The only way to make "new upstream features work the day they ship, without waiting for a VMR release" is to not touch any byte we don't need to touch. Go's json.RawMessage makes this clean.

3. Solo maintenance = fewer dependencies = safer. Four Go module deps vs a 50+ npm package tree + Electron + React. Supply-chain surface, audit cost, build time — all orders of magnitude apart. For a solo project, "I can write this with only the standard library" is a survival advantage.

4. Byte-faithful pass-through (not "good-enough" pass-through)

"Byte-faithful" is not marketing copy in VMR. Three concrete mechanisms:

// 1) json.RawMessage: keep the whole request body raw; parse only model/stream
type CanonicalRequest struct {
    Model  string
    Stream bool
    Raw    json.RawMessage // every other byte passes through untouched
    Header http.Header
    Facts  RequestFacts
}

// 2) MarshalNoEscape: refuse Go's default HTML escaping.
// Default json.Marshal turns "<div>" into "\u003Cdiv\u003E" — semantically
// equal, byte-different. VMR cares about bytes.
func MarshalNoEscape(v any) ([]byte, error) {
    var buf bytes.Buffer
    enc := json.NewEncoder(&buf)
    enc.SetEscapeHTML(false)
    enc.Encode(v)
    return bytes.TrimSuffix(buf.Bytes(), []byte("\n")), nil
}

// 3) Byte-level model rewrite: scan bytes, replace the "model" key's value.
// No JSON unmarshal, no object round-trip. Key order, whitespace, unknown
// fields stay exactly as the client sent them.
func RewriteModel(body []byte, realModel string) []byte { /* byte-level state machine */ }

VMR never unpacks the request body into an object. json.RawMessage is a zero-copy byte container; the model substitution is pure string manipulation. That's why go.mod stays at four deps:

github.com/fsnotify/fsnotify  → config hot-reload
github.com/klauspost/compress → zstd audit-log compression
golang.org/x/image            → image downscale
gopkg.in/yaml.v3              → config parsing

No ORM, no web framework, no DI container, no logging library. HTTP uses net/http; logging uses log. ~31K lines of Go, one static binary.

5. Why we refuse cross-protocol translation

VMR exposes both an OpenAI and an Anthropic entrance, but routes each within its own protocol family — it never translates between them. This is a stance, not a gap:

SSE event semantics can't be losslessly mapped. Anthropic's message_start/content_block_delta/tool_use stream vs OpenAI's choices[0].delta.tool_calls[] chunks — every new event type upstream adds is another patch to the translation layer.
Forward compatibility breaks. New upstream event type → translation layer doesn't know it → dropped or error. Byte pass-through is natively forward-compatible.
Mainstream providers already expose both faces. MiniMax, DeepSeek, OpenRouter offer OpenAI and Anthropic compatible surfaces. The translation layer no longer creates value.

We could do it — we've tested both protocol faces against real providers. We just think the complexity isn't worth it. (This is also the crux of the difference from LiteLLM-class gateways, where the translation layer is a core feature.)

6. Agent-first, not human-first

Two design decisions that only make sense if the user is an agent, not a person:

Session-Sticky affinity: keep a long conversation pinned to one physical endpoint so the provider's Prompt Cache stays warm. Observed hit rates 85–95%, ~50–70% cost reduction on cache discounts (self-measured).
Error-class-aware cooldowns: rate-limit (429) honors Retry-After; auth failure (401) gets a long cooldown; content-filter rejection (403) switches provider without penalizing the endpoint — the provider is healthy, it's just doing its compliance job; 5xx gets exponential backoff. Most gateways treat all failures the same. VMR doesn't.

Plus the flight recorder: per-request JSONL audit (client↔VMR, VMR↔upstream) grouped by Session → Task → Turn — built to answer "which model did this agent session use, for how many tokens, and where did it go off the rails," not "what does this chat look like in a table."

7. Honest tradeoffs

No dashboard, no web UI. Observability is vmr report / vmr story on the CLI. Deliberate, but it does filter out users who want a graphical view.
Providers are config, not built-in. Currently validated: MiniMax, DeepSeek, OpenRouter. Adding one is near-zero code (a config block), but you verify compatibility yourself.
Strategy dimensions are partial. priority works; weight/round_robin are "in progress."
Two known architectural tradeoffs: image downscaling happens at the server layer (can't be un-done after failover), and the concurrency gate is global, not per-model.
The real one: the project is young and small (community, iteration velocity are nothing like CCR's). If you pick it, accept you may write your own provider configs and verify compatibility. The upside: 4 deps and 31K lines mean even if development stalled, handover cost is low.

8. Try it

If you're running your own coding agents and want a transparent, self-healing, evidence-keeping data plane between them and your LLM providers, the code is on GitHub: github.com/bigfatsea/vmr. We're actively working on the scheduling dimensions (weight/round_robin) and the vmr story diff-and-replay tooling. Issues and PRs welcome.