Saving 50-70% on Coding Agent Bills: Protecting Prompt Cache Affinity with VMR

#ai #agents #openclaw #apigateway

TL;DR — With per-token pricing now the default everywhere, the biggest line item in a coding-agent bill is usually not "smart model" but repetition bought at full price. Modern providers (Anthropic, DeepSeek, OpenAI) charge 10-20% for a cache hit on a repeated input prefix. The winning move is not making your prompts smaller — it's making sure your multi-turn session never leaves the upstream endpoint where its cache is warm. In our long-session load tests (150 req/s), pinning sessions with sticky routing took cache-hit rate from "luck" to 80-95% and cut the bill 50-70% (self-observed, not third-party audited).

The problem: your router is the reason your cache is cold

Here's the counterintuitive part. If you run a coding agent (Claude Code, Cursor, OpenClaw — anything with long multi-turn context), and you configured multiple API keys or providers for resilience, your gateway is very likely killing your cache on every round.

Prompt cache is isolated per upstream endpoint — per account, per physical node. When your load balancer round-robins request #1 to account A, #2 to account B, #3 back to A, every round looks like a brand-new session to every account. Nothing ever hits. Everything is billed at full price, including the 90% of tokens that were already computed in earlier rounds.

Worse: the classic gateway scheduling algorithms — smooth weighted round-robin, greedy quota balancing — are designed to spread traffic evenly. Cache locality wants the exact opposite: traffic pinned. In long sessions, "fair" and "cheap" are mutually exclusive.

The insight: cache economics beats token compression

Two routes dominate the "make agents cheaper" space, and they rest on opposite assumptions about how upstream billing works:

Local compression (e.g. OpenProxy's RTK): assumes upstream is stateless and bills everything full-price, so shrink the payload — re-encode tables, inject terse-output instructions, prune tool schemas. Claims 20-40% savings.
Cache-affinity protection (VMR): assumes upstream is stateful — that 90% of long-session cost is determined by prompt cache. So touch nothing, keep the byte prefix identical, and pin the session to one endpoint so the repeated prefix keeps hitting the discounted tier.

The conflict is structural: every compression "help" breaks the cache. Dynamic re-encoding, injected instructions, schema pruning — each change invalidates the prefix hash from token zero. In a 30-round session, compressing 30% of tokens doesn't compensate for re-paying the full price of 29 rounds of accumulated context. The mechanism-level conclusion: in cache-enabled 2026, compression can make you more expensive.

(To be fair: OpenProxy is a genuinely broader tool — OAuth subscription pooling for ChatGPT/Codex/Gemini, a real web UI, hedging and fusion scheduling, MCP/multimodal extensions. The critique above is specifically about RTK compression in long sessions, not the project as a whole.)

The fix: session-sticky routing, in two layers

VMR's sticky registry (internal/sticky/sticky.go) is an in-memory map from a session fingerprint to the endpoint that last served it. The fingerprint is a hash of system prompt + first user message — deliberately not the client session ID, because clients (especially agent frameworks) regenerate IDs on restart while the cache prefix is determined by content.

Routing has two layers, highest priority first:

Sticky Pin — if the fingerprint is in the registry and the endpoint is healthy, force the request back to it. This outranks all quota scheduling. Even if that account's quota is exhausted, we keep the session pinned and eat the overage, because switching mid-session zeroes the cache and the recompute cost of tens of rounds is an order of magnitude larger than a short overage.
New-session spread — only sessions with no sticky binding participate in scheduling. After priority sorting ties, assign greedily by quota slack. Notably, we do not use SWRR: it needs a persistent accumulator, and spreading traffic is itself anti-cache-locality. A fresh session picks an endpoint once; layer 1 holds it there.

Two engineering details worth stealing:

TTL is tiered. Global sticky_ttl defaults to 10m (fine for Anthropic/OpenAI memory caches), hard-capped at 24h by a memory-eviction backstop so the registry can't grow unbounded. Disk-cached providers like DeepSeek keep caches for hours-to-days — override per-endpoint (sticky_ttl: 2h) to match cache lifetime.
Expiry sweep is opportunistic, not a ticker goroutine — event-triggered and throttled, avoiding a global lock walk on every call.

What it looks like

models:
  coding:
    capabilities: [text, tools]
    max_context_tokens: 256000
    endpoints:
      - protocol: openai
        provider: deepseek
        models: [deepseek-v4-flash]
        sticky_ttl: 2h   # disk cache lives hours-to-days; match it
      - protocol: anthropic
        provider: anthropic
        models: [claude-sonnet-5]
        # sticky_ttl: 10m (default)

Client points at http://localhost:8080/v1 (or /v1/messages for the Anthropic protocol), model name = your virtual model. Same session, same endpoint, every round.

Same-session trace before → after (illustrative micro-numbers within realistic ranges):

before  req#0127  endpoint=deepseek-01   cache_hit=0    input=185K  cost=$0.41
before  req#0128  endpoint=anthropic-02  cache_hit=0    input=199K  cost=$1.86
before  req#0129  endpoint=deepseek-01   cache_hit=0    input=213K  cost=$0.47

after   req#0127  endpoint=deepseek-01   cache_hit=182K  input=185K  cost=$0.03
after   req#0128  endpoint=deepseek-01   cache_hit=195K  input=199K  cost=$0.04
after   req#0129  endpoint=deepseek-01   cache_hit=208K  input=213K  cost=$0.04

Pin the session, and the accumulated context stops being re-bought at full price.

Trade-offs we accepted

No global quota-optimal scheduling. A perfect scheduler that preserves cache and shuffles quota between sessions is possible in theory, but needs persistent state and prediction. For an individual developer, the complexity-to-bug ratio isn't worth it. Session-local lock + inter-session greedy is the pragmatic optimum.
No hedging. Firing the same request at two providers and taking the faster one adds latency insurance but doubles full-price consumption and breaks the cache. Negative expected value for personal use.
TTL granularity is coarse. 10m is conservative for very long sessions; we ship per-endpoint overrides instead of adaptive TTLs. Known future work.

A 3-step checklist for your setup

Check your cache-hit rate first. Look at cache_read vs fresh in your billing/status output. Below ~50% on long sessions means your config is leaking money — fix that before any other optimization.
Stop spreading sessions across keys. If your tool supports session affinity, turn it on. If not, assign one key per long session manually and let failover handle only real outages.
Keep the system prompt byte-stable. Dynamic injection (timestamps, random vars) invalidates the prefix on every request. And if you use a "compressing" proxy, know that it's trading cache hits for shorter payloads — usually a bad trade in long sessions.

Self-hosted, zero-database, one static binary (~15MB, Go). If that sounds like your kind of tool, VMR is open source at https://github.com/bigfatsea/vmr.

Data caveat: the 50-70% figure is from our own long-session load tests and project records, not third-party audited. Actual savings depend on provider cache pricing — DeepSeek-class disk caches give the biggest wins.

DEV Community