I Built a Drop-in AI API Caching Proxy — Save 70% on Inference Costs

#ai #api #productivity #showdev

The TL;DR: A caching proxy for OpenAI/Anthropic/OpenRouter that sits between your code and the provider. Change one line — baseURL — and every duplicate request returns from cache at <1ms for $0. Pay 20% of what you save. Nothing if you save nothing.

→ Try it: https://cachepilot.koaw.workers.dev

Why I Built This

I was burning ~$300/mo on Claude API calls. I checked my usage:

70% of requests were identical — same system prompt, same few-shot examples, same context, repeated across sessions
Every single one paid full price ($3/M input for Sonnet)
I was paying ~$210/mo for the exact same response over and over

I threw together a simple cache. Bill dropped to $90.

Then I realized: every AI SaaS company has this problem. Every user session sends the same system prompt. Every background batch job re-requests identical completions. Multi-tenant apps repeat the same query across users.

Helicone, Portkey, Langfuse offer caching — but it's a sidebar feature. They charge per-request (even on cache hits!) and it's exact-match only.

So I built the opposite: a product where caching is the only feature, and we only make money when you save money.

The Architecture (Runs on Cloudflare Workers — Always On, $0)

Your App → cachepilot.koaw.workers.dev
  │
  ├─ L0: In-Memory LRU (30s TTL, nanosecond lookup)
  ├─ L1: Cloudflare Edge Cache (1h TTL, tenant-isolated)
  ├─ L1.5: Normalizer (whitespace-blind cache keys)
  └─ L3: Provider Optimization (prompt cache markers)
       │
       └─ OpenAI / Anthropic / OpenRouter

L0 — In-Memory: Hot responses in process memory with LRU eviction. Sub-millisecond for frequent duplicates.

L1 — Edge Cache: Cloudflare Cache API stores full responses at the edge. Tenant-isolated by hashed API key. Millisecond lookup.

L1.5 — Normalizer: "Hello" and " hello " produce the same cache key. Whitespace, key ordering, formatting differences are ignored.

L3 — Provider Optimization: Automatically injects Anthropic prompt caching markers on session boundaries. Detects multi-turn conversations without any developer configuration.

The "Session-Aware" Differentiator

This is what Helicone/Portkey don't do:

Turn 1: Cache miss → forwarded to provider
Turn 2: Only the new message is different. We detect it's a continuation and mark the stable prefix for prompt caching → 37-89% savings on input tokens
Turn 1 repeated: L0 cache HIT → zero cost

No headers. No SDK. No developer configuration.

Try It Right Now

Step 1: Go to cachepilot.koaw.workers.dev → enter your email → get an API key

Step 2: Change the baseURL in your AI client:

// Before
const openai = new OpenAI({ apiKey: "sk-..." });

// After
const openai = new OpenAI({
  baseURL: "https://cachepilot.koaw.workers.dev/v1",
  apiKey: "sk-cache-xxx-yyy",
});