I Built a Proxy That Cut My AI Agent's API Costs by 90% — Here's How

Khalid A — Tue, 14 Jul 2026 18:34:09 +0000

TL;DR: LeastGen is a self-hosted caching proxy that sits between your AI agents and LLM APIs. It caches repetitive prompts (tool calls, code reviews, etc.) in a local SQLite database and serves cached responses instantly. One benchmark run went from 1,860 seconds to 0.08 seconds with zero tokens consumed. Same 30 problems. Same outputs. Just 23,000x faster and effectively free.

Try it out: leastgen.com

The Problem: Your AI Agent Is Bleeding Money

If you've built an AI agent that does anything non-trivial, you know the pain.

Your agent makes the same tool calls over and over. It re-analyzes the same code. It re-generates the same validation logic. Every single time, you pay for the tokens. Every single time, you wait for the round-trip.

I was running a moderately complex agent workflow — nothing crazy, just some code review tasks, tool orchestration, and iterative refinement. My API bills were climbing, and my development loop was painfully slow.

The worst part? Most of those calls were identical. Same prompts. Same context. Same responses. I was paying OpenAI (or Anthropic) to do the same work repeatedly.

That's when I built LeastGen.

The Solution: A Transparent Caching Proxy

LeastGen is a lightweight proxy that sits between your AI agent and your LLM provider. Here's what it does:

Intercepts every request — It acts as a drop-in replacement for your API endpoint.
Hashes the prompt — Creates a unique fingerprint of the request (including system prompt, user message, and parameters).
Checks the cache — If it's seen this exact prompt before, it serves the cached response from a local SQLite database.
Misses go upstream — If it's a new prompt, it forwards to the real API, caches the response, and returns it to your agent.

Zero code changes. Just point your agent to the proxy instead of the API directly.

The Numbers That Made Me Rethink Everything

I ran a simple benchmark: 30 coding problems, each requiring tool calls and iterative refinement.

Cold run (no cache):

⏱️ 1,860 seconds (~31 minutes)
📊 360,836 tokens passed to the cloud

Hot run (cache populated):

⏱️ 0.08 seconds
📊 0 tokens consumed

Same 30 problems. Same outputs. 23,000x faster. Effectively free.

Metric	Cold Run	Hot Run	Improvement
Time	1,860s	0.08s	23,250x
Tokens	360,836	0	100% reduction

This isn't just about saving money. It's about unlocking new workflows. When your agent can iterate in milliseconds instead of minutes, everything changes.

How It Works Under the Hood

LeastGen is deliberately simple:

Local SQLite database — No external dependencies, no cloud services. Everything runs on your machine.
Prompt hashing — Uses a deterministic hash of the full request to identify duplicates.
Transparent proxy — Your agent doesn't know the difference. It just sees faster responses.

The architecture is intentionally minimal. No complex routing. No semantic similarity matching. Just exact-match caching for deterministic, repeatable prompts.

Why exact match? Because in agent workflows, tool calls and code reviews are often byte-for-byte identical across runs. Caching these with semantic matching would introduce latency and complexity. Exact matching is fast, simple, and catches the 80% case that matters most.

Who Is This For?

AI agent developers — If you're building with LangChain, CrewAI, or custom agents, this will cut your dev loop costs dramatically.
Teams doing iterative testing — Run the same test suite 50 times? Pay for it once.
CI/CD pipelines — Cache API responses in your build process and watch your pipeline times drop.
Anyone tired of paying for the same work twice

Getting Started

pip install leastgen

That's it. Set your agent's API base URL to the proxy address, and you're done.

The proxy handles all the caching logic automatically. You can inspect the SQLite database directly if you want to see what's being cached.

What's Next

LeastGen is early, but the core is solid. Here's what I'm working on:

Semantic caching — Matching similar (not just identical) prompts
Cache invalidation rules — TTLs and manual purge
Analytics dashboard — See exactly how much you're saving
Team sharing — Share cache across a development team

The Source Code

Important note: LeastGen is currently in private beta. The MIT license on the website reflects our intention to open-source the project, but we're holding the code back temporarily while we polish the experience and ensure stability.

If you want early access or want to contribute, reach out. I'm prioritizing developers who are actively building agents and can provide real feedback.

Why I Built This

I was frustrated. I was paying too much. I was waiting too long. And I knew there had to be a better way.

Caching is not a new idea. CDNs have been doing it for decades. But for some reason, the AI community has been slow to adopt it for agent workflows.

LeastGen is my attempt to fix that. It's not magic. It's just a proxy with a SQLite database. But it saves me time and money every single day.

I hope it saves you some too.