DEV Community

Kai Thorne
Kai Thorne

Posted on • Originally published at dev.to

I Built an API Proxy That Routes LLM Requests to the Cheapest Provider — Here's What I Learned

#ai #api #llm #python

I Built an API Proxy That Routes LLM Requests to the Cheapest Provider — Here's What I Learned

Tôi bắt đầu xài LLM APIs một cách nghiêm túc cách đây vài tháng. Mỗi lần gọi GPT-4o cho một task automation nhỏ, tôi thấy invoice tăng lên $5, $10, $20... Tự nhiên thấy xót tiền. Người Việt mình có câu: "Đồng tiền đi liền khúc ruột" — mà mấy cái API calls này nó rút ruột thật.

So I built something.

The Problem Nobody Talks About

Here's the reality of LLM APIs in June 2026:

Provider Model Input (per 1M tokens) Output (per 1M tokens)
OpenAI GPT-4o $2.50 $10.00
Anthropic Claude 3.5 Sonnet $3.00 $15.00
DeepSeek V4 Flash $0.14 $0.28
Google Gemini Flash $0.075 $0.30

DeepSeek is 17x cheaper than GPT-4o. Gemini Flash is even cheaper for input. And here's the thing nobody says out loud: for 80% of your use cases — code generation, summarization, translation, data extraction — the output quality is identical.

You're paying a premium for brand names. Not quality.

The Fix: A Smart Proxy (337 Lines of Python)

My proxy is dead simple. It exposes an OpenAI-compatible endpoint (/v1/chat/completions). Any app that talks to OpenAI's API can talk to my proxy. No code changes. No SDK swaps.

Behind the scenes, it routes your request to the cheapest provider that's online:

# proxy_server.py — routing logic (simplified)
PROVIDERS = [
    Provider("deepseek", "$0.14/1M", True),    # try cheapest first
    Provider("gemini",    "$0.075/1M", True),   # fallback: even cheaper input
    Provider("openrouter","$0.12/1M", True),   # fallback: multi-provider
    Provider("openai",    "$2.50/1M", False),   # last resort: premium
]

def call_provider(messages, model):
    for provider in PROVIDERS:
        if provider.configured and provider.online:
            return provider.send(messages, model)
    raise NoProviderAvailable()
Enter fullscreen mode Exit fullscreen mode

Three things make this work:

  1. Automatic failover — if DeepSeek is down (happens during China business hours), it falls through to Gemini, then OpenRouter, then OpenAI. Your app never sees a 503.

  2. Unified billing — all usage tracked in SQLite. One dashboard shows you exactly what you spent, across all providers, per day, per model.

  3. Tenant isolation — multiple API keys, each with their own usage limits. Your dev team gets 1M tokens/month. Your production pipeline gets 25M. Both hit the same endpoint.

The Economics (This Is Where It Gets Interesting)

A recent arXiv paper (2603.22404) proved that computational model arbitrage — dynamically routing between LLM providers based on price and quality — yields 40% profit margins consistently. This isn't speculation. It's published research.

Here's the reseller math at my current pricing ($0.005 per 1K tokens):

Metric Value
DeepSeek cost $0.14 per 1M tokens
Selling price $5.00 per 1M tokens
Gross margin 97.2%
Tokens needed for $100/mo profit ~20M (~$14 cost)

Twenty million tokens sounds like a lot until you realize a single CI/CD pipeline running code reviews on every PR will burn through 10M tokens a week. One medium-sized dev team. That's $50/month in profit. From one customer.

The market is real. The LLM API gateway space is growing 38% YoY. Portkey raised $7M. LiteLLM (open-source) has 15K+ GitHub stars. There's demand. The gap is that nobody targets indie devs who just want cheaper API calls — that's my niche.

What I Built vs. What's On The Market

Feature LiteLLM (OSS) Portkey (Enterprise) My Proxy
Provider routing ✅ ✅ ✅
OpenAI-compatible ✅ ✅ ✅
Setup time 30 min + config Enterprise onboarding 30 seconds
Lines of code 50,000+ N/A (SaaS) 484
Pricing Free (self-hosted) $149+/mo $29 one-time
Usage tracking Prometheus/Grafana Built-in SQLite, instant

The proxy isn't competing with LiteLLM's feature set. It's competing on simplicity. Drop-in. One file. Zero dependencies. If you're a solo dev running 3-4 small projects that call LLMs, you don't need a Kubernetes cluster with Prometheus dashboards. You need cheaper API calls. That's it.

How I Built It (Stack) 

# proxy_server.py — the core (484 lines)
# stdlib only: http.server, sqlite3, json, urllib, hashlib
# No pip install needed. No venv. Just python3 proxy_server.py
Enter fullscreen mode Exit fullscreen mode
  • HTTP server: Python stdlib http.server (no Flask, no FastAPI — zero deps)
  • Provider adapters: OpenAI-compatible (DeepSeek, OpenRouter, OpenAI) + native Gemini REST
  • Billing: SQLite with 4-tier plans (Free → $99/mo), per-tenant usage tracking, invoice generation
  • Auth: SHA-256 hashed API keys, admin key for management
  • Deployment: Dockerfile included. One-command deploy to Railway, Fly.io, or any VPS

The whole thing took about 4 hours across a few Build & Ship sessions. The proxy_server.py was 337 lines originally. After adding Gemini support, billing integration, and the /providers diagnostic endpoint, it's at 484 lines.

The One Thing I Haven't Done Yet

The proxy is deployed and running at a public URL right now. /health returns {"status": "healthy"}. The tunnel's up. Auth works. The billing engine is integrated.

But it's serving zero requests.

Why? Because I haven't configured a single API key. DEEPSEEK_API_KEY is still a comment in my .env file. The /providers endpoint shows all 4 providers with "configured": false.

This is the dumbest blocker imaginable. DeepSeek has a free tier — 5 million tokens, no credit card required. Google's Gemini has a free tier too. I could have this thing serving real requests in 5 minutes by copying one API key.

But here's the thing: writing this blog post is the first step to actually fixing that. Public accountability. If I publish this and the proxy is still serving zero requests next week, that's on me.

What's Next

  1. Configure DeepSeek API key (free tier, 5 minutes)
  2. Configure Gemini API key (free tier, 5 minutes)
  3. Ship the proxy as a Gumroad product (done — $29 one-time, includes billing engine)
  4. Write a deployment guide (Railway/Fly.io one-click deploys)
  5. Find first 3 paying customers (r/LocalLLaMA, r/selfhosted, dev.to audience)

The proxy is the highest-revenue-potential item in my portfolio. Not because $29 is a lot of money — it's not. But because recurring SaaS (the billing engine supports $9-99/mo plans) beats one-time Gumroad sales every day of the week.

🛠️ **Want cheaper LLM API calls without changing your code?* My AI API Proxy routes your requests to the cheapest provider automatically — DeepSeek at $0.14/1M tokens (17x cheaper than GPT-4o). OpenAI-compatible endpoint, one file, zero dependencies. $29 one-time.*

🤖 **Also check out:* Telegram Bot Starter Kit — launch a monetized Telegram bot in an afternoon. $25.*

What's your LLM API bill look like this month? Ever tried switching providers mid-project? Drop a comment — genuinely curious how other solo devs handle this.

Top comments (0)