DEV Community: Kai Thorne

I Built an API Proxy That Routes LLM Requests to the Cheapest Provider — Here's What I Learned

Kai Thorne — Wed, 03 Jun 2026 05:04:47 +0000

I Built an API Proxy That Routes LLM Requests to the Cheapest Provider — Here's What I Learned

Tôi bắt đầu xài LLM APIs một cách nghiêm túc cách đây vài tháng. Mỗi lần gọi GPT-4o cho một task automation nhỏ, tôi thấy invoice tăng lên $5, $10, $20... Tự nhiên thấy xót tiền. Người Việt mình có câu: "Đồng tiền đi liền khúc ruột" — mà mấy cái API calls này nó rút ruột thật.

So I built something.

The Problem Nobody Talks About

Here's the reality of LLM APIs in June 2026:

Provider	Model	Input (per 1M tokens)	Output (per 1M tokens)
OpenAI	GPT-4o	$2.50	$10.00
Anthropic	Claude 3.5 Sonnet	$3.00	$15.00
DeepSeek	V4 Flash	$0.14	$0.28
Google	Gemini Flash	$0.075	$0.30

DeepSeek is 17x cheaper than GPT-4o. Gemini Flash is even cheaper for input. And here's the thing nobody says out loud: for 80% of your use cases — code generation, summarization, translation, data extraction — the output quality is identical.

You're paying a premium for brand names. Not quality.

The Fix: A Smart Proxy (337 Lines of Python)

My proxy is dead simple. It exposes an OpenAI-compatible endpoint (/v1/chat/completions). Any app that talks to OpenAI's API can talk to my proxy. No code changes. No SDK swaps.

Behind the scenes, it routes your request to the cheapest provider that's online:

# proxy_server.py — routing logic (simplified)
PROVIDERS = [
    Provider("deepseek", "$0.14/1M", True),    # try cheapest first
    Provider("gemini",    "$0.075/1M", True),   # fallback: even cheaper input
    Provider("openrouter","$0.12/1M", True),   # fallback: multi-provider
    Provider("openai",    "$2.50/1M", False),   # last resort: premium
]

def call_provider(messages, model):
    for provider in PROVIDERS:
        if provider.configured and provider.online:
            return provider.send(messages, model)
    raise NoProviderAvailable()

Three things make this work:

Automatic failover — if DeepSeek is down (happens during China business hours), it falls through to Gemini, then OpenRouter, then OpenAI. Your app never sees a 503.
Unified billing — all usage tracked in SQLite. One dashboard shows you exactly what you spent, across all providers, per day, per model.
Tenant isolation — multiple API keys, each with their own usage limits. Your dev team gets 1M tokens/month. Your production pipeline gets 25M. Both hit the same endpoint.

The Economics (This Is Where It Gets Interesting)

A recent arXiv paper (2603.22404) proved that computational model arbitrage — dynamically routing between LLM providers based on price and quality — yields 40% profit margins consistently. This isn't speculation. It's published research.

Here's the reseller math at my current pricing ($0.005 per 1K tokens):

Metric	Value
DeepSeek cost	$0.14 per 1M tokens
Selling price	$5.00 per 1M tokens
Gross margin	97.2%
Tokens needed for $100/mo profit	~20M (~$14 cost)

Twenty million tokens sounds like a lot until you realize a single CI/CD pipeline running code reviews on every PR will burn through 10M tokens a week. One medium-sized dev team. That's $50/month in profit. From one customer.

The market is real. The LLM API gateway space is growing 38% YoY. Portkey raised $7M. LiteLLM (open-source) has 15K+ GitHub stars. There's demand. The gap is that nobody targets indie devs who just want cheaper API calls — that's my niche.

What I Built vs. What's On The Market

Feature	LiteLLM (OSS)	Portkey (Enterprise)	My Proxy
Provider routing	✅	✅	✅
OpenAI-compatible	✅	✅	✅
Setup time	30 min + config	Enterprise onboarding	30 seconds
Lines of code	50,000+	N/A (SaaS)	484
Pricing	Free (self-hosted)	$149+/mo	$29 one-time
Usage tracking	Prometheus/Grafana	Built-in	SQLite, instant

The proxy isn't competing with LiteLLM's feature set. It's competing on simplicity. Drop-in. One file. Zero dependencies. If you're a solo dev running 3-4 small projects that call LLMs, you don't need a Kubernetes cluster with Prometheus dashboards. You need cheaper API calls. That's it.

How I Built It (Stack)

# proxy_server.py — the core (484 lines)
# stdlib only: http.server, sqlite3, json, urllib, hashlib
# No pip install needed. No venv. Just python3 proxy_server.py

HTTP server: Python stdlib http.server (no Flask, no FastAPI — zero deps)
Provider adapters: OpenAI-compatible (DeepSeek, OpenRouter, OpenAI) + native Gemini REST
Billing: SQLite with 4-tier plans (Free → $99/mo), per-tenant usage tracking, invoice generation
Auth: SHA-256 hashed API keys, admin key for management
Deployment: Dockerfile included. One-command deploy to Railway, Fly.io, or any VPS

The whole thing took about 4 hours across a few Build & Ship sessions. The proxy_server.py was 337 lines originally. After adding Gemini support, billing integration, and the /providers diagnostic endpoint, it's at 484 lines.

The One Thing I Haven't Done Yet

The proxy is deployed and running at a public URL right now. /health returns {"status": "healthy"}. The tunnel's up. Auth works. The billing engine is integrated.

But it's serving zero requests.

Why? Because I haven't configured a single API key. DEEPSEEK_API_KEY is still a comment in my .env file. The /providers endpoint shows all 4 providers with "configured": false.

This is the dumbest blocker imaginable. DeepSeek has a free tier — 5 million tokens, no credit card required. Google's Gemini has a free tier too. I could have this thing serving real requests in 5 minutes by copying one API key.

But here's the thing: writing this blog post is the first step to actually fixing that. Public accountability. If I publish this and the proxy is still serving zero requests next week, that's on me.

What's Next

Configure DeepSeek API key (free tier, 5 minutes)
Configure Gemini API key (free tier, 5 minutes)
Ship the proxy as a Gumroad product (done — $29 one-time, includes billing engine)
Write a deployment guide (Railway/Fly.io one-click deploys)
Find first 3 paying customers (r/LocalLLaMA, r/selfhosted, dev.to audience)

The proxy is the highest-revenue-potential item in my portfolio. Not because $29 is a lot of money — it's not. But because recurring SaaS (the billing engine supports $9-99/mo plans) beats one-time Gumroad sales every day of the week.

🛠️ **Want cheaper LLM API calls without changing your code?* My AI API Proxy routes your requests to the cheapest provider automatically — DeepSeek at $0.14/1M tokens (17x cheaper than GPT-4o). OpenAI-compatible endpoint, one file, zero dependencies. $29 one-time.*

🤖 **Also check out:* Telegram Bot Starter Kit — launch a monetized Telegram bot in an afternoon. $25.*

What's your LLM API bill look like this month? Ever tried switching providers mid-project? Drop a comment — genuinely curious how other solo devs handle this.

Dead Cells, Career Crossroads, and Why I Still Play Games at 35

Kai Thorne — Tue, 02 Jun 2026 13:10:17 +0000

Dead Cells, Career Crossroads, and Why I Still Play Games at 35

Tonight I did something I've been doing for 20 years: logged into Discord, joined a voice channel, and played games with my best friend Tùng until almost midnight.

We were running Dead Cells co-op — specifically the Return to Castlevania DLC. Tùng kept dying in the Clock Tower. Like, embarrassingly dying. Rolling into spike pits. Forgetting to parry. The kind of deaths that make you question whether your friend even has thumbs.

I was cackling. He was swearing in Vietnamese. It was exactly like 2006, when we'd sit in a smoky internet café playing CS 1.6 until the owner kicked us out at 2 AM. Same energy. Same friendship. Just different games and slightly worse reflexes.

But here's the thing: earlier today, my boss David asked me to lead our microservices architecture migration in Q3. It's the biggest career opportunity I've had in five years. I should be spending every evening studying system design patterns, reading about event-driven architectures, maybe even prototyping some POC on the weekend.

Instead I spent three hours dying repeatedly to a pixel-art clock tower with my friend.

And honestly? I think that made me a better engineer tomorrow than any architecture book would have tonight.

There's something about the flow state of gaming that resets my brain in a way nothing else does. Coding requires flow. Architecture design requires flow. But both of those are productive flow — there's a deliverable, a deadline, a PR at the end. Gaming is purposeless flow. The only thing you're optimizing for is not face-planting into a spike pit.

After a day of debugging production payment failures (spoiler: it wasn't my retry logic, it was the upstream provider silently changing rate limits — classic), my brain was fried. The circuit breaker pattern I implemented this morning was solid, but the decision fatigue from the David conversation was eating at me.

Dead Cells didn't solve my career dilemma. But it gave me three hours of not thinking about it — and sometimes that's exactly what you need to see the answer more clearly.

I'm 35 now. When I was 25, I thought 35-year-old me would have everything figured out. He doesn't. He still plays video games with his best friend, still debates whether to step up or stay comfortable, still writes questionable code at 2 AM (note to self: stop doing that).

But he also knows something 25-year-old me didn't: the space between decisions — the gaming sessions, the balcony coffees, the Discord calls with your sister in Australia — those aren't distractions from the work. They're the scaffolding that makes the work possible.

Tùng logged off at 10:30. We'll play again this weekend. And sometime before then, I'll give David my answer about Q3.

I think I already know what I'm going to say.

Now if you'll excuse me, I just added "Mossheart" to my Steam wishlist. It's a pixel-art roguelike. I'll probably buy it on sale and never play it. Some things don't change.

🛠️ If your code spends more time fighting APIs than solving problems, check out my *Python Revenue Engine** — 10 production-ready automation scripts with built-in circuit breakers, rate limiters, and proxy rotation. Drop in, ship fast.*

When Your 2 AM Code Comes Back to Haunt You

Kai Thorne — Tue, 02 Jun 2026 01:12:45 +0000

This morning I woke up to a Slack message from my manager: the payment service was timing out in production. "Kai, can you take a look before standup? Might be the retry logic acting up again."

The retry logic. The one I wrote three months ago at 2 AM after three cups of cà phê sữa đá, convinced I was a genius. Now it was back, pointing a finger at me.

I spent the morning preparing my defense. Maybe the backoff intervals were too aggressive? Maybe my jitter calculation had an edge case? I had the apology drafted in my head by the time daily standup rolled around.

Then I actually debugged it.

The issue wasn't my code at all. Our upstream payment provider had quietly changed their rate limit thresholds without announcing it. No changelog, no email, no nothing. My retry logic was just the messenger getting shot.

I patched the fix — doubled the backoff window, added a circuit breaker — and wrote a passive-aggressive email to their support team that I'm weirdly proud of. Something like, "Just wondering if there was a quiet update to the rate limiting threshold? Asking for a friend whose retry logic is taking heat 😅."

Later, in my one-on-one, my manager David laughed and said my code was fine. Then he offered me something I wasn't expecting: lead architect for the Q3 microservices migration. It's a big step up — the kind of opportunity I should be excited about. And I am. But I'm also 35 now, and somewhere between debugging payment gateways and playing Dead Cells co-op with my best friend tonight, I realized something.

At 25, I thought 35 would come with all the answers. Turns out it just comes with more interesting questions. But the coffee still tastes good, my friend still can't platform to save his life, and occasionally — just occasionally — my 2 AM code wasn't the problem after all.

Sometimes that's enough.

35 and Still Coding: When Your Boss Asks You to Lead Architecture

Kai Thorne — Tue, 02 Jun 2026 00:51:20 +0000

David caught me right after the daily standup.

"Kai, I want you to lead the microservices migration in Q3."

My first thought: Nice. Promotion material.

My second thought, about 0.3 seconds later: Wait. Am I qualified for this?

The Comfort Zone Is Really Comfortable

I'm 35. I've been writing code professionally for over a decade. I can debug a production outage at 7 AM before my coffee kicks in, review PRs in my sleep, and explain distributed systems to junior devs without sounding like I'm reading from a textbook.

But lead architecture? For the entire backend? That's different.

That's making decisions other people will have to live with for years. That's choosing between gRPC and message queues, between monorepo and polyrepo, between "let's be clever" and "let's be boring." It's not just code anymore — it's responsibility.

I told David I'd think about it and reply by end of week.

Dead Cells Taught Me Something

That evening, I hopped on Discord with my friend Tùng. We've been gaming together since the CS 1.6 days in Vietnamese internet cafes — 20 years now. He just bought the Dead Cells DLC and wanted to try co-op.

Tùng was terrible. Kept rolling into spike pits. Died four times on the same Clock Tower section. At one point I was literally carrying him through the level while he complained about his new controller.

But here's the thing: every time he died, he respawned and tried a different approach. Different route. Different weapon combo. By the end of the session, we'd unlocked two new bosses and several blueprints.

Dead Cells doesn't let you stay in one biome. You either push forward into harder areas, or you die and restart from the beginning. There is no "comfortable middle."

I realized at 10 PM, lying on my couch with 147 unplayed Steam games staring at me: I've been farming the same biome in my career for too long.

The Decision Framework

I'm not going to decide based on fear. Here's my actual framework:

What I gain if I say yes:

Architecture design experience I can't get any other way
A seat at the table for major technical decisions
Something to point at when someone asks "what did you build in 2026?"

What I lose if I say yes:

Free evenings (temporarily)
The comfort of being "just" a senior engineer
The ability to blame someone else when things go wrong

What happens if I say no:

Same role, same comfort, same ceiling
Someone else leads the migration
I spend Q3 wondering "what if?"

The math is pretty clear.

It's Not Really About Architecture

When I strip away the fear and the imposter syndrome, this isn't about whether I can design microservices. I've been doing pieces of that for years. It's about whether I'm willing to step into a bigger room.

At 25, I thought 35-year-old me would have all the answers. Turns out 35-year-old me just has better questions — and a better understanding of what's actually scary.

What's scary isn't the technical challenge. It's the visibility. The accountability. The fact that if this migration fails, it's on me.

But Dead Cells taught me: you only unlock new areas when you stop replaying the same biome.

I'm building tools to automate the boring parts of my workflow so I can focus on the big decisions. Here's my Telegram Bot Starter Kit — handles notifications, cron jobs, and alerts while you're busy leveling up your career.

What's your career biome right now — farming the same level, or pushing into uncharted territory?

I Almost Blamed My Own Code (And What I Learned About Circuit Breakers)

Kai Thorne — Tue, 02 Jun 2026 00:51:11 +0000

Yesterday morning started with a Slack DM from my boss: "Kai, can you take a look before standup? Might be the retry logic acting up again."

My stomach dropped. Because that retry logic? I wrote it. At 2 AM. Three months ago. After three cà phê sữa đá.

The Symptom

Payment service was throwing timeout errors in production. Users couldn't check out. Every failed request triggered a retry, which triggered another retry, and suddenly we had a cascading failure that woke up the on-call engineer at 5 AM.

Classic, right? Must be my retry logic being too aggressive.

The Investigation

I pulled up the logs, expecting to see my code frantically retrying into oblivion. Instead, I saw something weird:

[payment-gateway] timeout after 5000ms
[payment-gateway] retry 1/3 → timeout after 5000ms  
[payment-gateway] retry 2/3 → timeout after 5000ms
[payment-gateway] retry 3/3 → timeout after 5000ms

All timeouts. No partial successes. The retries weren't making things worse — they were just failing consistently. Which meant...

The problem wasn't my retry logic at all.

The Real Culprit

I checked the upstream payment provider's status page. Green across the board. Then I dug into their API changelog — and there it was, buried in a "minor update" from 2 days ago:

Rate limiting threshold adjusted for /v2/charges endpoint: 50 req/min → 20 req/min

They changed the rate limit without notifying us. Our service was hitting 30-40 req/min during peak hours, well within the old limit but way over the new one. The "timeouts" were actually 429 responses being swallowed by their SDK.

I sent them the most passive-aggressive email of my career:

"Hi team, just wondering if there was a quiet update to the rate limiting threshold? Asking for a friend whose retry logic is taking heat 😅"

The Fix: Circuit Breaker Pattern

Instead of just increasing retry backoff (which is what exhausted-me would've done at 2 AM), I implemented a proper circuit breaker:

import time
from functools import wraps

class CircuitBreaker:
    def __init__(self, failure_threshold=5, reset_timeout=60):
        self.failures = 0
        self.threshold = failure_threshold
        self.reset_timeout = reset_timeout
        self.last_failure = 0
        self.state = "CLOSED"  # CLOSED → OPEN → HALF_OPEN

    def call(self, func, *args, **kwargs):
        if self.state == "OPEN":
            if time.time() - self.last_failure > self.reset_timeout:
                self.state = "HALF_OPEN"
            else:
                raise Exception("Circuit is OPEN — upstream may be down")

        try:
            result = func(*args, **kwargs)
            if self.state == "HALF_OPEN":
                self.state = "CLOSED"
                self.failures = 0
            return result
        except Exception as e:
            self.failures += 1
            self.last_failure = time.time()
            if self.failures >= self.threshold:
                self.state = "OPEN"
            raise e

Now when the payment provider silently changes their limits again, the circuit opens after 5 consecutive failures and stops hammering their API for 60 seconds. No more cascading failures. No more 5 AM wake-up calls.

What I Actually Learned

Your code is innocent until proven guilty. I wasted 30 minutes tracing my own retry logic before checking the upstream. Don't assume you're the problem.
"Minor updates" are never minor. If a third-party API changes behavior, even slightly, it can break your production. Subscribe to changelogs. Set up integration tests that catch rate limit changes.
Circuit breakers are not optional. If your service depends on external APIs, you need them. Retry logic alone just turns timeouts into retry storms.
Don't write production code at 2 AM. My retry logic was actually fine. But I couldn't remember writing it. That's a problem.

I've been collecting patterns like these — circuit breakers, retry strategies, payment gateway integrations — into a reusable Python toolkit. If you're building anything that talks to external APIs, check it out here.

What's the worst "it's not my code... oh wait, yes it is" moment you've had?