I was mass-sending everything to GPT-4. Here's what I changed.

Juan — Tue, 07 Apr 2026 01:19:39 +0000

I'm a solo dev from Argentina building AI tools. For months I was doing what most of us do — every API call went straight to GPT-4 (now GPT-4o). Summarization? GPT-4. Formatting a JSON? GPT-4. Answering "what's 2+2"? You guessed it.

Then I looked at my bill and did some math.

The numbers that made me stop

Here's what the main LLM providers charge per 1M tokens right now:

Model	Input	Output
GPT-4o	$2.50	$10.00
GPT-4o-mini	$0.15	$0.60
Llama 3.1 8B (via Groq)	$0.05	$0.05
Claude 3.5 Sonnet	$3.00	$15.00

Look at that gap between GPT-4o and Llama. That's a 50x price difference on input tokens.

And here's the thing — for probably 70% of what I was sending to GPT-4o, Llama would've given me the same answer.

What I tried first

The obvious solution: just add some if/else logic.

`pythonif is_simple(prompt):
    model = "llama-3.1-8b-instant"
else:
    model = "gpt-4o"`

Sounds easy. It's not.

What's "simple"? How do you define that? Token count? Keywords? And then you need different API clients for OpenAI vs Groq. Different error handling. Fallback logic when one provider goes down. Rate limiting per provider.
It turned into spaghetti real fast.

What I actually ended up building

I spent a few months building a proxy that handles all of this automatically. You point your OpenAI SDK at it, and it figures out which model to use per request.

The routing logic is basically:

Classify the prompt — is it casual chat, coding, analysis, math, translation? Each has a different complexity baseline.
Check complexity — token count, multi-step signals, risk level
Route — simple stuff goes to Llama (nearly free), complex stuff goes to GPT-4o.
Validate — a background process compares cheap vs premium responses to catch quality drops.

The integration looks like this:

`pythonfrom openai import OpenAI

# just change the base_url, that's it
client = OpenAI(
    base_url="https://your-proxy-url/v1",
    api_key="your-routing-key"

)

use it exactly like before

response = client.chat.completions.create(
model="gpt-4o", # the router overrides this
messages=[{"role": "user", "content": "hello"}]
)`

One line change. Everything else stays the same.

What I learned

A few things that surprised me:

Most classification doesn't need AI. I started with GPT-4o-mini classifying each prompt (ironic, paying for AI to decide if I should pay for AI). Switched to regex + heuristics. Zero cost, runs in <1ms.
Fallbacks matter more than routing. If Groq goes down and you don't have a fallback, your users don't care about your 80% savings. They care that the app is broken.

-Quality validation is the hard part. Routing is easy. Knowing if the cheap model gave a good enough answer — that's the real problem. I built a shadow engine that samples responses and compares them. Still not perfect.

where I'm at now

$0 MRR. Zero paying customers. The product works — I use it myself every day. But I'm at the "talking to people" stage now, which is way harder than building.
If you're spending $100+/mo on LLM APIs and want to try it, the project is called NeuralRouting. Free tier available, no credit card.

But honestly I'm more interested in hearing: how are you handling multi-model routing? Roll your own? Using a gateway? Just sending everything to one model and accepting the cost?

DEV Community: Juan

I was mass-sending everything to GPT-4. Here's what I changed.

use it exactly like before