Qubax AI

Posted on Oct 13 Originally published at qubax.ai

Stop using one model for everything: a 60-line Python cost audit that cut our test bill 98%

#ai #llmcost #modelrouting #python

Most apps pick one strong model and send it everything: ticket labels, JSON extraction, summaries, the lot. It works, and it's the most expensive way to run an LLM product.

The fix isn't "use a cheaper model". It's use the cheapest model that passes a check for each job, and prove it with a script instead of a hunch. Here's the 60-line version I use. It runs against any OpenAI-compatible API.

The idea in one sentence

For each task your app does, write a tiny pass/fail test. Run it on a few models, keep the cheapest one that passes, and route that task there.

The script

"""cost_audit.py: cheapest model that still passes, per task."""
import json, os, time
from openai import OpenAI

client = OpenAI(base_url="https://api.qubax.ai/v1",
                api_key=os.environ["QUBAX_API_KEY"])

# USD per 1M tokens (input, output). Pull live prices from GET /v1/models.
PRICES = {
    "claude-opus-5.5": (1.21458, 6.07292),
    "claude-sonnet-5": (0.3835, 1.5339),
    "glm-5.3-flash": (0.011278, 0.0426),
    "deepseek-v4.1-flash": (0.014706, 0.058824),
}

def as_json(out):
    return json.loads(out[out.find("{"): out.rfind("}") + 1] or "{}")

TASKS = [
  {"name": "classify_ticket", "max_tokens": 400,
   "messages": [{"role": "user", "content":
     "Classify this support ticket as one of billing, bug, feature, other. "
     "Reply with the label only.\n\nTicket: I was charged twice for my "
     "October invoice."}],
   "check": lambda o: o.strip().lower().strip(".") == "billing"},
  {"name": "extract_json", "max_tokens": 600,
   "messages": [{"role": "user", "content":
     "Extract name, company and plan as JSON with keys name, company, plan. "
     "JSON only.\n\nHi, this is Maria Lopez from Northwind. We want to move "
     "from the Starter plan to Pro next month."}],
   "check": lambda o: (lambda d: d.get("name") == "Maria Lopez"
       and d.get("company") == "Northwind"
       and "pro" in str(d.get("plan", "")).lower())(as_json(o))},
  {"name": "summarize", "max_tokens": 600,
   "messages": [{"role": "user", "content":
     "Summarize in one sentence of at most 25 words: Our Q3 revenue grew 18% "
     "to $4.2M, driven by enterprise deals, while churn fell from 3.1% to "
     "2.4%. Hiring is paused until Q1."}],
   "check": lambda o: 0 < len(o.split()) <= 30 and "18" in o},
]

def run(model, task):
    t0 = time.time()
    r = client.chat.completions.create(model=model, temperature=0,
        messages=task["messages"], max_tokens=task["max_tokens"])
    out = (r.choices[0].message.content or "").strip()
    try: ok = bool(task["check"](out))
    except Exception: ok = False
    pin, pout = PRICES[model]
    usd = (r.usage.prompt_tokens * pin + r.usage.completion_tokens * pout) / 1e6
    return {"model": model, "ok": ok, "usd": usd,
            "ms": int((time.time() - t0) * 1000)}

route = {}
for task in TASKS:
    rows = sorted((run(m, task) for m in PRICES), key=lambda x: x["usd"])
    print(f"\n{task['name']}")
    for x in rows:
        print(f"  {x['model']:<22} {'PASS' if x['ok'] else 'FAIL'}"
              f"  ${x['usd']:.6f}  {x['ms']} ms")
    passing = [x for x in rows if x["ok"]]
    route[task["name"]] = passing[0]["model"] if passing else "claude-opus-5.5"

print("\nRouting table:", json.dumps(route, indent=2))

What it printed (real run, Oct 11, 2026)

Task	Opus 5.5	Sonnet 5	DeepSeek V4.1 Flash	GLM 5.3 Flash	Cheapest passing
classify_ticket	$0.000094 ✅	$0.000027 ✅	$0.000002 ✅	$0.000003 ✅	DeepSeek V4.1 Flash (98% cheaper)
extract_json	$0.000962 ✅	$0.000101 ✅	$0.000008 ✅	$0.000006 ✅	GLM 5.3 Flash (99% cheaper)
summarize	$0.001653 ✅	$0.000112 ✅	$0.000015 ✅	$0.000012 ✅	GLM 5.3 Flash (99% cheaper)

Every model passed every check. For these three jobs, sending everything to the flagship meant paying 50–140× more for the same pass rate.

Fractions of a cent look like nothing, so scale them: 1 million ticket classifications a month is about $94 on Opus versus $2 on DeepSeek V4.1 Flash.

Things the table doesn't show

Latency moved the other way. The flash models were slower here (2–4s vs 1.4–2s for Sonnet), mostly because of hidden reasoning tokens. For background jobs that's fine. For user-facing calls, set reasoning_effort="low" or keep a faster model for that one route.
Your checks are the product. "Contains 18" is a weak summary test. Use 20–50 real examples per task from your logs, and grade with exact matches, JSON schemas or a small judge model. The script stays the same; only check gets stricter.
Re-run it monthly. Prices and models change fast. On the market I used, prices for some models moved more than 40% in a week. Commit the routing table and rerun the audit in CI.

Turning the table into routing

ROUTE = json.load(open("route.json"))   # output of the audit

def llm(task, messages, **kw):
    return client.chat.completions.create(
        model=ROUTE.get(task, "claude-sonnet-5"), messages=messages, **kw)

That's the whole router. No framework, and no gateway to run.

I ran this against Qubax, an OpenAI-compatible API with 400+ models, often well below OpenRouter prices, so every model in the test is one model= string away. The audit itself is provider-neutral: point base_url at any OpenAI-compatible endpoint. Live per-model prices are on the price index.

FAQ

How many test examples do I need per task?

Start with 20–50 real examples from your logs. The three single examples in this post show the method; real routing decisions need more coverage.

Won't cheaper models be slower?

Sometimes. In this run the flash models took 2–4 seconds, mostly because of hidden reasoning. For user-facing calls, set reasoning_effort low or keep a faster model on that route.

How often should I re-run the audit?

Monthly, or in CI. Model prices and quality change quickly, so a routing table from three months ago is probably costing you money.

References

OpenAI Python SDK: https://github.com/openai/openai-python
OpenRouter model pricing (comparison baseline): https://openrouter.ai/models
DeepSeek API pricing: https://api-docs.deepseek.com/quick_start/pricing

Which task in your app are you sure needs the big model? Run the audit and tell me if it survives.

Originally published at qubax.ai. Qubax gives you every top AI model with one API key, cheaper than OpenRouter.

DEV Community