The Backend Engineer's Guide to Cutting Your AI Bill in Half (Or More)

The Backend Engineer's Guide to Cutting Your AI Bill in Half (Or More)

I learned this lesson the hard way. Last year, my team burned through $14,000 in three months on what was supposed to be a "simple" LLM-powered feature. The thing is, it was simple. We just weren't paying attention to what we were sending, to what model, and how often. Once I started treating API spend like a database query cost problem (because it basically is one), the numbers dropped off a cliff.

fwiw, this isn't about finding magic enterprise discounts. It's about the same engineering hygiene you'd apply to any expensive external dependency: profile it, route around it, cache what you can, and don't send bytes you don't need to send.

Here's what actually moved the needle for us.

---

1. Stop Using GPT-4o for Everything

This is the big one. The single biggest lever, IMO, is picking a model that matches the actual task complexity. Most "AI features" I've seen in production don't need a frontier reasoning model. They need a thing that turns text into slightly different text.

I keep a routing table taped to my monitor at this point:

Workload	What I used to use	What I use now	Cut
Casual chat	GPT-4o ($10.00/M output)	DeepSeek V4 Flash ($0.25/M)	97.5%
Classification	GPT-4o-mini ($0.60/M)	Qwen3-8B ($0.01/M)	98.3%
Code generation	GPT-4o ($10.00/M)	DeepSeek Coder ($0.25/M)	97.5%
Summarization	GPT-4o ($10.00/M)	Qwen3-32B ($0.28/M)	97.2%
Translation	GPT-4o ($10.00/M)	Qwen-MT-Turbo ($0.30/M)	97%

Yeah, those percentages are real. No, I'm not making them up. The gap between frontier and small models on commodity tasks is absurd in 2025/2026. It's like renting a semi-truck to go grocery shopping.

Here's the routing helper I ended up writing:

MODEL_ROUTER = {
    "chat":         "deepseek-v4-flash",   # $0.25/M
    "code":         "deepseek-coder",      # $0.25/M
    "classification": "Qwen/Qwen3-8B",     # $0.01/M
    "reasoning":    "deepseek-reasoner",   # $2.50/M
    "summarization": "Qwen/Qwen3-32B",     # $0.28/M
    "translation":  "Qwen-MT-Turbo",       # $0.30/M
}

def pick_model(task_type: str) -> str:
    return MODEL_ROUTER.get(task_type, "deepseek-v4-flash")

# usage
resp = client.chat.completions.create(
    model=pick_model(classify(user_input)),
    messages=[{"role": "user", "content": user_input}],
)

If you're using global-apis.com/v1 as your OpenAI-compatible base URL, this drops in without changes. Seriously, that's the entire integration story.

---

2. Tier It Like Your Database Queries

Once the basic router was in place, I went one step further. Why send anything to an expensive model if a cheap one can do the job?

This pattern is straight out of the cache hierarchy playbook: try L1 first, escalate to L2 if you miss, hit L3 only when you must. Same idea, different substrate.

def tiered_generate(prompt: str, budget_usd: float = 0.50):
    # Tier 1: ultra-cheap classifier/chat tier
    resp = call_model("Qwen/Qwen3-8B", prompt)  # ~$0.01/M
    if confidence(resp) >= 0.8:
        return resp  # ~80% of traffic stops here

    # Tier 2: standard quality
    resp = call_model("deepseek-v4-flash", prompt)  # ~$0.25/M
    if confidence(resp) >= 0.9:
        return resp  # ~15% of traffic

    return call_model("deepseek-reasoner", prompt)  # ~$2.50/M

Anecdote time: a customer support pipeline I worked on went from $420/month to $28/month. The 85th-percentile query was a "where's my order" question that absolutely does not require a reasoning model. Most of them don't.

The honest version is that you also need a confidence() function that isn't nonsense. For us it was: short, deterministic-looking responses with high overlap against the retrieved context get a pass. Anything that starts hedging like "I think maybe perhaps…" escalates. Your mileage will vary, but the principle doesn't.

---

3. Cache Identical Requests (Obviously)

I'll be honest, I can't believe I shipped systems without this. Caching identical or near-identical requests is so obvious it hurts.

import hashlib, json, time

_cache = {}

def cached_chat(model, messages, ttl=3600):
    key = hashlib.md5(
        json.dumps({"model": model, "messages": messages}, sort_keys=True).encode()
    ).hexdigest()

    entry = _cache.get(key)
    if entry and time.time() - entry["ts"] < ttl:
        return entry["resp"]  # free

    resp = client.chat.completions.create(model=model, messages=messages)
    _cache[key] = {"resp": resp, "ts": time.time()}
    return resp

Hit rates for FAQ-style content are 50-80%. If you don't believe me, instrument it for a week. You'll be annoyed at how much money you were leaving on the table.

For semantic similarity (not just exact matches), embedding-based caches work too, but they're a separate engineering project. Start with exact-match caching. It's boring and it pays rent.

---

4. Compress Your Prompts Before Sending Them

Tokens are bytes. Bytes are money. Treat them the same way.

The trick is using a cheap model to summarize long context, then sending the summary to the expensive model. You save more in total than you spend on the compression step.

def compress_prompt(text: str, target_ratio: float = 0.5) -> str:
    if len(text) < 500:
        return text

    target_chars = int(len(text) * target_ratio)
    summary = call_model(
        "Qwen/Qwen3-8B",
        f"Summarize this in {target_chars} chars: {text}",
    )
    return summary

Quick math because I always want to see the math: a 2,000-token system prompt compressed to 400 tokens saves $0.024 per request on DeepSeek V4 Flash. At 10,000 requests a day, that's $240/day, which is $87,600/year. For one feature. With one compression call.

You should not be sending your entire codebase, every README, and all of yesterday's logs as part of every request. Under the hood, the same RAG techniques that make retrieval useful make bills smaller.

---

5. Batch the Easy Stuff

If you have N independent requests with no real-time requirement, don't make N round trips. Combine them.

# before: N calls, N*input_tokens billed
for q in questions:
    client.chat.completions.create(
        model="deepseek-v4-flash",
        messages=[{"role": "user", "content": q}],
    )

# after: 1 call, single shared system prompt
batch_prompt = "\n".join(f"{i+1}. {q}" for i, q in enumerate(questions))
resp = client.chat.completions.create(
    model="deepseek-v4-flash",
    messages=[
        {"role": "system", "content": "Answer each numbered question."},
        {"role": "user",   "content": batch_prompt},
    ],
)

answers = parse_numbered_responses(resp.choices[0].message.content, len(questions))

This is also where most providers (including Global API, fwiw) have explicit batch endpoints with an additional discount. If you're doing async processing anyway, use them. Reference: it's the same idea as RFC 9221 (priority hints) — don't pay express prices for ground shipments.

Savings here: 10-20% on top of whatever you've already done. Not glamorous. Worth it.

---

6. Set Hard Spending Limits

Unpopular opinion: don't trust yourself to "watch the dashboard." Automate it.

class BudgetGuard:
    def __init__(self, monthly_budget_usd: float):
        self.budget = monthly_budget_usd
        self.spent = 0.0
        self.month = time.strftime("%Y-%m")

    def check(self, estimated_cost_usd: float) -> bool:
        if time.strftime("%Y-%m") != self.month:
            self.spent = 0.0
            self.month = time.strftime("%Y-%m")

        if self.spent + estimated_cost_usd > self.budget:
            raise RuntimeError(f"over budget: ${self.spent:.2f}/${self.budget:.2f}")
        return True

    def record(self, actual_cost_usd: float):
        self.spent += actual_cost_usd

guard = BudgetGuard(monthly_budget_usd=500.0)

# before each call
estimated = estimate_cost(model, messages)
guard.check(estimated)

resp = client.chat.completions.create(model=model, messages=messages)
guard.record(usage_to_cost(resp.usage, model))

The first time this fires in production, you will either be furious or relieved. For me it was the latter.

---

7. Track Costs Per Feature, Not Globally

Per-team totals hide the truth. You want per-feature, per-route, per-tenant if you can swing it. Anything less and you'll optimize the wrong thing.

A simple tagged client is enough:

class TaggedClient:
    def __init__(self, base_url: str, api_key: str):
        self._client = OpenAI(base_url=base_url, api_key=api_key)
        self._costs = defaultdict(float)

    def chat(self, *, model, messages, tags: dict):
        resp = self._client.chat.completions.create(
            model=model, messages=messages
        )
        cost = usage_to_cost(resp.usage, model)

        label = tags.get("feature", "unknown")
        self._costs[label] += cost

        # also push to your metrics backend
        metrics.increment("llm.cost.usd", cost, tags=tags)
        return resp

    def report(self):
        for k, v in sorted(self._costs.items(), key=lambda kv: -kv[1]):
            print(f"{k:30s}  ${v:8.2f}")

Run the report weekly. The first one I ran made it instantly obvious which feature was the cash burn. I'm not going to tell you which one it was because it was embarrassing.

---

What I Actually Spend Now

Pre-optimization: roughly $4,600/month for what was, frankly, a modest workload.

Post-optimization, same traffic:

• ~$320/month on model routing
• ~$210/month after caching
• ~$150/month after prompt compression
• ~$120/month after batching

That's a 97% reduction. The ironic part is that latency went down, because cheap models are typically faster. The only thing I gave up was the warm fuzzy feeling of seeing "gpt-4o" in the logs.

---

The Stack I'd Build Today

If I were starting from scratch:

1. OpenAI-compatible client pointed at global-apis.com/v1
2. A 6-entry routing table in MODEL_ROUTER
3. cached_chat() with a 1-hour TTL as the default entry point
4. compress_prompt() for any context over 1k tokens
5. BudgetGuard wired into a Slack alert
6. Weekly cost report per feature tag

That's a weekend of work. It probably saves you five figures a year. The math doesn't lie even when the bills do.

---

The Price List I Actually Use

For reference, these are the numbers behind the percentages in the table at the top. Pin them somewhere visible — every $10/M model choice vs a $0.25/M model choice is a real line item on someone's invoice.

Model	Input ($/M)	Output ($/M)
GPT-4o	$2.50	$10.00
GPT-4o-mini	$0.15	$0.60
DeepSeek V4 Flash	$0.05	$0.25
DeepSeek Coder	$0.05	$0.25
DeepSeek Reasoner	$0.50	$2.50
Qwen3-8B	$0.01	$0.01
Qwen3-32B	$0.06	$0.28
Qwen-MT-Turbo	$0.05	$0.30

If you're running anything on Global API already, the pricing is laid out the same way and the SDK swap is basically changing a base URL. Check it out at global-apis.com/v1 if you haven't already — it took me about twenty minutes to migrate a service and I've never looked back.

That's the playbook. Nothing exotic. Just engineering.