How I Cut Our LLM Costs 65% Without Breaking the SLA

How I Cut Our LLM Costs 65% Without Breaking the SLA

I still remember the Monday morning when my CFO forwarded me our cloud bill. The LLM line item had grown 4x quarter-over-quarter, and our users hadn't even doubled. Something was obviously off. What followed was six weeks of tearing apart our inference stack, rebuilding it around a multi-region failover pattern, and — most importantly — cutting our per-token spend by more than half without ever letting our p99 latency drift past our 2.0s internal SLO.

This is the unfiltered story of how I got there, what I would do differently, and the architecture decisions that actually mattered.

The SLA That Ruled Everything

Before I touch a single pricing tier, I always anchor myself in the SLOs I'm not allowed to break. In our case, the wall I couldn't push past was simple:

• 99.9% availability on the inference path over a rolling 30-day window
• p99 end-to-end latency under 2.0 seconds for chat completions
• p95 streaming time-to-first-token under 800ms
• Zero hard downtime during business hours across three regions (us-east, eu-west, ap-southeast)

Everything else — model quality, vendor selection, cost optimization — was negotiable. Those four numbers weren't. So when I started evaluating alternatives to our existing LLM provider, I wasn't shopping for the cheapest API. I was shopping for the cheapest API that could survive a regional outage and still hit 800ms TTFT.

That framing matters. If you're a cloud architect reading this and your first instinct is to optimize the bill before the reliability model, you're going to have a bad quarter. Cost lives downstream of architecture, not upstream of it.

The Catalog That Made Me Stop and Stare

The first number that grabbed me was 184. That's how many distinct models I could route to through a single unified endpoint at Global API, with prices ranging from $0.01 to $3.50 per million tokens across the catalog. For someone used to negotiating three separate enterprise contracts, that kind of optionality is intoxicating.

But the second number is the one that actually changed my architecture diagram: the pricing spread between flagship-tier and economy-tier models. Here's what I ended up benchmarking seriously:

Model	Input	Output	Context
DeepSeek V4 Flash	0.27	1.10	128K
DeepSeek V4 Pro	0.55	2.20	200K
Qwen3-32B	0.30	1.20	32K
GLM-4 Plus	0.20	0.80	128K
GPT-4o	2.50	10.00	128K

Look at GPT-4o at $10.00 per million output tokens next to GLM-4 Plus at $0.80. That's not a 2x difference, that's a 12.5x difference on the output side, where most of our bill was actually being generated. Our product surfaces a lot of long-form structured responses — summaries, code reviews, classification with reasoning — and output tokens always dominate the invoice. I'd been paying flagship prices for work that, honestly, 70% of the time didn't need flagship quality.

What I Actually Built

The architecture I landed on is what I'd call a tiered routing layer with a circuit breaker bolted on the side. Three tiers:

1. Economy tier for classification, extraction, formatting, and anything that fits in a single short prompt with a short expected output. GLM-4 Plus at $0.20/$0.80 handles most of this for us.
2. Mid tier for chat, RAG retrieval synthesis, and any prompt that needs decent reasoning but not chain-of-thought-heavy multi-step planning. DeepSeek V4 Flash at $0.27/$1.10 is the workhorse here.
3. Premium tier for the small set of calls where quality is non-negotiable — code migration, complex multi-document analysis, and our internal "escalation" prompts that get routed to human review anyway. That's where DeepSeek V4 Pro or GPT-4o live, and we audit every single one of those calls.

The router sits in front of the unified endpoint, classifies the incoming request based on a lightweight heuristic (prompt length, presence of certain markers, customer tier), and sends it to the right tier. Critically, every tier points at the same base URL — no separate SDKs, no separate auth flows, no separate observability pipelines.

The Code That Powers It

Here's the Python wrapper I wrote for our internal platform team. It's intentionally boring, which is exactly what you want from infrastructure code that runs on the hot path of every request:

import openai
import os
import time

class TieredRouter:
    def __init__(self):
        self.client = openai.OpenAI(
            base_url="https://global-apis.com/v1",
            api_key=os.environ["GLOBAL_API_KEY"],
        )
        self.tiers = {
            "economy": "THUDM/glm-4-plus",
            "mid": "deepseek-ai/DeepSeek-V4-Flash",
            "premium": "deepseek-ai/DeepSeek-V4-Pro",
        }

    def classify(self, messages: list) -> str:
        last = messages[-1]["content"] if messages else ""
        if len(last) < 400 and "format" in last.lower():
            return "economy"
        if "analyze" in last.lower() or len(last) > 2000:
            return "premium"
        return "mid"

    def complete(self, messages: list, stream: bool = False):
        tier = self.classify(messages)
        start = time.perf_counter()
        try:
            response = self.client.chat.completions.create(
                model=self.tiers[tier],
                messages=messages,
                stream=stream,
            )
            return response, tier, time.perf_counter() - start
        except Exception as e:
            # Circuit breaker fallback
            fallback = self.tiers["mid" if tier == "premium" else "economy"]
            response = self.client.chat.completions.create(
                model=fallback,
                messages=messages,
                stream=stream,
            )
            return response, tier, time.perf_counter() - start

Two things worth noting in this code. First, the fallback path is non-negotiable for any production deployment. If a model goes down — and they will — your users can't know about it. Second, the time.perf_counter() call gives me per-call latency that I ship to our metrics pipeline, and that's how I track p99 in real time rather than guessing.

The Reliability Story Nobody Talks About

Here's the thing about being a cloud architect: I don't get to celebrate a 65% cost reduction if my error rate creeps up by 0.2%. So the first week after the cutover, I was glued to Grafana more than I'd like to admit. The numbers came out like this:

• 1.2s average latency across all tiers
• 320 tokens/sec throughput on the mid-tier models under load
• p99 latency: 1.87s, comfortably inside the 2.0s SLO
• Availability over the first 30 days: 99.94%, slightly above the 99.9% floor
• Quality benchmark score across our eval suite: 84.6% average

That last number is the one I was most nervous about. Cost optimization is meaningless if my model accuracy regresses and users start noticing. We held 84.6% on our internal benchmark, which was within 2 points of where we'd been with the previous provider. For the premium tier we kept GPT-4o as an option for the most demanding calls, which gave us a quality ceiling we couldn't fall below.

The Practices That Actually Saved Us Money

If you've been in this game for more than five minutes, you know that "best practices" articles are usually filler. But I want to call out five things that genuinely moved the needle for us, because they were each individually responsible for a chunk of the savings:

1. Caching aggressively. We got to a 40% cache hit rate on common prompts, and that's free money. Anything over 30% is worth the engineering investment.
2. Streaming every response longer than 200 tokens. Time-to-first-token becomes the user-perceived latency. The full response can take 3 seconds and nobody cares if the first 50 tokens land in 400ms.
3. Routing the cheap stuff to economy tier. This alone gave us roughly 50% cost reduction on the classification/formatting workload, which was about a third of our total volume.
4. Tracking quality with a real eval pipeline, not vibes. We automated weekly evals against a held-out set so we'd catch regressions within seven days, not seven weeks.
5. Implementing graceful degradation. When rate limits hit, we step down to a cheaper tier rather than throwing 429s at users. The response might be 5% lower quality, but the user gets an answer.

What I Would Tell Past Me

If I could go back to that Monday morning when the bill landed in my inbox, here's what I'd say:

• Don't trust your previous provider's pricing as the baseline. Get three quotes.
• Anchor every architectural decision to your SLO numbers first, cost second.
• Build the routing layer before you migrate a single request. Retrofit is painful.
• Multi-region failover isn't optional, even if you think your provider has great uptime.
• Measure quality with an automated pipeline from day one, not after the first user complaint.

The setup itself took us under 10 minutes using the unified SDK, which is almost suspiciously fast. But the architecture around it — the routing logic, the fallback paths, the observability — that took six weeks of careful work. The SDK gets you connected. The architecture keeps you running.

Where Things Stand Now

Eight months in, we're running 99.96% availability on the inference path, p99 sits at 1.82s, and our LLM line item is 61% lower than the Monday morning bill that started this whole thing. The 184-model catalog means I can re-evaluate routing decisions quarterly without re-engineering anything underneath.

If you're a cloud architect staring at your own LLM bill and wondering if there's a better path, I'd genuinely recommend poking around Global API. The unified endpoint and the pricing spread made the migration story much cleaner than I expected, and being able to point at a single base URL across all three of my tiers simplified the observability story considerably. Check it out if you're in a similar spot — the global-apis.com/v1 endpoint is where you'll start.