How I Cut My LLM API Costs by 70% Without Touching My Code

I was spending $200 a month on AI APIs. Now I’m down to $60. Same quality, same codebase, same use cases. The only thing that changed was how I routed my requests.

I didn’t switch to a cheaper model entirely—I still use GPT-4 for complex reasoning and code generation. But I stopped throwing every query at the most expensive engine. And I did it without rewriting a single line of application logic.

Here’s exactly how I did it.

The wake‑up call

Six months ago I was building a suite of AI‑powered tools for content summarisation, code review, and customer support triage. Everything was wired directly to OpenAI’s API with model: "gpt-4" as the default. It worked beautifully—accurate, fast enough, and the integration was trivial.

But when the monthly bill arrived, I nearly choked. $215 for a single developer account. Most of that was for simple classification tasks that a smaller model could have handled just as well. I started reading other people’s cost‑optimisation posts and realised I was leaving money on the table.

The obvious fix was to manually choose different models for different tasks. But that meant updating every call site, maintaining a configuration matrix, and testing each combination. My team was already stretched, and I didn’t want to introduce a brittle abstraction layer.

I needed a solution that let me keep using a single API endpoint while the routing logic happened transparently on the backend.

The gateway trick

The key insight is that most LLM providers expose an OpenAI‑compatible API these days. Anthropic, Google, Mistral, even local servers running llama.cpp. If you normalise the request format, you can send a single payload to a gateway that decides which provider to call based on rules you define: model name, prompt length, time of day, or even random load‑balancing.

I set up a lightweight proxy that accepts standard OpenAI chat completion requests and maps them to the cheapest available provider that can handle the task. The mapping lives in a simple JSON config:

{
  "routing": {
    "gpt-4": ["openai/gpt-4", "anthropic/claude-3-opus"],
    "gpt-3.5-turbo": ["openai/gpt-3.5-turbo", "google/gemini-1.5-flash"],
    "default": ["openai/gpt-3.5-turbo", "anthropic/claude-3-haiku"]
  }
}

When my code sends a request with model: "gpt-4", the gateway first tries the primary provider. If that’s down or too slow, it falls back to the secondary. But more importantly, I can add a “cheap” model alias that routes to Gemini Flash or Claude Haiku automatically.

The client‑side code didn’t change at all:

import openai

client = openai.OpenAI(
    api_key="my-key",
    base_url="https://my-gateway.example.com/v1"  # <-- only change
)

response = client.chat.completions.create(
    model="gpt-4",   # still looks like gpt-4
    messages=[{"role": "user", "content": "Summarise this email:"}]
)

Behind the scenes, the gateway decided that for this short prompt it could safely use claude-3-haiku instead of gpt-4. The response quality was identical, but the cost per call dropped from $0.03 to $0.0025.

More than just routing

Routing alone saved me about 40%. The rest came from two other techniques I layered on top: caching and prompt compression.

Caching repeated requests

Many of my summarisation calls were for the same documents—users refreshing a page, retrying a failed job, or re‑analysing a piece of code. I added a simple in‑memory cache with TTL that stored the completion for identical prompt + model combinations.

cache = {}

def get_cached_or_fresh(client, model, messages):
    key = (model, json.dumps(messages, sort_keys=True))
    if key in cache and cache[key]["expires"] > time.time():
        return cache[key]["response"]
    response = client.chat.completions.create(model=model, messages=messages)
    cache[key] = {"response": response, "expires": time.time() + 3600}
    return response

This eliminated about 15% of my API calls entirely—no latency, no cost.

Prompt compression

I also started stripping unnecessary context. Many of my prompts were bloated with system instructions that had been copied from one task to another. I wrote a small pre‑processor that removes redundant whitespace, shortens verbose instructions, and trims examples that are longer than needed.

A typical “code review” prompt went from 1200 tokens to 650 tokens. Combined with the cheaper model, that cut the cost per review by 70%.

The numbers

After three weeks of tuning, here’s what my monthly spend looked like:

Category	Before	After
GPT‑4	$130	$30
GPT‑3.5	$40	$10
Claude	$30	$15
Other	$15	$5
Total	$215	$60

Quality metrics (response accuracy, user satisfaction) stayed flat. Latency actually improved because I was using faster models for simple tasks.

The gateway I actually use

I built a proof‑of‑concept proxy myself, but maintaining it became a chore—new providers, rate limits, key rotation. So I looked for a hosted solution that did the same thing.

That’s when I found tai.shadie‑oneapi.com. It’s a pay‑as‑you‑go gateway that aggregates a dozen providers behind a single OpenAI‑compatible endpoint. You send your requests with whatever model name you like, and it routes to the cheapest available option that meets your quality threshold. No monthly commitment, no infrastructure to manage.

I’m not affiliated with them—I just use it now because it saved me the maintenance headache. If you’re in a similar boat, give it a try. It’s the same “change one line of config” approach that worked for me.

You don’t have to trade quality for cost

The biggest lesson I learned is that you can optimise spending without rewriting your application. A routing layer, a cache, and a little prompt hygiene can cut your bill by 70% or more. The code stays clean, the user experience improves, and your wallet thanks you.

If you’re currently paying hundreds a month for LLM APIs, start by auditing your traffic. You’ll probably find that most of your calls don’t need the most expensive model. Then pick a gateway—build your own or use a hosted one—and let it do the heavy lifting.

Your code doesn’t have to change. Your costs will.