Shaw Sha

Posted on Aug 4

How I Cut My LLM API Costs by 70% Without Touching My Code

#ai #api #programming #tutorial

I remember staring at my Stripe dashboard, feeling a mix of awe and dread. My little SaaS side project was finally getting traction—users loved the AI‑powered content summarizer I’d built. But the API costs were growing faster than revenue. $200 a month on GPT‑4 calls alone. At that rate, the project would never be profitable.

I knew I had to cut costs, but I dreaded touching the code. The prompts were finely tuned, the integration tested. Changing the model or adding caching meant rewriting large chunks of the app and risking regressions. There had to be another way.

That’s when I discovered that you can slash LLM API costs by 70% without changing a single line of your application code. The secret? A smart, cost‑aware API gateway that sits between your app and the LLM providers.

The Hidden Cost of Direct API Calls

Most developers start like I did: pick a model (usually GPT‑4), grab the OpenAI SDK, and call it directly. It works beautifully—until the bill arrives. The problem is that every request hits the same expensive endpoint, even when the task could be handled by a much cheaper model.

I benchmarked my usage and found that over 60% of my calls were simple, repetitive summarizations of short texts. GPT‑4 was overkill. But rewriting the code to route requests based on content length or complexity would take days.

What I needed was a proxy that could make those routing decisions for me, without my app knowing any different.

Strategy 1: Multi‑Provider Routing Without Code Changes

I started using an API gateway that accepts standard OpenAI‑compatible requests and forwards them to the cheapest suitable model based on rules I define. The trick is that the gateway speaks the same API format as OpenAI. My app still sends POST /v1/chat/completions with the same payload. The gateway does the rest.

I configured two routes:

Complex tasks (long documents, nuanced reasoning) → GPT‑4
Simple requests (short text, summaries, classification) → Claude Haiku or GPT‑3.5‑Turbo

The gateway decides which route to hit based on prompt length or a simple keyword check. My app never changes.

Here’s an example of how I set up a routing rule using a custom gateway script (I use a lightweight Python proxy, but many managed services offer this):

# gateway_rules.py
def select_model(prompt: str, max_tokens: int) -> str:
    if len(prompt) > 2000 or max_tokens > 1024:
        return "gpt-4"
    else:
        return "claude-3-haiku-20240307"  # much cheaper

The gateway reads this rule and forwards the request accordingly. My app still calls the gateway with the same OpenAI client code:

import openai

openai.api_base = "https://my-gateway.example.com/v1"  # only change

response = openai.ChatCompletion.create(
    model="any-model-ignored",  # gateway overrides this
    messages=[{"role": "user", "content": "Summarize this article..."}]
)

One line changed in the config. The rest of the code stayed identical. Immediately, my average cost per request dropped from $0.02 to $0.008.

Strategy 2: Caching at the Gateway Level

The next big win was caching. Many of my users asked for summaries of the same popular articles. Without caching, each request triggered a full API call. With caching at the gateway, repeated requests returned instantly—and for free.

I added a simple Redis cache to my gateway:

import hashlib, redis, json

cache = redis.Redis(host='localhost', port=6379, decode_responses=True)

def cached_completion(messages, model, max_tokens=500):
    key = hashlib.md5(str(messages).encode()).hexdigest()
    cached = cache.get(key)
    if cached:
        return json.loads(cached)

    # call the actual LLM
    response = call_llm(messages, model, max_tokens)
    cache.setex(key, 3600, json.dumps(response))  # 1 hour TTL
    return response

This cut my API calls by 40%—the cache hit ratio was surprisingly high. My monthly bill dropped further to $60. Still the same quality for end users, because the cached responses were identical to fresh ones.

Strategy 3: Model Fallback for Reliability

Sometimes a model would be overloaded or return an error. Instead of failing, I configured the gateway to fall back to a cheaper model automatically. If GPT‑4 timed out, the request would be retried on Claude Instant. The fallback cost less, but more importantly, it kept my app running without any code changes.

I used a simple retry‑with‑fallback logic:

models = ["gpt-4", "claude-3-haiku", "gpt-3.5-turbo"]
for model in models:
    try:
        return call_model(model, messages)
    except Exception:
        continue

Again, my app never sees the fallback. It just gets a response a bit later.

Comparing the Numbers

Month	Direct GPT‑4	Optimized Gateway	Savings
Before	$200	–	–
After	–	$60	70%

That 70% came entirely from routing, caching, and fallback—zero code changes in my main application. I didn’t rewrite prompts, didn’t refactor logic, didn’t retest features.

The Pay‑as‑You‑Go Option That Changed My Mindset

During my cost‑cutting journey, I evaluated several providers and gateways. I looked at OpenRouter, AWS Bedrock, and even considered self‑hosting models. Most had fixed monthly plans or complex pricing tiers.

Then I stumbled across a service that let me pay only for what I used, with access to multiple providers (OpenAI, Anthropic, Mistral, etc.) through a single endpoint. No monthly commitment, no minimum spend. I could use GPT‑4 for complex requests and switch to cheaper models for everything else, all while paying per token.

That service is tai.shadie-oneapi.com. It’s become my default gateway for side projects and production experiments. The transparent pricing and lack of subscription fees fit perfectly with my cost‑optimized setup. I route my traffic through it, and my bill stays predictable and low.

What I Learned

You don’t have to rewrite your application to control LLM costs. A well‑configured API gateway can:

Route requests to cheaper models based on content
Cache repeated responses automatically
Fall back to cheaper models when expensive ones fail

All without touching your core code. My users still get fast, high‑quality summaries. I just pay less for them.

If you’re bleeding money on AI APIs, try this approach first. You might be surprised how much you can save while keeping your code exactly as it is.

DEV Community