Shaw Sha

Posted on Oct 15

How I Cut My LLM API Costs by 70% Without Touching My Code

#ai #api #programming #tutorial

I was staring at my monthly API bill last quarter, and the number genuinely made me wince — $212 for LLM calls alone. My app wasn't even that popular. I had maybe 3,000 daily active users, nothing viral, yet I was hemorrhaging cash on every request. After a weekend of digging through logs and provider pricing pages, I got that number down to $61. Same app, same code, same user experience. The difference was purely in how I structured my API layer.

It turns out most of us are overpaying for LLM APIs without realizing it. Here's what I learned, and how I did it.

The Problem: We Treat Every Call as a Flagship Call

When I first integrated LLMs into my product, I did the obvious thing. I picked the best model available, wired it up, and called it a day. Every request — from a simple "classify this email" to a full document rewrite — went to the most expensive endpoint I had access to.

The math was brutal. Let me break it down:

GPT-4o: $5.00 / 1M input tokens, $15.00 / 1M output tokens
Claude 3.5 Sonnet: $3.00 / 1M input, $15.00 / 1M output
Cheaper small models (e.g., GPT-4o-mini or similar): $0.15 / 1M input, $0.60 / 1M output

Do you see the gap? A factor of 30x on input, 25x on output. And yet, in my initial integration, I sent almost everything to the premium tier.

So the first question I had to ask myself was: does every single feature in my app need a flagship model?

In my case, the answer was a resounding no. About 70% of my calls were simple classification, entity extraction, or short-form summarization tasks. Tasks where tiny models outperform the big ones in latency but cost 25x less.

Strategy 1: Task-to-Model Matching

I started categorizing every API call in my app into one of three tiers:

Simple tasks — keyword extraction, sentiment, tag generation, short translations
Medium tasks — summarization, code explanation, structured data extraction
Complex tasks — long-form creative writing, multi-step reasoning, code generation from scratch

For tier 1, I now use small models exclusively. For tier 2, I use medium-tier models. Only tier 3 gets the premium flag.

Here's an example of the routing logic I implemented — a simple config map in Python that decides which endpoint to hit:

class ModelRouter:
    # Task type -> (provider, model, max_tokens)
    ROUTES = {
        "classify": ("openai", "gpt-4o-mini", 50),
        "extract": ("openai", "gpt-4o-mini", 200),
        "summarize": ("anthropic", "claude-3-5-haiku", 500),
        "code_review": ("anthropic", "claude-3-5-sonnet", 1500),
        "long_form": ("openai", "gpt-4o", 4000),
    }

    @staticmethod
    def get_endpoint(task_type: str):
        return ModelRouter.ROUTES.get(
            task_type,
            ModelRouter.ROUTES["long_form"]  # Safe default
        )

# Usage stays the same at the call site
def call_llm(task_type: str, prompt: str):
    provider, model, max_tokens = ModelRouter.get_endpoint(task_type)
    # ... existing logic, just with different params

The beauty of this — and I want to stress this — is that I didn't change how my code invoked the LLM. The function signature stayed the same. I just swapped the configuration underneath. All the call sites still do call_llm("summarize", text) and things just work.

Result: I cut my token spend by about 55% in week one alone.

Strategy 2: Caching at the API Layer (Not Just at the App Layer)

I had caching in my app — sure. Redis, short TTLs, the works. But I was only caching exact matches. The reality is that many of my calls were semantically identical, not textually identical.

Example: my users frequently ask "what's the refund policy?" in different phrasings: "can I get a refund", "how do refunds work", "money back process". These all hit the LLM individually.

I built a lightweight semantic cache in front of the LLM. Every prompt goes through an embedding step first. If the embedding is within a cosine similarity threshold of a previous response (and the model used was the same) — I serve the cached response.

Here's a simplified version of what I'm running:

import numpy as np
from openai import OpenAI

client = OpenAI()
cache = {}  # In production, use Redis or a vector DB

def llm_with_cache(prompt, task_type, threshold=0.92):
    emb = client.embeddings.create(
        input=prompt,
        model="text-embedding-3-small"
    ).data[0].embedding

    for cached_prompt, cached_response in cache.items():
        if np.dot(emb, cached_prompt) > threshold:
            return cached_response  # Cache hit, $0 spent

    response = call_llm(task_type, prompt)  # Normal path
    cache[tuple(emb)] = response
    return response

The embedding call costs fractions of a cent. The cache hit saves $0.01–$0.50 per request depending on the model. In my app, roughly 30% of repeat user queries hit this cache.

Result: Another 25% reduction in overall spend.

Strategy 3: Stop Over-Paying for Tokens You Don't Need

This one was embarrassing to discover. I was generating responses with max_tokens set to huge values — often the model's maximum — and paying for unused tokens. No, wait, actually the bigger issue was the reverse.

I was generating way more tokens than needed because I wasn't constraining output. For a classification task that requires just a "yes" or "no" label, I was letting the model ramble three paragraphs and paying for every token.

Fixing this was embarrassingly simple. I added explicit token caps per task type (you can see it in the router config above — the max_tokens field). And I started using JSON mode or structured output constraints where possible.

The difference between a 200-token output and a 2,000-token output on a premium model is roughly 10x cost — for the same logical result. Trimming output alone saved me another $20–$30 per month.

Strategy 4: Multi-Provider Pricing Arbitrage

Here's where things get interesting. I realized that not all providers charge the same for comparable quality. In fact, at the time of writing this post:

Provider / Model	Input (per 1M)	Output (per 1M)	Notes
OpenAI GPT-4o	$5.00	$15.00	Great all-rounder
Anthropic Claude 3.5 Sonnet	$3.00	$15.00	Better at long context
Google Gemini 1.5 Pro	$3.50	$10.50	Strong coding
Smaller / mid-tier providers	$0.15–$1.00	$0.60–$5.00	Good for most tasks

What I discovered is that for ~80% of my workloads, medium-tier models and alternative providers performed identically in terms of user satisfaction. My users weren't grading essays — they wanted answers.

I set up a fallback chain: try the cheaper provider first, and if it returns an error or times out, fall back to the premium one. Again, no change in my application code — just a different API base URL and a small wrapper.

This approach is much easier today, by the way. There are now API gateways that act as a unified front-end to multiple LLM providers, letting you pick a model dynamically based on your budget and quality requirements.

By the way — and this is just my personal setup, not a sponsored rec — I've been using tai.shadie-oneapi.com for my multi-provider routing. It's a pay-as-you-go gateway that lets me switch between OpenAI, Anthropic, and a few cheaper alternatives with a single API key, and it even supports the kind of model routing I described above out of the box. I like it because I only pay for what I use, no monthly commitment.

The Bottom Line: What Actually Mattered

Here's my honest breakdown of where the 70% came from:

Task-to-model matching: ~35% off my bill (the single biggest win)
Semantic caching: another ~20% off
Token budget trimming: ~10% off
Multi-provider pricing differences: ~5% off the remaining

Combined, I went from ~$200/month to ~$60/month, and that's with roughly the same request volume.

There's one more thing I want to mention, because it might save you some pain: watch your context window. Many of us stuff massive system prompts into every request. If your system prompt is 500 tokens and your actual input is 50 tokens, you're paying 91% for boilerplate. I did a full audit of my system prompts, removed duplication, moved static instructions to the app layer, and trimmed my base prompt by 40%.

Not Magic, Just Discipline

None of these techniques are revolutionary. They're just the boring, practical work of actually looking at where your money goes. The LLM API market is still young, pricing is all over the place, and most developers (myself included) default to "just use the best model" out of laziness.

The good news is that you can cut your bill massively without rewriting your application logic. It's all in how you route, cache, and budget your calls. And once you do it, it becomes automatic — your future integrations just inherit the savings.

If you're spending more than $50/month on LLM calls, I strongly recommend spending one afternoon on this. Audit your request patterns, apply a tiered routing model, and set up some kind of cache. The 70% reduction I saw wasn't a one-off — it's replicable on most production apps that rely heavily on LLM calls.

And if you don't want to build the multi-provider routing layer yourself, services like the one I linked above can get you there in an afternoon. Either way, your future budget will thank you.

DEV Community