How I Cut My LLM API Costs by 70% Without Touching My Code

I remember the exact moment I opened my OpenAI invoice and nearly choked on my coffee. $214.37 for a single month. Not for some massive enterprise deployment, but for a side project — an AI-powered support assistant that maybe 300 people used daily. That's roughly 71 cents per user per day. Absurd, right?

Now, four months later, that same project costs me $58–$65 a month. Same features. Same response quality. Same codebase — well, almost. I didn't rewrite the business logic or abandon my model. I just stopped treating every API call like it deserved the most expensive model on the shelf.

Here's how I did it, and more importantly, why you can probably do the same in an afternoon.

---

The problem wasn't the model. It was me.

Let me be honest: I'm lazy. When I first integrated LLM APIs, I picked gpt-4o as the default for literally everything. Summarizing a log entry? gpt-4o. Extracting a customer name from an email? gpt-4o. Answering a "what's the refund policy?" question? gpt-4o with a 2000-token system prompt.

The cost per call felt negligible — a few cents here, a few cents there. But multiplied by 15,000–20,000 calls per month, it wasn't negligible anymore. It was my entire API budget on one model that was being overkill for 80% of my traffic.

I took a step back and categorized my actual workloads:

• Trivial tasks (extraction, classification, form-filling): I don't need creative reasoning. I need reliable, structured output.
• Moderate tasks (summarization, standard Q&A, basic translation): These need decent comprehension but not deep reasoning.
• Complex tasks (multi-step planning, code generation, nuanced tone analysis): These genuinely justify the premium tier.

Turns out, only about 22% of my calls actually fell into that third bucket. The other 78% was money evaporating.

The fix: a cheap routing layer

I wrote a simple Python module that routes requests to different providers and models based on the task type. No framework, no fancy orchestration — just a dictionary, some heuristics, and the litellm library (honestly, if you're not using it, you're reinventing the wheel).

Here's a simplified version of what I run:

import litellm

# Model mapping: task_type -> (provider, model)
MODEL_ROUTES = {
    "extraction": ("openai", "gpt-4o-mini"),
    "classification": ("groq", "llama-3.3-70b-versatile"),
    "summarization": ("openai", "gpt-4o-mini"),
    "qa": ("anthropic", "claude-3-5-haiku-latest"),
    "code": ("anthropic", "claude-3-5-sonnet-latest"),
    "reasoning": ("openai", "gpt-4o"),
}

def smart_call(task_type: str, prompt: str, **kwargs):
    provider, model = MODEL_ROUTES.get(task_type, MODEL_ROUTES["reasoning"])
    return litellm.completion(
        model=f"{provider}/{model}",
        messages=[{"role": "user", "content": prompt}],
        temperature=kwargs.get("temperature", 0.2),
        max_tokens=kwargs.get("max_tokens", 500)
    )

# Usage
result = smart_call("summarization", user_email_text)
print(result.choices[0].message.content)

The magic isn't the code — it's the discipline. When I create a new feature, I don't ask "which model do I want?" I ask "what's the cheapest model that gets this job done well?" That's a completely different question, and it leads to very different choices.

---

Context is the silent killer

Here's something that surprised me: token pricing isn't the biggest driver of cost. Context length is.

My original implementation eagerly stuffed every conversation into the system prompt. The full knowledge base. All customer history. Every log for the last month. Each request was a 5,000–8,000 token monster, even for simple queries.

When I actually tested my tokens, I found something depressing: for my Q&A endpoint, the pre-prompt was 6,400 tokens and the user's actual question was 12 tokens. I was paying for 500 words of "context" to answer a 2-word question. That's like renting a 10,000-square-foot warehouse to store a single shoebox.

Strategies that actually saved me money

1. Session-aware context pruning

Instead of sending the whole conversation history, I keep only:

• The last 1 user message and 1 assistant response (for continuity)
• Any facts the model actually referenced in its previous reply
• A compressed summary of what's been discussed (regenerated every 5 turns)

This cut my context tokens by about 60%. It also made responses faster, which is a nice side effect for user experience.

2. Caching with a semantic layer

For repetitive questions ("What's your refund policy?" — ask 30 people a day), I store the embedding of the question and compare it against previously answered queries. If the cosine similarity exceeds 0.95, I serve the cached answer — no API call at all.

The numbers here were wild. I had about 40 repeated questions that accounted for 55% of my traffic. Caching those eliminated over half of my API calls entirely. That alone brought me down from ~15,000 calls/month to ~6,500.

Here's the rough logic:

import numpy as np
from sklearn.metrics.pairwise import cosine_similarity

def cache_or_call(question: str, answer_cache: dict, embed_fn, threshold=0.95):
    q_vec = embed_fn(question).reshape(1, -1)
    for cached_q, cached_a in answer_cache.items():
        c_vec = cached_q.reshape(1, -1)
        if cosine_similarity(q_vec, c_vec)[0][0] >= threshold:
            return cached_a
    # no cache hit -> call API and store
    fresh = smart_call("qa", question)
    answer_cache[embed_fn(question)] = fresh
    return fresh

3. Switching providers based on availability and price

I also stopped being a brand loyalist. The same task might cost $0.001 on one provider and $0.005 on another for nearly identical quality. I use a small aggregator that checks live price feeds and routes accordingly. Yes, there was a learning curve, but the savings are permanent.

One provider that's been quietly consistent for me is shadie-oneapi.com — it's a pay-as-you-go gateway that lets me route to multiple models through a single API key. What sold me is that billing is per-request and it doesn't lock me into a subscription I'm not fully using. If I need claude-opus for one tricky codegen task and gpt-4o-mini for a thousand extraction calls, both are available without me managing separate accounts. It simplified the operational side of what I built above.

---

The numbers after 90 days

Here's what I tracked last week, comparing March (before changes) and June (after):

Metric	March	June
Total API calls	16,240	7,892
Avg tokens per call	7,120	2,480
Total tokens	~115M	~19.5M
API spend	$214.37	$59.80

That's a 66.8% cost reduction on a month where I increased feature usage by about 8%. Not bad for a few hours of refactoring.

---

What I'd do differently (and what I'd keep)

If I could go back, I'd have done this from day one instead of after the invoice shock. Setting up model routing, caching, and context pruning from the start would've saved me around $400 in the first two months alone.

But I'd keep the same principle — the model you ship isn't your identity, it's your infrastructure. Choosing the right model for the job is a cost decision, not a pride decision. Sometimes the 70-billion-parameter model wins. Sometimes a 7-billion-parameter model with a good prompt does the same thing for 1/10th the price.

Practical takeaways

• Measure your actual token usage. Don't guess. Log every call with its token count and which model served it.
• Classify your tasks. If you can't name 3 distinct tiers of complexity in your workloads, you haven't looked hard enough.
• Cache aggressively. Most of the internet is repeated questions. Your API budget shouldn't pay for the same answer twice.
• Use a gateway or aggregator to compare pricing across providers in real-time.
• Set a hard max_tokens. I had a bug where a single call consumed 6,000 output tokens on a summarization task. That call cost $0.18 for something a human could write in 4 lines.

---

Look, I'm not telling you to switch to the cheapest model for everything — that path leads to watered-down responses and frustrated users. But being intentional about where the smart money goes is a skill, and it's one I'm glad I learned the hard way.

If you're building on top of LLMs right now, check what you're paying per request. Then check what you could be paying. The gap is usually wider than you think.

As for me, I'm running about $60/month now, and that shadie gateway is still my go-to for day-to-day routing — mainly because it cut my integration time from hours to minutes and I don't have to maintain five API keys anymore. And honestly, being able to switch models without touching my application code again feels like a superpower.

Got your own cost-cutting tricks? I'd love to hear what worked for you — and if you've caught yourself paying for a 10,000-token context when a 500-token context would do, I promise you're not alone.