How I Cut My LLM API Costs by 70% Without Touching My Code

I was spending $200/month on AI APIs. Now it's $60. Same quality, same codebase, same endpoints. The only thing that changed? How I think about the problem.

Let me back up a bit.

I run a small SaaS product that summarizes legal documents for small firms. It's a niche tool, but it solves a real pain point — lawyers hate reading 80-page contracts. The entire product revolves around LLM calls: extracting clauses, summarizing sections, answering questions about the document. I can't just "use a cheaper model" because my users need accuracy. Hallucinating a clause is worse than not summarizing at all.

For months, I accepted the cost as a necessary evil. Then my AWS bill hit $240 in a single month, and I had a breakdown over a cup of coffee. That's when I started digging into how I could reduce costs without sacrificing quality.

Here's what actually worked.

The Naive Approach (and Why It Failed)

My first instinct was to switch from GPT-4 to GPT-3.5 everywhere. That took about 20 minutes to implement and immediately cut costs by 60%. But two weeks later, I had three support tickets about inaccurate summaries. My users noticed. The feedback was brutal: "This missed a liability limitation clause."

I switched back within a day and ate the cost.

The lesson? You can't just swap models. You need to think about when you need expensive models and when you don't.

Trick #1: Model Tiering (Not Just "Cheap vs Expensive")

I realized I was using GPT-4 for everything — even trivial tasks like extracting a date from a text snippet. That's like using a forklift to move a pencil.

So I built a simple routing layer that classifies each request before sending it to a model:

import re

def route_request(prompt, content_length, task_type):
    """
    Decide which model to use based on task complexity.
    Returns a model name.
    """
    # Simple extraction tasks can be handled by smaller models
    if task_type == "extraction" and content_length < 500:
        return "gpt-3.5-turbo"

    # Summarization requires nuance — use a strong model
    if task_type == "summarization" and content_length > 2000:
        return "gpt-4"

    # For medium documents, try a mid-tier model first
    if task_type == "summarization" and 500 < content_length < 2000:
        return "claude-3-haiku"

    # Classification and labeling is easy for any model
    if task_type == "classification":
        return "gpt-3.5-turbo"

    # Default to a solid all-rounder
    return "claude-3-sonnet"

Now, before any API call, I run this function. It's not sophisticated, but it saved me about 30% of my bill within a week.

The key insight: not every token is equal. Extracting keywords from a paragraph is easy. Summarizing a 30-page contract with nuanced legal language isn't. Route accordingly.

Trick #2: Prompt Compression (The Hidden Goldmine)

Here's where the real savings came from. I looked at my prompts and realized I was sending the same context every time — lawyer names, firm names, document metadata — even when the model didn't need it.

I built a system that tracks what information has already been sent to the model in the conversation and only sends new information plus a compressed summary of the old context.

The math: my average request was sending 3,500 tokens of context. After compression, that dropped to 1,200 tokens. That's a 65% reduction in input tokens, and since input tokens cost real money, the savings stacked up fast.

def compress_context(context: list[dict], max_tokens: int = 1500):
    """
    Compress conversation history by keeping only high-signal turns.
    Uses a cheap LLM to summarize low-value messages.
    """
    total_tokens = sum(len(m["content"].split()) * 1.3 for m in context)
    if total_tokens <= max_tokens:
        return context

    # Keep system message and last user message always
    keep = [context[0], context[-1]]
    middle = context[1:-1]

    # Summarize the middle section with a cheap model
    combined = "\n".join(m["content"] for m in middle if m["role"] == "user")
    summary = summarize_with_cheap_model(combined)  # uses gpt-3.5-turbo

    return [context[0], {"role": "user", "content": f"[Previous context: {summary}]"}, context[-1]]

This function cut my bill by another 25%. The overhead of running a cheap summarization call is negligible compared to what I saved on the expensive model's input tokens.

Trick #3: Caching (Obvious but Undervalued)

I was ashamed to discover I wasn't caching anything. Every time a user opened the same legal document, I'd re-summarize the whole thing — even though nothing had changed. The document was immutable once uploaded.

I added a simple Redis cache with a content hash as the key:

def get_summary(document_text):
    doc_hash = hashlib.sha256(document_text.encode()).hexdigest()
    cached = r.get(f"summary:{doc_hash}")
    if cached:
        return cached
    result = call_llm(document_text)
    r.set(f"summary:{doc_hash}", json.dumps(result), ex=86400*7)
    return result

This alone saved about $30/month because my users repeatedly open the same documents.

The Numbers After 30 Days

Here's the breakdown of what changed:

Before:

• GPT-4 calls: 1,200 per month
• Average tokens per request: 4,800
• Monthly cost: $200

After:

• GPT-4 calls: 420 per month
• GPT-3.5 calls: 780 per month
• Average tokens per request: 1,900
• Monthly cost: $60

That's a 70.5% reduction. The interesting part? My error rate didn't change. My users didn't complain. Everything just worked.

The Uncomfortable Truth

The biggest cost optimization wasn't technical — it was structural. I had been treating LLM APIs like a monolithic service instead of a set of tools with different prices and capabilities. Once I started treating them individually, the savings followed naturally.

I also stopped over-engineering. I used to add "think carefully" and "ensure accuracy" to every prompt, which increased token usage without improving output. Removing fluff from prompts saved about 10% on its own.

What I Use Now

During this process, I shopped around for different providers and pricing models. Some platforms charge a flat monthly fee, which sounds great until you realize you're paying for capacity you don't use. Others charge per token but with confusing tier structures.

What I've settled on is a pay-as-you-go style routing service that sits in front of all the major models. It handles the model routing, token compression, and even retry logic for me. It's called Shadie OneAPI. I'm not saying it's the only option, but it's the one that makes sense for my workflow — I only pay for what I use, and the rate limits are generous enough that I never feel squeezed.

If you're doing this yourself, the trick isn't to find the single cheapest provider. It's to find a setup that lets you mix and match models based on the task. That flexibility is what actually cuts the bill.

Final Thoughts

The whole process took me about two days of focused work. The routing logic, the compression function, and the caching layer — maybe 300 lines of code total. Nothing complex, no ML expertise required, just careful thinking about when and how much context you're sending.

If you're looking at your API bill and wincing, I'd suggest starting with these three things in this order:

1. Caching first — lowest effort, immediate savings.
2. Prompt compression — medium effort, best long-term impact.
3. Model routing — requires the most thought, but unlocks the ceiling.

None of these require changing your product's behavior from the user's perspective. The same responses come back. The quality is identical. The only difference is how much you pay to generate them.

That's the kind of optimization I can get behind.