How I Cut My LLM API Costs by 70% Without Touching My Code

I was staring at my credit card statement, and it wasn't pretty. $217.43 on AI APIs in a single month. For a solo developer building a side project that wasn't even generating revenue yet, that was painful.

The worst part? I knew I was being wasteful. I just didn't know how wasteful until I actually dug into the numbers.

So I spent a weekend doing what any reasonable developer would do: I treated my AI API usage like a performance bug. I profiled it, broke it down, and rebuilt the pipeline piece by piece. By Monday, my monthly bill was projected at $63. Same output quality. Same code. Just a fundamentally different approach to how I talk to these models.

Here's exactly how I did it.

---

Step 1: I Stopped Overprovisioning Every Request

The first thing I discovered was embarrassing. I was sending max tokens of 4096 on every single call. For everything. Even simple classification tasks that needed a yes/no answer.

Here's a snapshot of what my calls looked like before:

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {"role": "system", "content": "You are a helpful assistant that categorizes emails."},
        {"role": "user", "content": f"Categorize this email: {email_text}"}
    ],
    max_tokens=4096  # Way overkill for this task
)

I was paying for a truck when I needed a bicycle. When I checked the actual usage, most of my responses were coming back with 80-200 tokens. So I cut max_tokens down to 512 for classification tasks and 2048 for generation tasks. That single change shaved off about 15% of my costs right away.

But that was just the warm-up.

Step 2: I Realized I Was Asking Stupid Questions

Here's the thing about LLM APIs: you're not paying for the question. You're paying for the thinking. And I was making every model do a PhD-level analysis before answering basic questions.

The fix: cascading model tiers

I built a simple router that uses small models for small problems. My rule of thumb is:

• GPT-4o or Claude 3.5 Sonnet: Complex reasoning, multi-step planning, code generation
• GPT-4o-mini or Claude 3 Haiku: Summarization, extraction, classification
• GPT-3.5 Turbo or Llama 3.1 8B: Trivial tasks where a wrong answer costs nothing

Here's the logic that saved me the most:

async function smartCompletion(task) {
  const complexity = estimateComplexity(task);

  if (complexity === 'simple') {
    return callModel('gpt-4o-mini', task);  // ~2.5x cheaper per token
  }

  if (complexity === 'medium') {
    return callModel('gpt-4o-mini', task, { max_tokens: 2048 });
  }

  return callModel('gpt-4o', task);
}

function estimateComplexity(task) {
  if (task.type === 'extract' || task.type === 'categorize') return 'simple';
  if (task.type === 'summarize' || task.type === 'rewrite') return 'medium';
  return 'complex';
}

You'd be surprised how many "complex" tasks are actually simple. When I started logging prompts and checking their actual output, about 70% of my calls were handled by the mini tier without any quality drop.

That moved the needle. Hard.

Step 3: Caching Was My Biggest Blindspot

Look, I know "add caching" sounds like the most boring advice in the world. But let me show you the numbers from my own logs:

In a typical week, I was sending 11,847 identical requests.

Not similar.

Identical. Byte-for-byte the same prompt.

A lot of this came from my cron jobs — I was re-processing the same data every hour, even when nothing had changed since the last run. That's not intelligence. That's insanity.

I built a simple in-memory cache (and later Redis) keyed by a hash of the prompt:

import hashlib
import json
import redis

r = redis.Redis(host='localhost', port=6379, decode_responses=True)

def cached_completion(prompt, model):
    cache_key = hashlib.sha256(
        json.dumps({"prompt": prompt, "model": model}).encode()
    ).hexdigest()

    cached = r.get(cache_key)
    if cached:
        return json.loads(cached)

    response = call_llm(prompt, model)
    r.set(cache_key, json.dumps(response), ex=3600*24)  # 24hr TTL
    return response

That cut my request count dramatically. But here's the more interesting part: caching isn't just about deduplication. It's also about being smart about when to call.

For tasks that weren't time-sensitive (like batch processing yesterday's support tickets), I queued them up and ran them at off-peak hours. Some providers charge different rates at different times, but more importantly, I could batch multiple tasks into a single call with structured prompting. Batching 5 tasks into one call with a JSON output block cut my per-task cost by about 40%.

Step 4: Context Engineering Beat Prompt Engineering

Here's the mistake that cost me the most money without me knowing it: I was sending massive system prompts and entire conversation histories with every call.

I had one particular function that kept a 10,000-token conversation history alive so the model could "remember context." That's pure waste. For one call that needed 100 tokens of output, I was paying for 10,000 tokens of input context.

I switched to what I call "flash context" — strip everything down to just the essential inputs:

Before:

messages = [
    {"role": "system", "content": FULL_INSTRUCTIONS_BOOKLET},
    {"role": "user", "content": full_conversation_history},
    {"role": "assistant", "content": last_response},
    {"role": "user", "content": new_request}
]

After:

messages = [
    {"role": "system", "content": "You extract action items from support tickets. Return JSON only."},
    {"role": "user", "content": ticket_text[:2000]}  # Truncate to relevant portion
]

I tested this against my old approach with 50 real tickets. The extraction quality was statistically identical (97% overlap in extracted items). But the token input dropped by 80%.

Context compression technique

Here's a trick worth stealing: I built a "context preprocessor" that summarizes long inputs before sending them to the model. If the input document is over 3,000 characters, I run a cheap compression pass first.

def compress_context(text):
    if len(text) < 3000:
        return text
    # Use a small, cheap model to summarize to 500 words
    summary = call_small_model(f"Summarize: {text}")
    return summary

I benchmarked this against raw contexts. The results were within 2-3% accuracy for most tasks. But the cost savings were massive — because with GPT-4o you're paying roughly 4x more for input than output per token.

Step 5: I Got Rid of My "Always GPT-4o" Default

I'm embarrassed to admit this, but I was just defaulting to the biggest model because I didn't want to think about it. That's lazy engineering.

When I did the comparison test — same prompts, same temperature, same everything — here's what I found:

• For code generation tasks: GPT-4o was measurably better, but GPT-4o-mini was acceptable for 60% of cases
• For text classification: GPT-3.5 Turbo matched GPT-4o 94% of the time
• For data extraction: All three models were within noise of each other on accuracy

The gap between flagship and mid-tier models is shrinking. If your application doesn't have a critical failure mode, you can get away with considerably cheaper models with better prompt design.

At the end of that test weekend, I had a matrix of which model to use for which task type. And I let go of the idea that "big model = better output." Sometimes, it just means "bigger bill."

---

What This Actually Saved Me

Let me break down the real numbers:

Category	Before	After
Model tiering	$112	$31
Token waste (max_tokens)	$28	$8
Caching & deduplication	$45	$12
Context compression	$32	$12
Total	$217	$63

That's a 71% reduction, and I didn't touch a single line of application logic. The output quality stayed the same because every change happened at the infrastructure level, not the prompt level.

The Extra 10%: Rate Limits and Batching

Beyond the big wins, there are smaller optimizations that add up:

• Batch low-priority jobs into single calls where possible
• Use streaming for interactive responses — you're only charged for tokens you actually receive, and streaming lets you cut off early
• Monitor token usage per user — I found that 3 heavy users were consuming 40% of my API budget, so I implemented per-user quotas

What I Use Now

I ended up consolidating on a setup that gives me flexible routing across multiple providers without being locked into one vendor's pricing. One thing that made a real difference was using a gateway that lets me switch between model providers based on current costs and availability — because honestly, the "best" model changes month to month now.

These days I'm running through a unified API endpoint that handles the routing, caching, and fallbacks for me. If you're in the same boat and want to avoid vendor lock-in without building all that infra yourself, there's a service I've been using daily called tai.shadie-oneapi.com. It's a pay-as-you-go aggregator that plugs into multiple LLM providers and handles the cost-tiering in the background. No monthly minimums, no enterprise sales call — you just top up and go.

Look, I'm not saying it's magic. But if you're tired of watching your AWS bill creep up every month just because you wanted to call a chatbot API, it's worth a look.

---

The Takeaway

The biggest lesson from this entire exercise? I was paying for intelligence I didn't need. The models are powerful, sure. But most of my workload is boring, repetitive, deterministic work that doesn't need the full firepower of a frontier model.

Cutting costs wasn't about compromising quality. It was about matching the tool to the problem. That's always been good engineering. The AI API ecosystem just made it easy to forget that.

Your move: go check your logs. Look at the actual token usage and prompt lengths from the last week. I bet you'll find the same waste I did — the same 70% sitting there waiting to be reclaimed.

Happy coding. And may your API bills be small and your test coverage be large.