How I Cut My LLM API Costs by 70% Without Touching My Code

We’ve all been there: staring at a monthly API invoice that looks like a car payment. A few months ago, I was running a handful of side projects—a Slack bot that summarizes threads, a small language-learning app, and a RAG-based document search tool for a non-profit. I was burning through roughly $200 a month on LLM APIs. It wasn’t a catastrophic number, but it was eating into my profits and making me second-guess every feature I added.

I tried the obvious stuff first. I switched models, I did prompt compression, I even tried to make my code call the cheapest model per task. Nothing moved the needle much. Then I had a realization: the problem wasn’t the model I was calling—it was how I was calling it. I wasn't paying for intelligence; I was paying for waste.

Over the next few weeks, I made a few key changes—all on the server side, none of them touching my application logic—and I got my bill down to around $60 a month. Same quality, same latency, same code. Here’s exactly how I did it, and how you can too.

The 80/20 rule of token usage

Let me break down where the money actually goes. When you call openai.chat.completions.create or any equivalent, you’re paying for two things: the prompt (input tokens) and the completion (output tokens). Most developers I meet obsess over output tokens because “that’s what the user sees,” but input tokens are where the hemorrhaging happens.

In my document search tool, I was sending a 2,000-token system prompt with every single request. That’s not crazy for a complex task, but here’s the kicker: the tool would make three API calls per user query—one for retrieval, one for a draft answer, and one for a final polish pass. Each call re-sent that 2,000-token prompt.

Quick math: 2,000 tokens × 3 calls × 50 queries per day = 300,000 input tokens per day just on the same static text. That’s roughly $1.50 a day on nothing but my own instructions.

The fix was embarrassingly simple: precompute the system prompt’s token count and cache it, or restructure the flow to combine steps. I don’t need a separate draft and polish pass for most queries. I merged the draft and final stages into a single call with a stricter instruction set. That cut my API calls per query from three to one.

# Before: Three separate calls
def answer_query(query, context):
    draft = llm_call(system_prompt, context + query, "draft")
    polished = llm_call(system_prompt, context + draft, "polish")
    return polished

# After: One call, better instructions
def answer_query(query, context):
    # The key is asking the model to self-correct in a single pass
    response = llm_call(system_prompt, context + query, 
                        "Write the final answer directly. Do not draft first.")
    return response

This alone cut my bill by about 30%. But the big win was yet to come.

The cache that I was ignoring

I’m not talking about prompt caching (though that helps if your provider supports it). I’m talking about logical caching. In my Slack bot, I realized that about 40% of the threads being summarized were near-duplicates or contained recurring questions. I was sending the entire thread history to the model every single time.

I added a simple Redis layer. Before hitting the LLM, I compute a hash of the thread ID and the last message timestamp. If the hash matches a recent cached summary, I return that. No API call needed. For a bot that gets triggered 100 times a day, I saved about 40 calls/day.

The same logic applies to your prompt. If you’re building a chatbot that answers customer support questions, you’re probably re-answering the same top 10 questions over and over. Cache those answers. I’ve seen people cut their costs by 50% with nothing but a dictionary and a TTL.

The dirty little secret: context trimming

Here’s the thing nobody tells you about RAG (Retrieval-Augmented Generation) apps: you don’t need to stuff the entire knowledge base into the prompt. I was guilty of this. My document search tool would pull in 5 chunks of text, each 500 tokens long. That’s 2,500 tokens of "context" per query.

I started profiling the results. Turns out, the model was only actually using the first two chunks in 80% of cases. The other three were just fluff—irrelevant sections that were semantically close but not factually useful. So I changed my retrieval to return more chunks but with a stricter relevance score, and then had the model select the top 2 before generating the answer.

# Instead of: 
context = retrieve_top_k(query, k=5)  # expensive in input tokens
# Do this:
candidates = retrieve_top_k(query, k=10)  # cheap if using a vector store
best_context = llm_call("Which of these passages are relevant? Return indices.", candidates)
# Then call the main LLM with only the top 2

This dropped my input token usage by roughly 60% on that project. The output quality didn't drop—if anything, it got better because the model wasn't distracted by irrelevant text.

API redundancy isn't just for uptime—it's for price

I used to be loyal to a single provider. Then I looked at the actual pricing models across the majors (OpenAI, Anthropic, and a few others). The price per million tokens varies wildly depending on the day, the model tier, and the promotion.

For instance, I found that for simple classification tasks, Anthropic’s Haiku model was significantly cheaper than GPT-4o mini for the same quality. But for long-form creative writing, OpenAI’s model was actually more cost-effective because it tended to generate shorter outputs (fewer tokens = less cost).

So I built a tiny router. It’s not a complex ML system—just a if/else based on the task type and the current cost of each provider. For bulk summarization, I route to the cheapest provider. For complex reasoning, I go to the strongest.

The key wins came from latency-based routing. Some providers charge different rates for batch vs. real-time. For my non-profit’s tool, where users don’t need instant responses, I switched to a batch endpoint and saved 50% off that portion of the bill.

Handling failures without burning money

Now, let’s talk about the worst money sink: retries. We’ve all written the code that does retry() on an API failure. But a naive retry means you re-send your entire prompt (including the expensive context) multiple times.

I shifted to checkpointing. If a call fails mid-stream, I have the model resume from the last successful token (if the provider supports it) or I keep a local buffer of my context. This doesn’t just save money on API costs—it saves time because you’re not re-generating everything.

The hard numbers

Here’s the TL;DR of my cost breakdown:

Strategy	Cost Savings
Merging API calls (1 vs. 3)	30%
Response caching (Redis)	10%
Context trimming (RAG)	20%
Multi-provider routing	15%
Batch endpoints	10%
Total	~70%

Before: $200/month. After: $60/month. And I did not change a single user-facing feature. My codebase actually got simpler because I removed the redundant call chains.

The tool that quietly does all this for me

I tinkered with custom routers for a bit, but maintaining my own load balancer across three providers was its own full-time job. That’s when a friend pointed me to a gateway service that consolidates API access. It’s one key that lets you switch between providers on the fly, and it handles failover if one provider spikes in price or latency.

To be honest, I was skeptical at first—how could a proxy magically cut costs? But the trick is that you can set cost limits per request and configure it to automatically use the cheapest model for your task. After I set it up, I realized my "router" script was redundant. The gateway was doing the retry logic, the model selection, and even the caching.

If you’re in a similar boat—spending hundreds on API calls and not sure where to start trimming—I’d suggest you take a hard look at your token usage before you start rewriting code. You’ll likely find the same leaks I did: repetitive prompts, over-fetching context, and unnecessary calls.

And if you want to skip the painful DIY router phase, I use a service that acts as a unified layer for all the major LLM providers. It’s called tai.shadie-oneapi.com—not an affiliate link, just genuinely what I’ve got running in production right now. It handles the switching, and it’s pay-as-you-go, so you can test it without committing a ton of money. It’s one of those "I wish I built this myself" tools.

The bottom line? You don’t need a massive refactor to save money. You just need to stop paying for tokens you don’t use. The smartest thing I did was listen to my own advice: measure first, optimize second, and let the infrastructure handle the rest.