How I Cut My LLM API Costs by 70% Without Touching My Code

I was spending $200/month on AI APIs. Now it's $60. Same quality, different approach.

It started with a side project — a simple chatbot that answers questions about my documentation. I built it with GPT-4 because, well, GPT-4 is the default in every example you find online. I didn't think much about it until the credit card bill arrived. $200 in one month. For a side project. That hurt.

I knew AI APIs weren't cheap, but I assumed that was just the cost of doing business. Then I started digging into my usage data and found something embarrassing: 90% of my requests were short prompts. "Summarize this paragraph," "extract keywords," "rewrite this sentence." I was using a $30/M input token model to rewrite sentences. That's like using a Ferrari to go get groceries.

The good news? I cut the bill by 70% without rewriting any of my application logic. All it took was a few configuration changes and a better understanding of how to get the same quality for less.

The wake-up call

Let me show you what my original API call looked like. Standard stuff:

import openai

openai.api_key = "sk-..."
openai.api_base = "https://api.openai.com/v1"

response = openai.ChatCompletion.create(
    model="gpt-4",
    messages=[{"role": "user", "content": prompt}],
    max_tokens=500
)

Nothing fancy. The problem wasn't the code — it was the model choice. GPT-4 is incredibly capable, but if you're doing simple classification or extraction, it's massive overkill. And you pay for that overkill.

Fix #1: Routing to cheaper models without changing your code

I discovered that you don't need to change your code to change your model. If you're using the OpenAI SDK, you can point api_base to a gateway that aggregates multiple providers. That means you can swap GPT-4 for something like Claude 3 Haiku or Gemini 1.5 Flash by changing an environment variable, not your code.

Here's the same code after the change:

import openai
import os

openai.api_key = os.getenv("OPENAI_API_KEY")
openai.api_base = os.getenv("OPENAI_BASE_URL", "https://api.openai.com/v1")

response = openai.ChatCompletion.create(
    model=os.getenv("DEFAULT_MODEL", "gpt-4"),
    messages=[{"role": "user", "content": prompt}],
    max_tokens=500
)

And the environment file:

OPENAI_BASE_URL=https://your-gateway.example.com/v1
DEFAULT_MODEL=claude-3-haiku

That's it. The rest of my code stayed exactly the same. The gateway handles the request, converts it to the provider's format, and returns the response. I didn't have to learn a new SDK or rewrite any calls.

The impact was immediate. Claude 3 Haiku costs about $0.25 per million input tokens and $1.25 per million output tokens. GPT-4 costs $30 and $60. That's over 100x cheaper for the same kinds of tasks. For simple prompts, the quality difference was negligible. I moved about 80% of my traffic to Haiku, and I saved around 60% of my bill right there.

Fix #2: Finding the hidden token waste

The next thing I did was start tracking token usage on every response. Sounds obvious, but I had no idea how many tokens I was burning on long system prompts and repetitive calls.

I added a tiny logging function:

def log_cost(response):
    usage = response["usage"]
    in_tokens = usage["prompt_tokens"]
    out_tokens = usage["completion_tokens"]

    # Approximate cost per model, update this per model
    cost = (in_tokens / 1_000_000) * 0.25 + (out_tokens / 1_000_000) * 1.25
    print(f"Input: {in_tokens} tokens, Output: {out_tokens} tokens, Cost: ${cost:.4f}")

After a week, I had a spreadsheet of every request. The findings:

• My average prompt was 1,200 tokens because I was including a massive system prompt with instructions and examples.
• About 30% of my requests were exact duplicates — users asked the same questions repeatedly.
• 20% of my prompts were shorter than 100 tokens but still generated 500-token responses because I set max_tokens too high.

The system prompt waste was easy to fix: I trimmed it from 1,200 tokens to 200 tokens. The response over-generation was harder to fix without touching code, but I found that some gateways let you set default max_tokens values per model. So I set a global cap of 150 tokens for the simpler tasks. The duplicates were solved by adding a simple cache at the HTTP layer — nothing to do with my app code.

Fix #3: Shopping around for better prices

Once I started using a gateway, I realized I could compare model prices across providers without changing my setup. For example:

• GPT-4o: $5 / $15 per million tokens (input/output)
• GPT-4o-mini: $0.15 / $0.60
• Claude 3 Haiku: $0.25 / $1.25
• Gemini 1.5 Flash: $0.35 / $1.05

I know these numbers change quickly, but at the time, the difference was staggering. I ended up with a mix: Haiku for simple tasks, GPT-4o-mini for medium complexity, and GPT-4o for anything that needed serious reasoning. The gateway let me set up automatic fallbacks, so if one provider was down, another would handle the request without my code even knowing.

That's when I also came across tai.shadie-oneapi.com. It's a pay-as-you-go API gateway that aggregates several LLM providers — no monthly subscription, just pay per token. I started using it for a few projects because it gives me the same OpenAI-compatible interface but with access to cheaper models and usage-based pricing. It's been a solid part of my toolkit since then. Your mileage may vary, but it's worth checking out if you're doing the same kind of optimization.

The numbers

Let's put it together. My original bill was $200/month, all going to GPT-4.

After switching 80% of traffic to Haiku: $80/month (a 60% reduction).

After trimming system prompts and capping response tokens: $55/month (another 30% reduction on top).

After adding caching and using a gateway for best-price routing: $60/month, including some traffic I'd previously routed away because it was too expensive.

The final number: $60. That's a 70% reduction from $200. And I didn't have to rewrite a single API call in my code. I changed environment variables, trimmed some text, and added a few lines to log costs.

What I learned

The biggest lesson: model selection is the most important cost lever, and you don't need to change your code to pull it. API gateways let you experiment with different models without rewriting your application. Start with a cheaper model, measure the quality, and only escalate to a bigger model when you actually need it.

Also, don't underestimate token waste. System prompts and over-long responses are silent budget killers. Log your usage, look for patterns, and fix the obvious ones.

And finally, there's no reason to be loyal to a single provider. The LLM API landscape is changing weekly. A pay-as-you-go gateway like tai.shadie-oneapi.com gives you the flexibility to jump on better prices and new models without a migration project. I'm not saying it's perfect for everyone, but it worked for me.

If you're staring at a scary API bill right now, take a breath. You probably don't need to rewrite your app. You just need to reroute it.