Last March, I opened my monthly API invoice and just stared at it. $1,240 for what amounted to a side project that barely had 200 daily users. I wasn't running a startup—I was burning cash because I'd wired up the most capable (and most expensive) model for every single request, from generating alt text to answering user questions.
I spent the next two months actually treating my AI spend like a real engineering problem. Here's what moved the needle.
1. Stop using one model for everything
The biggest leak in my setup was routing every task to GPT-4-class models. Truth is, 70% of my workload was summarization, classification, or short completions—things a smaller model handles fine.
I built a tiny router:
def pick_model(task):
if task in ('embed', 'classify', 'summarize'):
return 'haiku' # cheap, fast
if task == 'chat':
return 'sonnet'
return 'opus' # only for hard reasoning
Just that change dropped my token cost by roughly 40% with zero noticeable quality drop in user-facing features.
2. Cache aggressively
If you're regenerating the same system prompt + static context on every call, you're paying rent twice. Most providers support prompt caching now. In Anthropic's API, prefix caching meant my 2,000-token system prompt stopped being billed at full rate after the first call.
client.messages.create(
model='sonnet',
system=[{
'type': 'text',
'text': SYSTEM_PROMPT,
'cache_control': {'type': 'ephemeral'}
}],
messages=messages
)
For my app, caching the system prompt alone saved about $90/month.
3. Trim your context window
I was sending full conversation history every time. Users don't need the 40-message backlog to get a good answer. I switched to keeping last 6 turns + a compressed summary. Less input tokens = direct savings.
4. Batch where you can
If you're processing things like backend jobs, user uploads, or nightly reports, batch APIs give meaningful discounts (often 50% off). I moved my document tagging to batch and stopped caring about latency for those.
5. Consolidate your API keys and providers
This one's less obvious. I was juggling three separate provider accounts, each with minimums and separate billing dashboards. When I needed to swap models mid-sprint, the friction meant I just kept using the expensive default.
I found https://xinghuo1300ai.com which aggregates 30+ models under one API key, and that removed the excuse of "switching is annoying." Now I can A/B a cheaper model on the same endpoint without reconfiguring auth or payment. It's not magic, but it made cost-aware routing something I'd actually do instead of putting off.
What the numbers looked like
| Change | Monthly savings |
|---|---|
| Model routing | $480 |
| Prompt caching | $90 |
| Context trim | $130 |
| Batch jobs | $200 |
Total: from $1,240 to ~$440. Not a heroic optimization—just boring, repeatable fixes.
One honest caveat: the router adds a little code complexity, and you need tests so a misclassified task doesn't silently degrade quality. I caught two bugs in the first week where 'summarize' was too aggressive and dropped key entities.
Where I landed
Six months in, the bill's stable and I actually understand what each dollar does. The shift wasn't about finding a secret cheap model—it was removing the lazy defaults. If you're on a similar ramp, audit one week of logs, tag each call by task, and you'll probably see the same pattern I did: most of your spend isn't on the hard problems.
Top comments (0)