There's a pattern I keep seeing across enterprise AI builds. Nobody talks about it because it's not a model problem - it's an architecture problem. And honestly, the teams making this mistake are doing everything right on the surface. Shorter prompts. Cheaper models where possible. Careful about what goes into context. It's just that none of that touches the real problem.
The real problem is structural. It's the decisions that were made or not made when the system was first designed. By the time you're optimizing prompts, you've already locked in 80% of your cost.
The eight moves that follow work at the structural level. That's where the money actually is.
1. You don't need GPT-4 for everything
Model Routing Architecture
This one sounds obvious until you look at how most production systems are built. Every single request - simple FAQ, complex reasoning, basic classification - routed to the same expensive model.
What you actually want is a layer that looks at the incoming request and asks: how hard is this, really? Simple stuff goes to a cheap model. Medium stuff to something mid-tier. Only the genuinely complex reasoning hits the expensive model. I've seen this change alone cut bills by 40 to 80 percent on day one. Not after weeks of optimization. Day one.
2. You're probably paying to answer the same question twice
Semantic Cache Architecture
In customer support, internal tools, knowledge assistants - a huge chunk of requests are near-identical to something that came in yesterday. Or an hour ago.
A semantic cache sits in front of your model and checks whether something similar has already been answered. If it has, it returns that response without touching the model at all. Redis plus embeddings similarity is the basic stack. It's not glamorous. But 60 percent fewer model calls is 60 percent fewer model calls.
3. RAG is only as good as what you put in
Precision Rag Pipeline
Everyone's doing retrieval-augmented generation now. Most people are doing it wrong.
The usual pattern is to retrieve a bunch of chunks and dump them all into context, hoping the model will figure out what's relevant. What you actually get is bloated token counts and a model that's distracted by noise. The fix is to retrieve more but send less. Pull 20 chunks, run them through a reranker that scores actual relevance, and only pass the top two or three to the model.
For the reranker, if you want something that just works out of the box, Cohere Rerank and Voyage AI are both solid. If you'd rather host your own and not pay per call, BGE Reranker v2 is the one I'd start with - it matches proprietary performance in most benchmarks. ColBERT is worth knowing about for high-precision use cases, and FlashRank is what you reach for when latency is the actual constraint.
The principle is simple: context quality beats context size every time.
4. Your prompts are too long
Prompt Compression Techniques
"Please carefully analyze the following and provide a detailed, well-structured response taking into account all relevant factors..." - that's 25 tokens that do nothing. Not because you're being careless. Because verbose prompts feel thorough. They don't perform better.
JSON schemas, reusable system prompt templates, instruction IDs instead of repeated full text. These aren't premature optimizations. They're just cleaner engineering. 20 to 50 percent token reduction with no quality loss is completely normal.
5. One big prompt is almost always the wrong design
Decomposed Pipeline
When a task feels complex, the instinct is to write one giant prompt that handles everything. That's usually the most expensive way to do it.
Break it into stages.
- First stage: extract intent, cheap model, pennies.
- Second stage: retrieve relevant data, no model cost at all.
- Final stage: the actual reasoning, expensive model, but only for the part that genuinely needs it.
Most of your volume hits the cheap stages. Only a fraction reaches the expensive one. Your cost curve looks completely different.
6. Agents that re-read their own history are burning your money
Memory Architecture for Context Control
Agentic systems have a quiet cost problem. Every turn, they re-send the full conversation history. At turn five that's fine. At turn thirty, you're sending thousands of tokens of context that mostly don't matter.
The architecture that fixes this is three layers of memory. A sliding window for recent turns. A vector database for older relevant context, retrieved on demand. And summarized episodes for longer-running sessions. The difference between a well-designed memory layer and a naive one is often $0.10 per session versus $10 per session. At any real volume, that's the difference between a viable product and one that can't scale.
7. If you're running the same task a thousand times a day, stop prompting it
Distillation Strategy
There's a point where it's cheaper to train a model on your specific task than to keep prompting a general-purpose one.
The playbook: run a frontier model on your task for a few weeks, collect the input-output pairs, fine-tune a smaller open-source model on them. What you end up with is a model that performs like Opus on your specific use case, at roughly Haiku pricing. Classification, extraction, structured output, domain-specific generation - these are the tasks where it pays off fastest.
8. You're generating tokens nobody reads
Streaming vs Full Generation
Most applications generate a complete response every time, even when the user gets what they needed from the first paragraph.
Stream your responses. Build simple logic to stop generation early when the task is done. For support bots and summarization tools especially, this is a quiet 20 to 30 percent reduction in output token costs with no user-facing change at all.
The actual mindset shift
The teams with $5K monthly AI bills and the teams with $50K monthly AI bills are often running similar models on similar tasks. The difference is almost never the model choice. It's whether someone sat down and asked: where in this system is intelligence being used when it doesn't need to be? That question not prompt engineering, not model selection is where the real leverage is.
Pick one of these eight things. Model routing or caching are the easiest starting points. Run it for 30 days and look at the numbers. The way you think about AI cost will be permanently different after that.
If this was useful, share it with someone who's building on LLMs and watching their cloud bill climb.
Satish Gopinathan is an AI Strategist & Enterprise Architect. More at eagleeyethinker.com
EnterpriseAI, AgenticAI, LLMArchitecture, AIStrategy, AICostOptimization, RAG, AIEngineering
Top comments (0)