Over the past year, LLMs became the backbone of our apps—from chatbots to AI assistants. The trend is simple: more features mean bigger prompts.That sounds harmless until you look at your AI bill.
While building AI features for Fanziz—our sports platform handling personalized news, semantic search, and live commentary—we hit this exact wall. Growing prompts meant spiking latency and inference costs.
Instead of upgrading models, we asked: How can we make our LLM smarter without sending it more data?
Here are the 6 engineering shifts that saved our pipeline.
1. Retrieve Context, Don't Dump It
Sending dozens of articles or historical records in a prompt is a rookie mistake. We implemented Retrieval-Augmented Generation (RAG). Articles are embedded in a vector database, and requests fetch only the most relevant snippets.
Smaller prompts = faster inference, better responses, and lower costs.
2. Stop Regenerating Static Context
We noticed background instructions, platform context, and reference data were being rebuilt for every request.
• The fix: A cached context strategy. We generate static blocks once and reuse them across requests, slashing token usage and response times.
3. Match Prompting Strategy to Complexity
Not every task needs examples. We stopped defaulting to heavy few-shot prompts and tailored strategy to the task:
• Simple transformations: Zero-shot
• Moderate reasoning: One-shot
• Complex workflows: Few-shot
4. Modularize System Prompts
As features grew, our system prompt became a giant, unmaintainable monolith. We split responsibilities into modular blocks (Safety, Formatting, Domain Rules, Output Style) to keep prompts lean, clean, and predictable.
5. Not Every Request Needs an LLM
LLMs shouldn’t solve everything. For intent detection, language recognition, or simple classification, we route to traditional, lightweight NLP models.
• The best AI architecture uses the right tool for the job.
6. Measure Tokens Like CPU
We track CPU, memory, and latency religiously—so why not tokens? Once we started monitoring input/output tokens, cost-per-request, and cache hit rates, optimization became second nature.
Final Thoughts
These techniques power real-time experiences inside Fanziz today, proving you don't need the biggest models to build great AI products—you just need the most efficient systems.
Next time you build, don't just ask: "Which model should I use?"
Ask: "Does this request even need an LLM?".
Top comments (0)