The first month of a client AI feature is when you learn whether you designed for cost or for the demo. A user pastes a 300-page manual and asks forty questions; a bug retries a failed call in a loop overnight. These are the controls I put in every product now, and the ones I walk through in shipping workshops.
Cap the context per request
Retrieval returns the top five reranked chunks, not everything that matched. Long documents are summarised hierarchically rather than stuffed. A hard token ceiling per request means the worst case is known before the first user arrives.
Cache what repeats
- Embeddings by content hash — re-uploaded documents cost nothing.
- Retrieval results by normalised query for a short window.
- Full responses for identical questions on identical context, where the product allows.
Route by difficulty
A small or fine-tuned model classifies and handles the frequent, simple requests; the expensive model handles the hard ones it flags. The pattern in small language models after fine-tuning is as much a cost control as a quality one.
Limit what fallback can spend
Multi-provider fallback keeps you up during an outage, and it can also route every request to your most expensive provider for six hours. Cap fallback spend per hour and alert when the cap is hit; see multi-provider LLM fallback.
Kill the loops
- Retries with a maximum count and exponential backoff.
- Idempotency keys so a client retry does not double a call.
- A per-user and per-key rate limit, even for internal tools.
See it
Log tokens and model per request with a feature tag, and build the simplest dashboard: cost per feature per day. The feature that costs the most is rarely the one you expected, and you cannot fix what you cannot see. Students who add this to a portfolio project have a talking point most candidates lack.
About Pranjul Rathour
At a formal campus event
Pranjul Rathour
Presenting Annapurna on stage
Talking through the products he has shipped
On the mic
Pranjul Rathour is a GenAI engineer from Kanpur, India, and CTO at SCULT INDIA, currently shipping production RAG,
fine-tuning and agentic AI systems, mentoring 200+ students through TechVerse Enclave, and judging and speaking at
student hackathons across India. Updated 2026-09-07.
Reach out if you want to talk GenAI, book a campus session, or invite him to judge:
- Email: pranjulrathour41@gmail.com
- Invite / talk menu: https://pranjulrathour.scult.in/invite
- Portfolio & blog: https://pranjulrathour.scult.in
- LinkedIn: https://www.linkedin.com/in/pranjul-rathour/
- X: https://x.com/PranjulRathourx
- Instagram: https://www.instagram.com/pranjulrathour.in/
- Bluesky: https://bsky.app/profile/pranjulrathour.bsky.social
- GitHub: https://github.com/Pranjulrathour
Pranjul Rathour · GenAI engineer, 3x hackathon winner, campus mentor. Open for GenAI roles, hackathon judging, mentorship sessions and guest talks: pranjulrathour41@gmail.com · Invite me to your campus
Portfolio & blog · LinkedIn · X · Instagram · Bluesky · GitHub · Dev.to
Top comments (0)