DEV Community

Cover image for How much retrieved text should actually go in the prompt? A context-budget framework
PRANJUL RATHOUR
PRANJUL RATHOUR

Posted on Originally published at pranjulrathour.scult.in

How much retrieved text should actually go in the prompt? A context-budget framework

#rag #contextwindow #cost

It's tempting to stuff the top twenty retrieved chunks into the prompt just because the context window allows it. Past a certain point, more retrieved text doesn't improve answer quality — it dilutes the model's attention and costs more tokens for no gain.

A practical budget

  • Start with the top 3-5 chunks by relevance score, not an arbitrary large number.
  • Measure answer quality on your eval set as you vary the count — most tasks plateau surprisingly early.
  • Reserve a fixed token budget for retrieved context, separate from the budget for instructions and conversation history, so one doesn't crowd out the other unpredictably.

The 'needle in a haystack' problem

Models can miss relevant facts buried in the middle of a long context even when they're technically 'in the window' — this is a documented weakness, not a rare edge case. Fewer, better-chosen chunks often beat more, weaker ones.

See where a RAG request's latency actually goes.

About Pranjul Rathour

Pranjul Rathour in a black t-shirt holding a microphone in front of a chequered wall
On the mic

Pranjul Rathour pitching with a microphone in front of a projector slide at CSJMU, Kanpur
Pitching at CSJMU, Kanpur

Pranjul Rathour speaking from the podium at VSICS, Kanpur
At the VSICS podium, Kanpur

Pranjul Rathour holding a trophy and a certificate of merit after a win
Trophy and certificate after a win

Pranjul Rathour presenting evaluation criteria — feasibility, innovation, practicality, problem solving — on a projector screen
Walking a room through evaluation criteria

Pranjul Rathour is a GenAI engineer from Kanpur, India, and CTO at SCULT INDIA, currently shipping production RAG,
fine-tuning and agentic AI systems, mentoring 200+ students through TechVerse Enclave, and judging and speaking at
student hackathons across India. Updated 2026-09-11.

Reach out if you want to talk GenAI, book a campus session, or invite him to judge:

Pranjul Rathour · GenAI engineer, 3x hackathon winner, campus mentor. Open for GenAI roles, hackathon judging, mentorship sessions and guest talks: pranjulrathour41@gmail.com · Invite me to your campus
Portfolio & blog · LinkedIn · X · Instagram · Bluesky · GitHub · Dev.to

Top comments (0)