DEV Community

Manohar Papasani
Manohar Papasani

Posted on

What role does context window have and how do you manage it given XYZ constraints?

#aiinterview #genai #llm #rag

Asked in a Gen AI engineer interview.

What they are really testing: whether you treat the context window as a scarce, priced resource with failure modes (lost-in-the-middle, latency, cost), and whether you can budget it deliberately instead of stuffing everything in.

A strong answer:

  • Start with a token budget per request: system prompt, tool schemas, conversation history, retrieved evidence, and reserved output tokens. Name the constraint you are working under (max context, latency target, cost per call) and allocate against it.
  • Keep the system prompt and tool definitions stable and short so they can be prompt-cached, and put volatile content after them.
  • Manage conversation memory in tiers: keep recent turns verbatim, compress older turns into a rolling summary, and persist durable facts (user preferences, decisions) in an external store that is retrieved on demand.
  • For retrieved context, use retrieval plus reranking and pass only the top few chunks (often 3 to 8 after reranking) rather than the top-k raw hits. Place the most relevant evidence near the start or end of the prompt, since models attend less reliably to the middle.
  • Enforce hard limits in code: count tokens with the model's own tokenizer, truncate or drop lowest-priority sections first, and fail gracefully when the budget is exceeded.
  • Measure the effect: track answer quality on an eval set against context size, p95 latency, and cost per request, and tune the budget from those numbers.

Common mistakes:

  • Treating a 1M-token window as permission to dump entire documents, which raises cost and latency and often lowers accuracy.
  • Truncating from the end of the prompt by accident, silently cutting the user's question or the output instructions.
  • Summarizing conversation history once and never revisiting it, so errors and stale facts compound across turns.

How would you split a fixed budget between history, retrieval and output for a support agent? Share your approach below.

Originally shared on LinkedIn.

An AI interview question every day, leadership for tech people and hands-on AI learning:

Top comments (0)