LlamaIndex re-retrieves your chunks — and re-sends up to 75% of your context — on every chat turn

You give a LlamaIndex agent a couple of tools, call it, and it loops — reason, call a tool, read the observation, reason again — until it answers. Clean API, great retrieval story. The part the quickstart doesn't put a number on is what the agent re-sends to the model on each pass of that loop.

The default memory keeps everything, and hands the model all of it

LlamaIndex agents remember the conversation through ChatMemoryBuffer. When the agent builds the next model call, it asks memory for the history:

def get(self, input=None, initial_token_count=0, **kwargs) -> List[ChatMessage]:
    chat_history = self.get_all()
    ...

get_all() is the entire accumulated buffer — every user turn, every assistant reasoning step, and every tool observation you've collected so far. That whole thing is the starting point for what gets sent. The buffer only starts dropping messages once it crosses a ceiling:

while token_count > self.token_limit and message_count > 1:
    ...

Below the ceiling, nothing is trimmed. The full history goes to the model, on every step.

And the ceiling is not small

Here's the number that surprises people. When you build memory with an LLM attached (the normal path), the ceiling is a fraction of the model's context window:

DEFAULT_TOKEN_LIMIT_RATIO = 0.75
token_limit = token_limit or int(context_window * DEFAULT_TOKEN_LIMIT_RATIO)

On a 128k-context model that's a ~96,000-token memory budget. So the buffer is allowed to grow to ~96k tokens of accumulated history — including fat tool outputs — and re-send up to that on each agent step. It isn't a runaway that grows forever; it's a very high floor that gets paid over and over. Ten steps of a tool-heavy agent don't add ten small increments — they re-send a large, near-constant buffer ten times.

That's the first mechanism. Cost tracks roughly buffer size × number of steps, and the default sizes the buffer at 75% of your context window.

And in a RAG chat, retrieval re-runs every turn

The buffer is one half. The other shows up in ContextChatEngine — what index.as_chat_engine() gives you by default. Its .chat() doesn't retrieve once and reuse; it re-runs the retriever against every new message (chat_engine/context.py on main):

def chat(self, message, chat_history=None, prev_chunks=None):
    ...
    nodes = self._get_nodes(message)   # -> self._retriever.retrieve(message)

Those freshly-retrieved chunks get re-stuffed into the system prompt each turn via the engine's template ("{context_str}"), with no dedup against what you already sent. In a follow-up-heavy chat ("and what about X?", "summarize that"), the retriever frequently pulls the same top-k chunks again — and you pay to upload them again, on top of the growing history buffer above. Two independent re-sends, stacking, every turn.

The knobs are real — set them on purpose

This isn't a bug and it isn't a strawman: LlamaIndex gives you the lever right there. Cap the buffer explicitly instead of inheriting the 0.75-of-context default:

from llama_index.core.memory import ChatMemoryBuffer

memory = ChatMemoryBuffer.from_defaults(token_limit=1500)

or move to a summarizing/vector memory so old turns and stale tool observations stop riding along. And for the re-retrieval half, CondenseQuestionChatEngine condenses history into one standalone query so you retrieve on the condensed intent instead of re-stuffing chunks each turn, or lower similarity_top_k. The point isn't "LlamaIndex is expensive" — it's that both the default re-send size and the per-turn re-retrieval are decisions the framework makes for you, and they compound.

Measure it before you argue about it

Before you tune anything, put a dollar figure on one real run — priced, not guessed. That's what @wartzar-bee/tokenscope does (npm i @wartzar-bee/tokenscope): it takes real usage and prices each bucket — input, output, cache-write (~1.25×), cache-read (~0.1×) — into an actual per-run cost, so "the deep agent costs N× the shallow one" stops being a hunch.

If it runs in CI, gate it: wartzar-bee/ci-guardrail is an Apache-2.0 GitHub Action (built on tokenscope) that fails the check when a run crosses an absolute max-usd ceiling — so a memory-config change doesn't ship as a silent 4× before anyone notices.

- uses: wartzar-bee/ci-guardrail@v1
  with:
    max-usd: "0.50"

If you run LlamaIndex chat or agents: what's your token_limit, how big are the tool observations in that buffer, and how many of the same chunks get re-retrieved across a real session? Worth pricing one real conversation before the next invoice does it for you.