Token Drift Explained: Why Your Agent Gets Slower and More Expensive

Your agent feels fast during a demo. Then a real session reaches twenty turns, several tools have returned large payloads, and every response starts taking longer and costing more.

This pattern is often called token drift: the effective input context grows as an agent carries more conversation history, tool output, retrieved documents, and state into each model call. The model itself is not gradually becoming less efficient. The application is asking it to process more material on every turn.

Token drift is manageable, but only if context is treated as a budgeted system resource rather than an unlimited transcript.

What Token Drift Actually Means

Most conversational agents build each request from several sources: a system prompt, tool definitions, recent messages, retrieved context, durable memory, and sometimes a summary of older work. Whether the application sends that state on every request or a provider manages part of it, the model still has an effective context to process.

Consider an illustrative session:

Turn	Effective input	What changed
1	1,200 tokens	System prompt, tools, and one user message
8	6,900 tokens	Conversation history and two tool results
20	18,400 tokens	More history, retrieved documents, and accumulated state

The exact price and latency depend on the model, provider, cache behavior, and workload. The important signal is the trend: later calls repeatedly process a larger context.

Why Agents Accumulate Tokens So Quickly

Conversation history is only one source of growth. Production agents often accumulate tokens in several places at once:

• Repeated transcripts: Every prior user and assistant message remains in context.
• Tool schemas: Large tool descriptions and JSON schemas may be attached to every model call.
• Tool results: Search results, stack traces, database rows, and API responses can be much larger than the user's request.
• Retrieved documents: RAG pipelines sometimes add too many chunks or keep stale retrieval results across turns.
• Retries and repairs: Failed tool calls and validation errors may be appended even after they stop being useful.
• Duplicated memory: A summary, structured state, and raw transcript may all contain the same facts.

This is why limiting the number of chat messages is useful but incomplete. A ten-message conversation can still be expensive if one tool returned 30,000 tokens of JSON.

The Cost Curve

Suppose a request has a fixed cost of s tokens for system instructions and tools, and each turn adds roughly m tokens of new history. The input on turn t is approximately:

input(t) = s + (t * m)

Per-turn input grows roughly linearly. However, the cumulative input processed across n turns is approximately:

session input = (n * s) + m * n * (n + 1) / 2

That second term is quadratic. This distinction matters: the final request is not necessarily exponentially larger, but repeatedly resending a growing transcript can make the total session cost rise much faster than the number of turns.

Prompt caching can reduce the cost of repeated prefixes on providers that support it. It does not remove context-window limits, irrelevant-history problems, or the need to control tool and retrieval payloads.

Measure Drift Before You Trim

Use the token usage returned by the model API whenever possible. Character-based estimates are acceptable for admission control, but they are not a reliable billing metric across models or languages.

Track usage per model call, not only per user request. One agent turn may contain planning, tool calls, retries, and a final response.

type UsageSample = {
  turn: number;
  step: string;
  inputTokens: number;
  outputTokens: number;
  cachedInputTokens?: number;
  durationMs: number;
};

function averageInputGrowth(samples: UsageSample[]): number {
  if (samples.length < 2) return 0;

  const byTurn = new Map<number, number>();
  for (const sample of samples) {
    byTurn.set(
      sample.turn,
      (byTurn.get(sample.turn) ?? 0) + sample.inputTokens,
    );
  }

  const totals = [...byTurn.entries()]
    .sort(([a], [b]) => a - b)
    .map(([, tokens]) => tokens);

  if (totals.length < 2) return 0;

  const deltas = totals.slice(1).map((value, index) => {
    return value - totals[index];
  });

  return Math.round(
    deltas.reduce((sum, delta) => sum + delta, 0) / deltas.length,
  );
}

A useful trace should also record the model, operation, tool name, retry count, retrieval chunk count, and whether cached input was used. Avoid recording sensitive prompt content unless your privacy policy explicitly permits it.

Use a Context Budget, Not a Message Limit

A sliding window is a reasonable fallback, but a token budget is the stronger control because message sizes vary dramatically. Reserve space for the output and for any tool calls the model may make, then allocate the remainder to fixed context and recent turns.

Treat a complete agent turn as the unit of removal. Dropping an individual tool result while keeping the assistant message that refers to it can leave an invalid or confusing transcript.

type AgentMessage = {
  role: 'user' | 'assistant' | 'tool';
  content: string;
};

type ConversationTurn = {
  id: string;
  messages: AgentMessage[];
  tokenCount: number;
};

type ContextPlan = {
  turns: ConversationTurn[];
  omittedTurnIds: string[];
  estimatedInputTokens: number;
};

type ContextBudget = {
  maxContextTokens: number;
  reservedOutputTokens: number;
  systemPromptTokens: number;
  toolSchemaTokens: number;
  summaryTokens: number;
  retrievedContextTokens: number;
};

function planContext(
  turns: ConversationTurn[],
  budget: ContextBudget,
): ContextPlan {
  const fixedTokens =
    budget.systemPromptTokens +
    budget.toolSchemaTokens +
    budget.summaryTokens +
    budget.retrievedContextTokens;

  let remaining =
    budget.maxContextTokens -
    budget.reservedOutputTokens -
    fixedTokens;

  if (remaining <= 0) {
    throw new Error('Fixed context exceeds the available input budget');
  }

  const selected: ConversationTurn[] = [];

  for (let index = turns.length - 1; index >= 0; index -= 1) {
    const turn = turns[index];

    if (turn.tokenCount > remaining) break;

    selected.unshift(turn);
    remaining -= turn.tokenCount;
  }

  if (selected.length === 0 && turns.length > 0) {
    throw new Error('The latest turn is larger than the context budget');
  }

  const selectedIds = new Set(selected.map((turn) => turn.id));
  const omittedTurnIds = turns
    .filter((turn) => !selectedIds.has(turn.id))
    .map((turn) => turn.id);

  return {
    turns: selected,
    omittedTurnIds,
    estimatedInputTokens:
      fixedTokens +
      selected.reduce((sum, turn) => sum + turn.tokenCount, 0),
  };
}

The token counts in this example should come from the provider's tokenizer or a compatible tokenizer for the chosen model. Recalculate the plan whenever tools, retrieval results, or the summary changes.

Separate Conversation From State

Not every fact belongs in the transcript. Long-running agents become more reliable when they store important state explicitly:

type TaskState = {
  objective: string;
  constraints: string[];
  decisions: Array<{
    choice: string;
    reason: string;
  }>;
  openQuestions: string[];
  artifactIds: string[];
};

Structured state is compact, inspectable, and easier to validate than a prose summary. It also prevents a critical decision from disappearing simply because an old message fell outside the recent window.

A practical context often has four layers:

1. Stable instructions: The system prompt and safety constraints.
2. Structured task state: Goals, decisions, identifiers, and unresolved work.
3. Compressed history: A summary of older turns.
4. Recent turns: The verbatim messages and tool exchanges needed for continuity.

This layered design is usually more useful than importance-scoring individual messages. Heuristic scores can be difficult to explain, and they may discard a quiet but essential constraint.

Summarize Deliberately

Summarization is valuable when sessions last long enough that a recent window loses necessary context. It should not run blindly on every turn.

Create or refresh a summary only when older turns cross a threshold. Ask the summarizer to preserve facts, decisions, constraints, failures, unresolved questions, and references to external artifacts. Store the summary separately from the raw transcript so it can be reviewed or regenerated.

Summaries are lossy and can introduce errors. Keep canonical values such as account IDs, approval state, financial amounts, and file paths in structured storage rather than trusting generated prose.

Bound Tools and Retrieval Too

Many apparent memory problems are actually payload problems. A useful context policy should also:

• Send only the tool definitions available in the current state.
• Replace large tool results with a compact, typed projection.
• Store full results externally and keep an ID or URL in context.
• Limit retrieval by both relevance and token budget.
• Deduplicate overlapping chunks before adding them to the prompt.
• Remove failed retries once their diagnostic value has expired.

For example, an agent rarely needs an entire database response. It may need five selected fields, the result count, and an identifier it can use to fetch more data later.

Choose a Strategy by Workload

Workload	Good starting strategy
Short support chat	Recent-turn window with a hard input budget
Tool-heavy workflow	Compact tool schemas and bounded results
Long research session	Structured state, summary, and recent turns
Knowledge assistant	Token-budgeted retrieval plus recent turns
High-stakes workflow	Structured state with validation and auditable summaries

Most systems do not need a sophisticated memory-ranking algorithm on day one. They need visibility, a budget, and clear rules for what may enter context.

Production Quality Gates

Context reduction is successful only if it lowers resource use without damaging task quality. Compare the policy against representative sessions and track:

• Input tokens per turn and per completed task
• Cached versus uncached input tokens
• Model-call count, retries, and tool failures
• Median and tail latency
• Task completion and human correction rates
• Missing facts or constraints after compaction
• Summary refresh frequency and cost

Run deterministic regression cases for critical facts and tool sequences. For example, verify that an agent still remembers an approval constraint after the originating turn has moved into summarized history.

A Practical Baseline

Start with a simple policy: measure real usage, reserve output headroom, cap the input context, keep structured task state, retain a small window of complete recent turns, and summarize older history only when necessary. Bound tool and retrieval payloads independently.

Token drift is not mysterious model degradation. It is accumulated application state. Once that state is measured and budgeted, cost and latency become predictable engineering trade-offs instead of late-session surprises.

Comments

[Tae Kim]

The "context as a budgeted resource" frame is right, but the underappreciated part is that tool outputs are usually the fastest-growing line item and the easiest to control. Raw tool results often contain boilerplate, headers, or pagination metadata the model will never use; a distillation step that extracts only the fields the next reasoning step actually needs can cut per-turn token growth by 40 to 60 percent in practice. Conversation history compression gets discussed more often, but it addresses a slower-growing source. Budget the tool output layer first, because that is where a single bad-day tool call can dump four thousand tokens into your context before the turn even starts.