How Context Window Degradation Breaks Long-Running Agents in Production—and How to Engineer Your Way Out

LLMs Lose Skills Quietly Before Truncation

Large context windows are overrated. A bigger window doesn't mean the model uses all of it well. Research on long-context behaviour—Lost in the Middle (Liu et al., 2023), RULER (Hsieh et al., 2024), and Chroma's Context Rot study—shows that model performance degrades as input grows, well before the advertised limit.

The failure mode in agents isn't chopped sentences. Agents still finish workflows, but core behaviours—following earlier instructions, using tools, chaining steps—erode quietly.

Illustrative agent log: a 4-hour tutoring session

[12:02] User: "Generate and solve a quadratic equation using the student's preferred method."
[12:03] Agent: "Here's a quadratic equation: x^2 + 5x + 6 = 0. Solving by completing the square..."
# [Correct process, correct answer]

...
[15:41] User: "Show another example, using the student's preferred method."
[15:41] Agent: "Here's a quadratic: 2x^2 + x - 3 = 0. I will use the quadratic formula..."
# [Incorrect: the agent ignores the preferred method established earlier]

The pattern is consistent: early in a session, persona, constraints and tool use hold. Deep into a long session, agents drift toward generic completions.

In summary: context saturation erodes instruction-following and multi-turn skills. It rarely shows up in standard metrics dashboards, because most evals test short interactions.

Context Rot Exposed in Task-Level Metrics

Context rot becomes visible when you test at the session level:

• Tool Amnesia: The agent stops using available tools after enough dialog.
• Hallucinated State: The agent responds about context that doesn't exist.
• Workflow Skip: Instructions or tasks from earlier turns quietly disappear.

Illustrative integration test output:

test_name: "context_persistence_workflow"
steps:
  - input: "Schedule a follow-up using the calendar tool."
    expected_tool: "calendar"
    position_in_session: "early"
    outcome: "Success"
  - input: "Now email this to John."
    expected_tool: "email"
    position_in_session: "late"
    outcome: "Failed - Agent replied with freeform text, ignored tool."
  - input: "Which tasks are pending?"
    position_in_session: "late"
    outcome: "Agent omits the follow-up item scheduled earlier."

The exact token count where this starts varies by model and task—modern models hold up far longer than older ones—but the direction is the same: the further a constraint sits from the current turn, the less reliably it's honoured.

Summarization Pipelines Fail Under Real Load

Off-the-shelf agent stacks often manage context with summarization: truncate, condense, prepend summary, repeat. Each pass loses precision.

Illustrative summarization failure

[Summary at 3K tokens]: "The session involves two students learning advanced calculus and preparing for the next exam."
[Later user turn]: "Can you show Sasha's substitutions on the integral from earlier?"
[Agent]: "There is no record of substitutions by Sasha."
# Data lost: the summary erased a specific sub-dialog

Summarization collapses on exactly the cases that matter: when the user references a specific earlier detail, or expects the agent to keep honouring a persistent constraint. That's a systemic data-loss problem, not something a better summary prompt fixes.

Deterministic Memory: Pin What Must Never Be Lost

Summarization is lossy by design. The alternative is deterministic memory: explicit rules for what stays in context, rather than hoping a summary keeps it.

The key distinction is between two kinds of context:

• Pinned context — user constraints, preferences, open workflows. Never evicted.
• Rolling context — recent dialog and tool results. Evicted oldest-first when the budget runs out.

A plain recency buffer isn't enough on its own: it evicts the oldest blocks first, and the oldest blocks are often the instructions you most need to keep.

deterministic_memory.py

from collections import deque


class DeterministicMemory:
    """Pinned blocks are always kept; rolling blocks are evicted oldest-first."""

    def __init__(self, max_tokens=2048):
        self.max_tokens = max_tokens
        self.pinned = []          # (block, token_count) — never evicted
        self.rolling = deque()    # (block, token_count) — FIFO eviction

    @property
    def current_tokens(self):
        return sum(t for _, t in self.pinned) + sum(t for _, t in self.rolling)

    def pin(self, block, token_count):
        """Use for constraints, preferences and open workflows."""
        self.pinned.append((block, token_count))
        self._enforce_limit()

    def add(self, block, token_count):
        """Use for ordinary dialog turns and tool results."""
        self.rolling.append((block, token_count))
        self._enforce_limit()

    def _enforce_limit(self):
        while self.current_tokens > self.max_tokens and self.rolling:
            self.rolling.popleft()
        if self.current_tokens > self.max_tokens:
            raise ValueError("Pinned context alone exceeds the token budget.")

    def retrieve(self):
        # Pinned blocks go first so they sit at a stable, prominent position.
        return [b for b, _ in self.pinned] + [b for b, _ in self.rolling]


# Usage: build the agent's context from memory on every turn
memory = DeterministicMemory(max_tokens=1500)
memory.pin("User preference: always round results to 2 decimals.", 15)
memory.add("Tool call: calculator used at step 3.", 20)
# ... many more turns ...
agent_context = "\n".join(memory.retrieve())

Before/after (illustrative)

# Before (summarization or plain FIFO):
User: "Always round to 2 decimals from now on."
[many turns later]
Agent: "The result is 2.3167"

# After (pinned constraint):
Agent: "The result is 2.32"

Pinning guarantees the constraint is present in every prompt. Presence doesn't guarantee compliance—which is why you still need the tests below—but it removes the most common cause of failure.

Embedding-Aware Retrieval Outperforms Recency Alone

Not everything can be pinned. For the large middle ground of past context, tag dialog blocks by skill or task and use embedding similarity to pull back what's relevant to the current turn. This scales to long sessions where recency heuristics fail.

Minimal embedding-aware selection using SentenceTransformers:

from sentence_transformers import SentenceTransformer, util

model = SentenceTransformer("all-MiniLM-L6-v2")

context_blocks = [
    {"text": "User prefers examples using the quadratic formula.", "tag": "method"},
    {"text": "Current user topic: advanced calculus.", "tag": "topic"},
    {"text": "Open workflow: schedule meeting.", "tag": "workflow"},
]

# Encode once and cache. In production, store embeddings alongside each
# block (or in a vector store) instead of re-encoding on every turn.
block_embeddings = model.encode([b["text"] for b in context_blocks], convert_to_tensor=True)


def select_context(query, blocks, embeddings, top_k=2):
    query_embedding = model.encode(query, convert_to_tensor=True)
    similarities = util.cos_sim(query_embedding, embeddings)[0]
    top_indices = similarities.argsort(descending=True)[:top_k]
    return [blocks[int(i)]["text"] for i in top_indices]


selected = select_context("quadratic equation example", context_blocks, block_embeddings)
print(selected)
# e.g. ['User prefers examples using the quadratic formula.', ...]

In practice, combine the two: pinned blocks always, retrieved blocks for relevance, and a small recent-turns buffer for conversational flow.

Test What You Truncate—Don't Assume Context Strategies Work

Few teams benchmark long conversations. Most test single input/output pairs, which can't catch context rot by construction.

Build these into your CI:

1. Instruction Persistence Test: Record explicit user constraints ("always round decimals"), replay long sessions, and assert the constraint still holds after compaction or truncation.
2. Tool Invocation Regression: Track tool-call rate against session position; alert if usage drops as sessions get longer.
3. Context Regression Harness:

test_context_rot.py

import pytest

FENCE = "`" * 3  # a Markdown code fence


@pytest.mark.parametrize("strategy", ["summarization", "fifo", "pinned_plus_retrieval"])
def test_instruction_persistence(make_agent, strategy):
    agent = make_agent(context_strategy=strategy)
    agent.feed("Always format answers as Markdown code blocks.")
    for _ in range(50):  # simulate a long session
        agent.feed("Give a random math example.")
    final_output = agent.feed("Continue.")
    assert FENCE in final_output, f"[{strategy}] lost the Markdown formatting instruction"

Run the same suite across context strategies. Track failure rates and the median session length to first failure—that number is your real context budget, regardless of what the model card says.

Engineered Context Eliminates Functional Drift in Practice

Naive summarization session (illustrative):

Turn 1:  User:  "Always explain with visual analogies."
Turn 15: Agent: "Here is a formula..."
Turn 35: Agent: "The answer is x = 25"   # instruction lost in summarization

Engineered session — pinned constraints + embedding-aware retrieval (illustrative):

Turn 1:  User:  "Always explain with visual analogies."
Turn 15: Agent: "Imagine this as stacking boxes—here is the formula..."
Turn 35: Agent: "Visual: think of x = 25 as the height of the stack."

---

LLM agents degrade as context grows—often silently, and well before the window is full. This is a software engineering problem, not a prompt engineering one. Pin what must persist, retrieve what's relevant, and stop trusting summaries with constraints. Then build session-level regression tests and read your logs: if you aren't engineering against context rot, it's already in your system.

Further reading

• Liu et al., Lost in the Middle: How Language Models Use Long Contexts (2023)
• Hsieh et al., RULER: What's the Real Context Size of Your Long-Context Language Models? (2024)
• Chroma Research, Context Rot: How Increasing Input Tokens Impacts LLM Performance

Comments

[ai_adam]

Đúng trọng tâm. Mình từng gặp case agent chạy 3-4 ngày liên tục thì bắt đầu hallucinate policy logic vì context bị tràn — system prompt quan trọng bị đẩy ra khỏi window mà không có cơ chế reclaim. Giải pháp thực tế team mình áp dụng: chia nhỏ thành episodic memory (vector store với TTL) + semantic compression định kỳ (summarize old turns thành bullet points, giữ lại decision rationale thay vì raw text). Quan trọng hơn: phải có context budget guard — set hard limit token cho mỗi component (system, history, RAG, tool output) và reject/trim proactively thay vì để model tự cắt. Có bạn nào thử approach hierarchical context (global + session + task-level) chưa? Mình thấy nó clean hơn sliding window thuần túy cho workflow multi-step (site: labagent .tech)

[Manoj Kumar Kagitha]

The tool amnesia failure mode is the one that hurts most in production. Your instruction persistence test is a solid addition, that is the exact class of regression most teams never think to write.