CompactionRL: Breaking the Context Barrier for Agentic LLMs in Long-Horizon Tasks

Large Language Models (LLMs) have revolutionized many aspects of AI, but a persistent challenge for their deployment in complex, long-running applications has been the finite context window. This limitation means that as an LLM agent interacts with an environment, generating more and more tokens, it eventually 'forgets' earlier parts of the conversation or task history. This bottleneck is particularly critical for agentic LLMs tasked with executing multi-step, long-horizon operations, where maintaining a comprehensive understanding of past actions and observations is paramount.

Imagine an AI agent trying to debug a complex codebase over several hours. It needs to remember previous errors, past attempts at fixes, and the overall architectural context. If its memory (context window) is limited, it might forget crucial details, leading to inefficient loops, repeated mistakes, or an inability to complete the task. This is the fundamental problem CompactionRL aims to solve.

The Bottleneck of Finite Context Windows

At the core of an LLM's operation is its ability to process a sequence of tokens—its 'context window.' This window has a fixed maximum length, typically ranging from a few thousand to hundreds of thousands of tokens, depending on the model. While these capacities are impressive, they are often insufficient for truly autonomous agents engaging in long-horizon tasks.

Long-horizon tasks are characterized by:

• Multiple steps: Requiring a sequence of actions and decisions over time.
• Dependencies: Later steps often depend on information or outcomes from earlier steps.
• Exploration and Re-planning: Agents may need to try different approaches, backtrack, and adapt their plans based on new information.

When an agent's interaction trajectory—the sequence of observations, thoughts, and actions—exceeds the context window, the oldest parts of the history are truncated. This loss of information can severely degrade an agent's performance, making it difficult to maintain coherence, learn from past mistakes, or complete complex tasks effectively. For agentic LLMs, which are designed to perform tasks autonomously, this memory limitation is a significant hurdle.

Introducing CompactionRL: A Novel Reinforcement Learning Strategy

To address this critical limitation, researchers have introduced CompactionRL, a novel reinforcement learning (RL) strategy specifically designed to train long-horizon agentic LLMs with integrated context compaction. The core idea is simple yet powerful: instead of discarding old information, summarize it.

CompactionRL integrates context summarization directly into the reinforcement learning loop. This isn't just a separate summarization step; it's a process where the LLM learns how to summarize its past states in a way that is most beneficial for future task execution. This intelligent compaction allows the agent to retain the most salient information from its extended history, effectively 'breaking' the traditional context window limits.

How Context Compaction Works

Traditional context management often involves a sliding window or simple truncation. CompactionRL takes a more sophisticated approach:

1. Summarization Module: An integral part of the agent learns to generate concise summaries of past interaction segments.
2. Reinforcement Learning Loop: The agent is trained using RL to optimize for task completion. Crucially, the summarization module is also optimized within this loop.
3. Dynamic Context: The LLM's effective context then comprises recent raw interactions plus intelligently compacted summaries of older interactions.

This dynamic and learned approach to context management ensures that the agent always has access to the most relevant information, regardless of how long the task trajectory becomes.

The Power of Joint Optimization

The key innovation behind CompactionRL's success lies in its joint optimization strategy. Rather than training the task execution model and the summarization model separately, CompactionRL optimizes both simultaneously. This ensures that the summaries generated are not just generic, but specifically tailored to aid the agent in completing its primary task.

This joint optimization is achieved through two primary mechanisms:

1. Token-Level Loss Normalization

In standard LLM training, losses are typically calculated per token. With compacted trajectories, directly applying these losses can be problematic because the compacted states are not raw, sequential input. CompactionRL introduces token-level loss normalization, which allows the agent to effectively learn from these compacted, extended trajectories. This technique ensures that the learning signal remains strong and relevant, even when dealing with summarized historical information.

2. Cross-Trajectory Generalized Advantage Estimation (GAE)

Reinforcement learning often relies on Generalized Advantage Estimation (GAE) to efficiently estimate the advantage of taking certain actions. CompactionRL extends this concept to cross-trajectory GAE. This means that the agent's learning signal is not just derived from a single interaction sequence but considers advantages across multiple interaction trajectories. This is particularly beneficial for long-horizon tasks where individual steps might not immediately yield rewards, and understanding the long-term impact of actions across different attempts is crucial.

By jointly optimizing task execution and summary generation through these sophisticated techniques, CompactionRL enables LLM agents to learn effectively from vastly longer interaction histories than previously possible. This represents a significant leap in overcoming the inherent constraints of current LLM architectures for complex, multi-step reasoning.

Empirical Evidence: Boosting Performance on Coding Tasks

The theoretical advantages of CompactionRL are strongly supported by its empirical performance, particularly on challenging agentic coding tasks. The research demonstrated consistent and significant performance gains across various open-source LLMs.

When applied to the open GLM-4.5-Air model (106B-A30B), CompactionRL achieved impressive absolute improvements:

• 7.0 points on SWE-bench Verified, reaching 66.8% Pass@1.
• 3.1 points on Terminal-Bench 2.0, reaching 24.5% Pass@1.

These benchmarks are critical for evaluating coding agents:

• SWE-bench Verified: A challenging dataset requiring agents to resolve real-world software bugs, often involving complex reasoning and long interaction histories.
• Terminal-Bench 2.0: Focuses on tasks executed within a terminal environment, simulating real-world developer workflows and requiring robust command-line interaction and problem-solving.

Further enhancements were observed with the more compact GLM-4.7-Flash model (30B-A3B), demonstrating that the benefits are not limited to larger models. This model saw Pass@1 scores boosted by:

• 5.5 points on SWE-bench Verified, reaching 56.0%.
• 6.8 points on Terminal-Bench 2.0, reaching 20.2%.

These results underscore the practical benefits of this proposed strategy, showing that even smaller, more efficient models can achieve substantial gains when equipped with intelligent context management.

Strategic Implications and Future Prospects

The impact of CompactionRL extends beyond academic research. Its strategic implications are substantial, as evidenced by its deployment in the RL pipeline for training the open GLM-5.2 model (750B-A40B). This real-world application in a cutting-edge open-source model demonstrates the scalability and efficacy of the CompactionRL framework.

This framework is actively pushing the boundaries of what open-source LLMs can achieve in complex, long-context scenarios. The ability to handle much longer interaction histories promises to unlock new capabilities for agentic AI systems, making them:

• More Capable: Able to tackle previously unmanageable long-horizon tasks.
• More Reliable: Less prone to errors caused by context loss.
• More Accessible: By improving the performance of open-source models, advanced agentic capabilities become more widely available to researchers and developers.

As LLM agents become more sophisticated, their need for robust, long-term memory solutions will only grow. CompactionRL offers a powerful, empirically validated approach to meet this demand, paving the way for a new generation of highly capable and autonomous AI systems.

Conclusion

The finite context window has long been a limiting factor for agentic LLMs attempting complex, long-horizon tasks. CompactionRL, through its novel integration of context summarization within a reinforcement learning framework, offers a compelling solution. By jointly optimizing task execution and summary generation using techniques like token-level loss normalization and cross-trajectory GAE, it enables LLM agents to effectively learn from and utilize vastly extended interaction histories.

The significant performance gains demonstrated on agentic coding benchmarks like SWE-bench Verified and Terminal-Bench 2.0, coupled with its deployment in advanced open-source models, solidify CompactionRL's position as a pivotal development in the field. This breakthrough promises to deliver more capable, reliable, and accessible AI agents, pushing the frontier of what LLMs can achieve in real-world applications.