DEV Community

Cover image for Context Compression: Making AI Agents Forget Without Losing the Plot
Rijul Rajesh
Rijul Rajesh

Posted on

Context Compression: Making AI Agents Forget Without Losing the Plot

#ai #machinelearning

Hello, I'm Rijul. I'm building git-lrc, a micro AI code reviewer that runs on every commit. It's free and source-available on GitHub. Star git-lrc to help more developers discover the project. Do give it a try and share your feedback

Suppose you are working with an AI agent to fix an API.

You give it a simple instruction:

Fix the API 500 error.

The agent might go through a workflow like this:

Agent:
→ Reads the logs
→ Searches the codebase
→ Checks the database
→ Checks recent commits
→ Runs tests
→ Tries a fix
Enter fullscreen mode Exit fullscreen mode

After 30 tool calls, the agent's context can become huge.

But most of that information may no longer be useful.

The agent does not need to keep every detail of the investigation forever.

It does not need:

  • Every line from every log
  • Every failed search
  • Repeated information
  • Old tool outputs
  • Intermediate steps that are no longer relevant

What it really needs is the current state of the investigation.

The Context Problem

Imagine the agent has accumulated 40,000 tokens of context during the investigation.

That context takes up valuable space.

As the agent continues working, the context window gradually fills up.

Eventually, the agent may have less room for new information, which can lead to:

  • Higher costs
  • Slower processing
  • Less room for future tool calls
  • Important information being pushed out of context

So instead of carrying the entire history forward, we can compress it.

The original 40,000 tokens might become:

Goal:
Fix the API 500 error.

Found:
The error started after deployment v1.4.
The database is healthy.
The payment service is missing PAYMENT_API_KEY.

Tried:
Restarting the service. No effect.

Next:
Fix the environment configuration and retest.
Enter fullscreen mode Exit fullscreen mode

This is context compression.

What Is Context Compression?

Context compression is not simply about making the context shorter.

The goal is to:

Remove the information that is no longer useful while preserving what the agent needs to continue working.

The agent does not need to remember every step it took.

It needs to remember the important conclusions from those steps.

Three Basic Context Compression Techniques

1. Pruning

The simplest technique is pruning.

You remove information that is no longer useful.

For example:

Agent:
→ Searched for config.yaml
→ Found nothing

Agent:
→ Searched for settings.yaml
→ Found nothing

Agent:
→ Searched environment variables
→ Found PAYMENT_API_KEY is missing
Enter fullscreen mode Exit fullscreen mode

Once the agent has found the actual cause, the failed searches may no longer be useful.

They can be removed from the active context.

The important information is:

PAYMENT_API_KEY is missing.
Enter fullscreen mode Exit fullscreen mode

Pruning is essentially:

Remove what the agent no longer needs.

2. Distillation

Instead of keeping the entire conversation, we can convert it into a structured summary.

For example:

Goal:
Fix the API 500 error.

Facts:
- The database is healthy.
- The error started after deployment v1.4.
- PAYMENT_API_KEY is missing.

Decisions:
- Do not modify the database.
- Fix the environment configuration.

Completed:
- Checked the logs.
- Verified database connectivity.
- Inspected environment variables.

Next Action:
- Add PAYMENT_API_KEY.
- Restart the service.
- Retest the API.
Enter fullscreen mode Exit fullscreen mode

The original investigation may have taken thousands of tokens.

But the distilled state contains the information the agent needs to continue.

A useful structure might be:

Goal
Facts
Decisions
Completed Work
Next Action
Enter fullscreen mode Exit fullscreen mode

Distillation is essentially:

Turn a long history into a structured state.

3. Generalisation

Sometimes an investigation contains knowledge that can be reused in future situations.

For example:

Specific experience:

Missing API key caused the payment API to fail.
              ↓
Reusable knowledge:

Check environment variables when an API integration fails.
Enter fullscreen mode Exit fullscreen mode

The agent is no longer just remembering what happened in one specific incident.

It is extracting a general rule from that experience.

This can be useful for:

  • Future investigations
  • Runbooks
  • Agent skills
  • Long-term memory

Generalisation is essentially:

Turn a specific experience into reusable knowledge.

The Big Idea

An AI agent does not need to carry its entire history forever.

It needs to preserve the parts of that history that are still useful.

A long investigation might look like this:

40,000 tokens of raw history
              ↓
       Context compression
              ↓
      Goal + Facts + Decisions
              ↓
          Next Action
Enter fullscreen mode Exit fullscreen mode

The agent can then continue working with a much smaller context while retaining the information that actually matters.

Context compression is not about forgetting everything.

It is about forgetting the right things.

AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.

git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.

Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.

Give it a ⭐ star on Github

Top comments (0)