DEV Community

Cover image for Stop Appending to Your Prompt: Implementing a "Garbage Collector" for AI Agents
Imran Siddique
Imran Siddique

Posted on

Stop Appending to Your Prompt: Implementing a "Garbage Collector" for AI Agents

#agents #showdev #ai #llm

We have a hoarding problem in AI Engineering.

When an agent makes a mistake—say, it outputs Markdown instead of JSON—we patch it. We add a line to the system prompt: "ALWAYS output valid JSON."
When it hallucinates a deprecated API, we add another line: "Do not use v1 endpoints."

This works... for a while. But over months, your system prompt grows into a 5,000-token Frankenstein monster of conflicting instructions, edge-case fixes, and emotional blackmail ("...or I will be very sad").

This is Context Rot. And it kills production agents.

I recently built a module called Semantic Purge for my Self-Correcting Agent Kernel. It acts as a Garbage Collector for agent instructions.

Here is the theory and the code.

The Theory: The Taxonomy of Lessons

The core realization is that not all "fixes" are equal.

If I have to tell an agent "Output JSON, not XML," that is a patch for a Model Defect. It's a capability gap in GPT-4.

If I have to tell an agent "The fiscal year starts in July," that is a Business Truth. It's a fact about my world.

When GPT-5 comes out, the "Output JSON" instruction becomes trash. The model will likely handle it natively. But the "Fiscal Year" instruction remains true forever.

I codified this into a Taxonomy of Lessons:

  1. Type A (Syntax/Capability): High Decay. Fixes for model stupidity.

    • Examples: "Use specific JSON format", "Don't hallucinate arguments", "Check constraints".
    • Lifecycle: Purge on Model Upgrade.

  2. Type B (Business/Context): Zero Decay. Facts about the world.

    • Examples: "Project X is deprecated", "Use the staging environment", "Medical advice policy".
    • Lifecycle: Retain Permanently (or move to RAG).

The Implementation: Scale by Subtraction

I implemented a PatchClassifier that analyzes every new instruction we give the agent and tags it.

Here is the logic from semantic_purge.py:

class PatchDecayType(Enum):
    SYNTAX_CAPABILITY = "syntax_capability" # Type A (Purgeable)
    BUSINESS_CONTEXT = "business_context"   # Type B (Permanent)

class PatchClassifier:
    def _determine_decay_type(self, patch: CorrectionPatch) -> PatchDecayType:
        content = str(patch.patch_content).lower()

        # Syntax indicators (Type A)
        syntax_patterns = [
            "output json", "format", "syntax", "validation error",
            "type mismatch", "parameter type", "schema injection"
        ]

        # Business indicators (Type B)
        business_patterns = [
            "fiscal year", "project", "policy", "deprecated",
            "company", "organization", "workflow", "does not exist"
        ]

        # Heuristic classification
        if any(p in content for p in business_patterns):
            return PatchDecayType.BUSINESS_CONTEXT

        if any(p in content for p in syntax_patterns):
            return PatchDecayType.SYNTAX_CAPABILITY

        # Default to safety (Business)
        return PatchDecayType.BUSINESS_CONTEXT
Enter fullscreen mode Exit fullscreen mode

It’s simple, but it changes how we handle the Agent Lifecycle.

The "Purge Event"

In traditional software, we have "Refactoring Sprints." In Agent Engineering, we need Purge Events.

When we upgrade the underlying model (e.g., swapping gpt-4-turbo for gpt-4o or claude-3-5), we don't just migrate the prompt. We run the Purge.

The SemanticPurge class iterates through all active patches and drops the ones marked as "Syntax/Capability."

    def purge_on_upgrade(self, old_version: str, new_version: str):
        print(f"🗑️  PURGE EVENT: {old_version} -> {new_version}")

        purged = []
        for patch_id, classified in self.classified_patches.items():
            if classified.decay_type == PatchDecayType.SYNTAX_CAPABILITY:
                # This patch fixed a defect in the OLD model. 
                # The NEW model likely doesn't need it.
                purged.append(patch_id)
                self.delete_patch(patch_id)

        print(f"✓ Reclaimed {len(purged) * 100} tokens")
Enter fullscreen mode Exit fullscreen mode

Why This Matters

We often talk about "Learning Agents." But learning isn't just accumulating knowledge; it's also forgetting the irrelevant.

If you don't implement a garbage collector for your context:

  1. Cost: You pay for "Output JSON" instructions on every single API call, forever.
  2. Latency: More input tokens = slower time-to-first-token.
  3. Confusion: "Instruction Conflict" rises as the prompt grows.

By splitting my agent's memory into Model Patches (Temporary) and World Knowledge (Permanent), I've reduced my system prompt size by 40-60% during model upgrades.

Stop letting your agents be hoarders. Build a garbage collector.

Resources:

Top comments (0)