Context Budgeting: Keep Your AI Assistant Sharp Past the First 20 Minutes

At minute 25, my AI assistant started asking questions I had already answered. At minute 30, it proposed a solution that directly contradicted a constraint I'd stated ten minutes earlier. At minute 40, I gave up, opened a new chat, and retyped every critical detail.

This isn't the model getting dumber. It's the context window filling up — and the oldest, most critical details getting silently pushed out.

The 25-minute memory cliff

I've been testing free AI coding assistants lately, and I keep seeing the same pattern: long-conversation quality doesn't degrade gradually. It falls off a cliff. Sharp for 20 minutes, then suddenly forgetful.

The reason is simple: your token budget runs out. Every sentence you type, every code snippet you paste, every model reply — it all consumes budget. When the budget is exhausted, the oldest messages get dropped. Usually the ones you spent the most time explaining.

MonkeyCode is an open-source AI coding assistant with free model access and a free server option, so you can skip the self-hosting setup entirely. Disclosure: This article was prepared as part of MonkeyCode's product outreach. But no matter which backend you connect to, context management is still your job. A free server solves the infrastructure problem, not the memory problem.

Context is a budget, not a feature

Context windows aren't a feature. They're a budget. That mental shift changed how I work.

Most assistants advertise a fixed number: 8K, 16K, 32K. But what you actually get depends on how long and how dense your conversation is. A session with full file contents burns tokens ten times faster than plain text chat.

That's why "start a new chat" is often the right answer — but if you've already invested 40 minutes building context, throwing it away hurts. What you need is a budget manager: it tracks token usage, and when you exceed the limit, it compresses old messages automatically while keeping the important stuff.

A 60-line context budget manager

The class below implements the full budget management loop. Core idea: maintain a message list, check total tokens after every addition, and when the budget overflows, collapse the oldest non-system messages into a single summary.

#!/usr/bin/env python3
"""context_budget.py — keep long AI conversations inside a token budget."""

from dataclasses import dataclass, field
from typing import Callable, List, Optional


def estimate_tokens(text: str) -> int:
    """Cheap token estimate: ~4 chars per token for code."""
    return max(1, len(text) // 4)


@dataclass
class Message:
    role: str  # "system" | "user" | "assistant" | "summary"
    content: str
    tokens: int = field(init=False)

    def __post_init__(self) -> None:
        self.tokens = estimate_tokens(self.content)


@dataclass
class ContextBudget:
    max_tokens: int
    keep_recent: int = 6
    summarize: Optional[Callable[[List[Message]], str]] = None
    messages: List[Message] = field(default_factory=list)

    def add(self, role: str, content: str) -> None:
        self.messages.append(Message(role, content))
        self._trim()

    @property
    def total_tokens(self) -> int:
        return sum(m.tokens for m in self.messages)

    def _trim(self) -> None:
        while self.total_tokens > self.max_tokens and len(self.messages) > self.keep_recent + 1:
            before = len(self.messages)
            self._compress_oldest()
            if len(self.messages) >= before:
                break

    def _compress_oldest(self) -> None:
        # Never touch the system prompt at index 0.
        system = self.messages[0] if self.messages and self.messages[0].role == "system" else None
        compressible = self.messages[1:-self.keep_recent] if system else self.messages[:-self.keep_recent]

        if not compressible:
            return

        if self.summarize:
            summary_text = self.summarize(compressible)
        else:
            summary_text = f"[{len(compressible)} earlier messages summarized]"

        summary = Message("summary", summary_text)
        prefix = [system] if system else []
        recent = self.messages[-self.keep_recent:]
        self.messages = prefix + [summary] + recent

Usage:

budget = ContextBudget(max_tokens=4000, keep_recent=6)

budget.add("system", "You are a Python expert. Be concise.")
budget.add("user", "Here's my code:\n" + open("buggy.py").read())
budget.add("assistant", "I see the issue. Line 14 has a race condition...")

# Continue the conversation...
# When the budget overflows, old messages collapse into a summary automatically.

for m in budget.messages:
    print(f"{m.role} ({m.tokens} tok): {m.content[:60]}...")

Two parameters matter:

• max_tokens: your context ceiling. Set it to 60-70% of the model's window, leaving room for the response.
• keep_recent: how many recent messages stay untouched. I use 6 — enough to maintain continuity for the current task.

One caveat: estimate_tokens is a rough heuristic (len(text) // 4). For production, swap in tiktoken or your provider's tokenizer. The structure stays the same.

Three compression strategies

The default strategy replaces old messages with a one-line placeholder. Fast and free, but it loses everything. Three options, from lazy to smart:

1. Placeholder (default). Zero cost, zero information. Fine for quick prototypes, not for serious work.

2. LLM summarization. One LLM call collapses old messages into a real summary. Keeps more information, but each compression costs 300-500 tokens. If the summary itself gets too big, it gets compressed again — recursively.

def llm_summarize(messages: List[Message]) -> str:
    text = "\n".join(f"{m.role}: {m.content}" for m in messages)
    prompt = (
        "Summarize this conversation. Keep all technical decisions, "
        "code snippets, and constraints:\n\n" + text
    )
    return call_llm(prompt)  # your provider call here

3. Key-information extraction. Use regex or heuristics to pull out code blocks, error messages, file paths, and decisions — and discard the chit-chat. No LLM needed, but you have to tailor it to your domain.

import re

def extract_key_info(messages: List[Message]) -> str:
    key_parts = []
    for m in messages:
        code_blocks = re.findall(r"```

.*?

```", m.content, re.DOTALL)
        error_lines = re.findall(r"(?:Error|Exception|Traceback).*", m.content)
        if code_blocks:
            key_parts.append("CODE: " + code_blocks[-1][:500])
        if error_lines:
            key_parts.append("ERROR: " + error_lines[-1][:200])
    return "\n".join(key_parts) if key_parts else "[no key info extracted]"

My advice: start with the default, then upgrade once you see what your conversations actually contain. For most sessions, LLM summarization gives the best cost-to-value ratio.

When to compress vs. when to restart

Compression isn't a cure-all. Some conversations should just die.

Compress when: the session is past 20 minutes, the context is code-heavy, and your constraints are scattered across multiple messages. Compression preserves the decision trail so you can keep going deeper.

Restart when: the task direction changed, you discovered an earlier analysis was wrong, or the conversation passed the one-hour mark. In those cases, compression just wraps a wrong direction in prettier packaging. Restart — but first write a short context document: three to five key decisions, pasted into the new chat's first message.

The test is simple: if explaining the compressed context costs more than what compression saved you, restart.

The takeaway

Your AI assistant doesn't get dumber at minute 25. It runs out of budget. A context budget manager lets you fit more useful work into the window — turning a cliff into a gentle slope.

Next time your assistant starts repeating itself, check the budget before you blame the model. If this approach fits your workflow, the MonkeyCode repo has current details on the free server and quota — worth a read before you commit to a setup.