By hour thirty of my 48-hour experiment, the model was answering in fragments and dropping the JSON fields I had asked for at the very start. I was running the job on a free server with a free model tier — in this case, MonkeyCode's free model access and free server option — which meant I could not simply buy a bigger context window. The conversation history had grown so long that my own system prompt was being evicted from the context window, and the model remembered the chat but forgot the rules.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
Sound familiar? I spent the next eighteen hours building a context budget that pinned the instructions back in place, and this is the field note I wish I had read before I started. It is not a clever prompt trick, and it does not require a bigger model, just a stricter accounting of what fits in the window.
What I tried first (and why it made things worse)
Like any reasonable developer, I assumed the model was degrading and started re-prompting. I rewrote the system prompt, added "remember to output JSON" to every user message, and even switched to a shorter output format to save tokens. Every one of those moves added tokens to the same overflowing context, which pushed the original instructions further out of reach.
The failure signatures were easy to spot once I knew what to look for:
- Output format drift: fields that were present at hour two vanished by hour thirty.
- The model started summarizing instead of answering, as if it had forgotten the task entirely.
- My own instructions started appearing verbatim in responses, a classic sign the model was echoing the last message it could still see.
What actually broke
The provider counts the system prompt as part of the same context window as the conversation, so when history grows, something at the top has to go. Most chat APIs truncate from the beginning or let the model quietly drop early tokens, and the system prompt is the first thing that dies. The failure is silent because the API still returns 200 OK with perfectly readable prose.
The fix is not a longer prompt, and it is not a shorter system prompt either. The fix is a budget that treats the system prompt as non-evictable and trims the history instead. Once I stopped blaming the model and started measuring the window, the path was obvious.
The context budget that fixed it
class ContextBudget:
"""Keep the system prompt pinned while trimming old history."""
def __init__(self, system_prompt, hard_limit, reserve=0.15):
self.system_prompt = system_prompt
self.hard_limit = hard_limit
self.reserve = reserve
self.history = []
def _estimate_tokens(self, text):
# ~4 characters per token; swap in the model's tokenizer when available
return max(1, len(text) // 4)
def _used_tokens(self):
system = self._estimate_tokens(self.system_prompt)
history = sum(self._estimate_tokens(m["content"]) for m in self.history)
return system + history
def add(self, role, content):
self.history.append({"role": role, "content": content})
self._trim()
def _trim(self):
budget = int(self.hard_limit * (1 - self.reserve))
while self._used_tokens() > budget and len(self.history) > 1:
self.history.pop(0)
def messages(self):
return [{"role": "system", "content": self.system_prompt}] + self.history
Three design decisions matter here:
- The reserve (15% by default) leaves room for the model's response, so the request never fills the window before the answer starts.
- Eviction always removes the oldest message first, never the system prompt, because the system prompt is re-injected on every call.
- The heuristic token estimate is a fallback; when the model's tokenizer is available, use it, because code-heavy JSON will wreck a character-based estimate.
What broke on day two
The heuristic was the first casualty. My job produced long JSON payloads, and the four-characters-per-token guess undercounted them badly, so the window overflowed and the model started hallucinating fields again. I swapped in the real tokenizer when it was available and kept the heuristic only as a cold-start fallback.
Then I hit the second problem: eviction alone is amnesia. The model handled the remaining history fine, but it had no idea what the evicted messages contained, so it repeated questions I had already answered. I added a rolling summary that compresses the dropped turns into a few lines, and the repetition stopped.
def add_with_summary(self, role, content, summarize=False):
if summarize and len(self.history) >= 4:
dropped = self.history[:2]
self.history = self.history[2:]
self.summary = "Earlier context: " + "; ".join(
f"{m['role']}: {m['content'][:120]}" for m in dropped
)
self.add(role, content)
def messages(self):
system = self.system_prompt
if getattr(self, "summary", None):
system += "\n\n" + self.summary
return [{"role": "system", "content": system}] + self.history
What I'd repeat, and what I wouldn't
- Pin the system prompt and never let history evict it.
- Reserve 15–20% of the window for the response; a full request is a failed request.
- Summarize before you evict, so the model keeps the gist of what it lost.
- Log token counts per call; the trend tells you a job is about to break before the output does.
Who should not use this approach? If your task needs full recall of every message, eviction is the wrong tool, and you should store messages in a database and retrieve relevant context instead. If your model has a generous context window and your conversations are short, this class is over-engineering. If you cannot observe how the provider truncates, measure first with a tiny probe before you trust any budget.
Limitations
The heuristic estimate is an estimate, so always prefer the real tokenizer when your model exposes one. This fixes client-side context management, not provider-side truncation or server memory loss, which are separate failure classes. Free tiers can also change context windows without notice, so re-measure whenever the output behavior shifts.
Forty-eight hours later, the job that had been drifting into prose was back to structured output, and the only code I changed was the budget, not the prompt. The lesson I'd repeat: your system prompt is a contract, so treat it like one and give it a budget it cannot outgrow. If you have ever watched a model quietly forget your rules, I would love to hear how you caught it.
Top comments (0)