The model never forgot my rules. My buffer deleted them.
This is a debugging retrospective. The bug was boring. The technique that found it is reusable.
The symptom was vague
Week three, answers got loose. No error, no 500, no parse failure. Just replies that ignored my JSON-only instruction.
It only happened in long sessions. Short chats were always fine. Ever debug something that only breaks after forty messages?
I did the wrong thing first. I rewrote the prompt. Twice.
The prompt was not the problem.
Step 1 — Diff the request, not the response
I was logging responses. That was the mistake.
Start logging the full request payload instead. Hash it, store it, compare the first turn to turn forty.
The diff took ten seconds to read. The system message was gone.
Not truncated. Not reordered. Absent.
Step 2 — Replay with a fake model
You do not need a real model to reproduce a payload bug. Replace it with a stub that echoes the messages it received.
This runs offline, costs nothing, and is deterministic.
# repro.py
from collections import deque
SYSTEM = "You are a JSON-only API. Reply with a single JSON object."
class Session:
def __init__(self, maxlen=4):
self.history = deque([{"role": "system", "content": SYSTEM}], maxlen=maxlen)
def add(self, role, content):
self.history.append({"role": role, "content": content})
def payload(self, user_msg):
return list(self.history) + [{"role": "user", "content": user_msg}]
s = Session(maxlen=4)
for i in range(6):
s.add("user", f"turn {i}")
s.add("assistant", "ack")
print(s.payload("next task")[0]["role"])
# expected: system
# actual: user
Run it. The first element is user.
Step 3 — Find the flip point
Do not guess which message number breaks. Print the boundary.
for n in range(1, 12):
s = Session(maxlen=4)
for i in range(n):
s.add("user", f"t{i}")
print(n, s.payload("x")[0]["role"])
Output is system up to a point, then user forever. That is your off-by-one, visible in one screen.
What made the bug nasty was the surface. The payload was still valid. The API accepted it. The model answered. Nothing threw.
Root cause: deque(maxlen=N) evicts from the left
I stored the system prompt as element zero of a rotating buffer. The buffer trims the oldest item on overflow. Element zero is the oldest item.
So the thing I treated as a constant was stored as the first thing to be thrown away. A one-line design decision, four weeks of confusing output.
The lesson is not "deque is bad". It is this: pinned context must live outside the sliding window.
The fix
Separate the invariant from the history.
class Session:
def __init__(self, maxlen=4):
self.system = {"role": "system", "content": SYSTEM}
self.history = deque(maxlen=maxlen)
def payload(self, user_msg):
return [self.system, *self.history, {"role": "user", "content": user_msg}]
Now the regression test is a real test, not a script you eyeball once.
def test_system_prompt_survives_long_session():
s = Session(maxlen=4)
for i in range(10):
s.add("user", f"turn {i}")
s.add("assistant", "ack")
assert s.payload("next")[0]["role"] == "system"
Keep it. It will catch the next refactor that "simplifies" your payload builder.
The guard that matters more than the test
Tests cover paths you remembered. Boundary assertions cover paths you did not.
def build_payload(system, history, user_msg):
payload = [system, *history, {"role": "user", "content": user_msg}]
assert payload[0]["role"] == "system", "system prompt evicted"
return payload
Two lines. One metric. Alert when it fires in production, not just in CI.
The second bug the same test caught
While fixing this, I found a debug wrapper. On failure it appended the last trace as a user message so the model could "see" it.
In a bounded buffer, that trace consumed a slot and evicted a real turn. My error reporting was destroying my context.
Same class of bug. Same test now covers both. Debug payloads belong in the request body, never in the history.
A trim rule that never touches the head
If you must shorten, cut from the middle. Never from the front.
def trim(msgs, keep_last=6):
if len(msgs) <= keep_last + 1:
return msgs
marker = {"role": "user", "content": "[earlier turns omitted]"}
return [msgs[0], marker, *msgs[-keep_last:]]
Also audit for these while you are in there: lru_cache on request builders, list[-N:] slices, and any pop() on shared state.
Fake model or live model?
| You are checking | Use | Why |
|---|---|---|
| Payload shape, ordering, eviction | Fake model | Deterministic, free, fast |
| Prompt actually obeyed | Live model | Only a real model can answer |
| Regression after refactor | Both | Shape check in CI, behavior check nightly |
| Token budget drift | Live or tokenizer | Stub cannot estimate cost |
The first two columns are the split that saved me. Shape bugs are 90% of this class. You can find them without any API call.
Where a free model and a free server fit
The nightly behavior check is the annoying one. It needs a key and a box that is awake at 3am.
This is where I reached for MonkeyCode. It offers free model access and a free server option. At the time of writing I am told that includes 10 million free tokens and a free server, but free tiers move, so read the current terms page before you plan around the number.
My loop is simple. Shape tests run against the stub on every commit. Once a night, I replay thirty saved fixtures against a real model and assert the JSON contract still holds. That is a low-traffic, high-value job. It fits a free tier well. If you already have a key and a box, you probably do not need it.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
Start with the failing test. Point it at whatever model endpoint you like.
Limitations, honestly
This workflow will not save you everywhere.
- If your product needs a pinned model version, verify the free tier allows it. Do not assume.
- If your prompts contain regulated data, check data handling before sending fixtures anywhere.
- If you need an uptime SLA for the nightly job, a free server is not the right home.
- If volume is high, free quotas will not cover it. Budget for paid capacity.
And a warning about the technique itself. A green test suite after this bug did not mean the system was correct. It meant one invariant was now enforced. That is all a test ever promises.
Your five-step checklist
- Log request payloads, not just responses.
- Hash the first turn and the fortieth turn. Diff them.
- Reproduce with a stub model before touching a real one.
- Find the flip point with a loop, not a hunch.
- Assert the invariant at the boundary, in production.
Ask yourself one question before the next outage. Is your most important context pinned, or is it just the oldest element in a queue?
That question cost me four weeks. I hope it costs you four minutes.
Top comments (0)