The Eval Passed and Production Still Broke: A Token-Counting Retrospective

An eval score measures how a model behaves on a test, not how it behaves inside your production path, and the gap between the two is usually a measurement mismatch rather than a bad model. The current debate about what AI scores actually measure is not academic; it shows up as production incidents like this one. This is a debugging retrospective about a failure that appeared after a free-tier model passed a 30-minute eval, and the trail led from a confusing symptom to two concrete root causes: a tokenizer mismatch and a cold-start timeout.

The eval ran against MonkeyCode's free model access and its free server option, and the point of the exercise was to decide whether a paid pipeline was justified. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The failure that followed is the more interesting part of the story, because it had nothing to do with model quality and everything to do with how the eval measured things.

The symptom showed up as a user-facing complaint: after about ten turns of conversation, the agent started ignoring its system prompt. It would answer with outdated instructions, drop constraints it had followed earlier, and occasionally repeat a previous answer verbatim. The eval had passed with a comfortable score, so the first instinct was to blame the model, and that instinct was wrong.

The first reproduction step was to take the exact production conversation, replay it against the model in a one-shot script, and watch the behavior appear without any user interaction. That narrowed the problem to the prompt itself, not the surrounding application logic. The second step was to dump the request payload that the client actually sent, because the bug could easily live in the serialization layer. The payload looked correct, and that is where the investigation stalled for a while.

The turning point came from counting tokens with two different tools. The eval harness had estimated the prompt at roughly 5,100 tokens using a naive word-count heuristic, while the API's own tokenizer reported 7,400 tokens for the identical text. The free model's context window was smaller than the eval assumed, so the server silently truncated the oldest messages, and the system prompt was the first thing to go. The model was not forgetting anything; it never received the instructions in the first place.

Here is the minimal reproduction of that mismatch:

# mismatch.py
import tiktoken

def naive_count(text: str) -> int:
    return len(text.split())

def real_count(text: str) -> int:
    return len(tiktoken.get_encoding("cl100k_base").encode(text))

prompt = "system: keep answers under 50 words. " * 400
print("naive:", naive_count(prompt))
print("real: ", real_count(prompt))

The naive count is roughly 30 percent lower on this text, and that percentage is enough to push a long conversation over the context boundary. The fix was to make the eval harness and the production client share one tokenizer, and to add a context budget guard that reserves space for the completion before trimming the oldest messages. Here is a simplified version of that guard:

# budget.py
def count_tokens(messages, tokenizer):
    return sum(len(tokenizer.encode(m["content"])) for m in messages)

def fit_in_context(messages, max_tokens, tokenizer, reserved=512):
    budget = max_tokens - reserved
    while count_tokens(messages, tokenizer) > budget:
        messages.pop(0)  # drop the oldest message, never the system prompt
    return messages

The guard keeps the system prompt pinned, trims from the oldest turn, and fails loudly if the budget is still exceeded after dropping everything else. That single change turned a silent truncation into a visible, testable behavior, and the regression test now replays the longest production conversation with the production parameters.

The second root cause appeared only after the first fix, because the agent started timing out instead of forgetting. The free server went idle after a few minutes, and the first request after idle took fourteen seconds to complete while the client's timeout was set to ten seconds. The retry logic then fired, sent the same request twice, and the agent executed the same side effect twice. This is the classic cold-start trap, and it is easy to confuse with a model failure.

You can measure it with a simple loop before you trust any server, free or paid:

for i in 1 2 3 4 5; do
  curl -s -o /dev/null -w "attempt $i: %{time_total}s\n" https://your-endpoint.example/health
  sleep 300
done

The first attempt after each sleep reveals the real cold-start latency, and the pattern tells you whether your timeout and retry policy can survive it. The fix here was a warm-up request after idle, a health check that the client consults before sending real work, and an idempotency key on the retry so a duplicated request cannot produce duplicated side effects.

The reusable lesson is that an eval should measure the whole path, not the model in isolation. Token counting, context windows, timeouts, cold starts, and retry semantics are all part of the contract, and each one can pass in isolation while failing together. Treat the eval score as one signal, then probe the deployment with the same prompts, the same parameters, and the same network conditions you will use in production.

The 10 million token figure and the free server availability are the operator's published claims, so verify them against your own usage rather than assuming they are permanent. Who should not use this approach: if your workload is single-turn and short, the token budget guard is unnecessary complexity, and if your latency requirement is strict, a shared free server with cold starts will disappoint you regardless of how good the model is. If you want to reproduce these failure modes cheaply, the free tier is a reasonable place to start, but measure it as you would any dependency. The techniques here are about measuring the gap between eval and production, and that gap is exactly where most "the model got worse" reports actually live.