The Model Was Fine. My Token Assumptions Weren't.

The model was never the problem, and that is exactly why the bug took three days to find. My ticket-classification service started returning the fallback label for long, non-English messages shortly after I moved the inference path to a cheaper endpoint, and every instinct pointed at the new model. The real culprit was a token-counting mismatch that silently truncated the prompt before the model ever saw the classification instruction.

The Symptom

The failure was remarkably consistent, which made it even more misleading. Messages under roughly two thousand characters classified correctly, while longer ones, especially in German and Japanese, fell through to a generic "other" bucket with a perfectly valid JSON response. The parser was not the issue, the prompt had not changed in weeks, and the retry logic never fired because the endpoint returned a normal 200 status.

My first assumption was that the cheaper model was simply weaker at long-context reasoning, so I ran a controlled comparison using the same fifty tickets against the previous endpoint. The old path classified all fifty correctly, the new one failed on nineteen, and that result seemed to confirm the model-quality theory. What bothered me was the distribution: the failures clustered exactly where the input length crossed a threshold, and no ticket under that threshold ever failed.

The Reproduction

To isolate the variable, I needed a clean environment where I could swap endpoints without touching the production deployment, and MonkeyCode's free server option turned out to be a practical debugging tool. The project is open source, and its free model access let me replay the failing tickets without spending my own quota, so I spun up a disposable instance and pointed the same harness at the same prompt. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The reproduction took about twenty minutes, and the result was identical on every retry: long inputs failed, short inputs passed.

The Root Cause

The breakthrough came when I logged the token count of the incoming payload instead of the character count. My client code had a hard character limit that was supposed to keep every prompt inside the context window, but the tokenizer used by the new endpoint split German and Japanese text into roughly twice as many tokens per character as English. The client-side truncation then cut the message at the character boundary, which happened to fall right before the classification instruction, so the model produced a confident fallback with no idea that the instruction had ever existed.

The Fix

The fix had three parts, and none of them involved changing the model. I replaced the character-based guard with a token-based guard that used the same tokenizer as the inference endpoint, so the length check and the actual consumption could never disagree again. I added a sentinel instruction that asked the model to include a marker in every response, and I rejected any output that lacked the marker, which turned silent truncation into a loud validation error. Finally, I added a regression test that fed the harness a set of long, non-English fixtures and asserted that the marker always appeared.

Here is the guard that fixed the production bug, and it is short enough to review in one sitting:

# guard.py
import json
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("your-endpoint-tokenizer")

INSTRUCTION = (
    'Classify the message as billing, bug, feature, or other. '
    'Reply with JSON only: {"label": "...", "marker": "DONE"}.'
)

def safe_prompt(message: str, max_tokens: int = 4000) -> str:
    fixed_tokens = len(tokenizer.encode(INSTRUCTION))
    budget = max(0, max_tokens - fixed_tokens - 32)
    message_tokens = tokenizer.encode(message)
    if len(message_tokens) > budget:
        start = max(0, len(message_tokens) - budget)
        message = tokenizer.decode(message_tokens[start:])
    return f"{INSTRUCTION}\n\n{message}"

def is_complete(response: str) -> bool:
    try:
        data = json.loads(response)
        return data.get("marker") == "DONE" and "label" in data
    except json.JSONDecodeError:
        return False

The key detail is that truncation happens on the message side, never on the instruction side, and the budget reserves a safety margin for the model's own output tokens. If you are replaying a production incident, the reproduction script is even simpler, and it only needs the endpoint URL and a long multilingual string:

# reproduce_drift.py
import requests

ENDPOINT = "https://your-endpoint.example/v1/chat/completions"

def classify(message: str) -> str:
    payload = {
        "model": "your-model",
        "messages": [{"role": "user", "content": message}],
    }
    response = requests.post(ENDPOINT, json=payload, timeout=30)
    return response.json()["choices"][0]["message"]["content"]

long_german = "Wir haben ein Problem mit unserer Rechnung und brauchen Hilfe. " * 200
print(classify(long_german))

If that call returns a valid JSON object without the DONE marker, you have reproduced the exact class of bug, and the fix is the guard above.

The sentinel marker deserves a moment of explanation because it looks redundant at first glance. The model was already returning valid JSON, so a JSON parser alone would never catch the truncation, and the marker lives inside the JSON object as a cheap integrity check. If the prompt was cut before the instruction, the model could still emit a plausible fallback, but it could not know to include the marker, which made the marker a reliable witness for what the model actually saw.

The Pattern

The decision table I keep next to my monitor summarizes the pattern for future incidents:

Symptom	Likely cause	First check
Valid JSON, wrong fallback label	Truncated instruction	Token count vs. context limit
Malformed JSON on long inputs	Output format drift	Schema validation
Failures only in one language	Tokenizer asymmetry	Multilingual fixtures
Timeouts that correlate with length	Context overflow	Log token counts

Limitations

This approach is not a cure-all, and some teams should not copy it blindly. If your pipeline cannot tolerate a validation layer that rejects a small percentage of responses, you need a retry strategy or a human fallback before you add a sentinel check. If you are bound by a strict data-residency policy, routing inference through a free server outside your region is a compliance problem rather than a debugging convenience, and the current free tier's ten-million-token allowance is a ceiling that a high-volume service will hit quickly.

The Takeaway

The lesson that stuck with me is that swapping a model endpoint changes more than the model; it changes the tokenizer, the context math, and the failure modes your tests never covered. The cheapest way to learn that lesson is to reproduce it in a disposable environment before it happens in production, which is exactly what the free server and free model access let me do here. If you want to run the same reproduction against your own long-tail messages, the two scripts above are a complete starting point, and the free tier I used is a reasonable place to run them.