I benchmarked a vision-language model and scored it at 0.31.
The real number was 0.70. Same model, same weights, same hardware, same 100 questions. The only thing that changed was how I read its output.
I had already written up the 0.31 as a capability finding and concluded the model was unsuitable. That conclusion was wrong, and the failure was entirely in my harness. Here is the mistake, because I doubt I am the only one making it.
The setup
I was evaluating a batch of open-weight and frontier models on a multiple-choice benchmark: multi-view driving scenes, four options per question, one correct answer. Standard stuff. The prompt asked for reasoning followed by a final line, Answer: X.
My scoring code did the obvious thing:
m = re.search(r"Answer:\s*([A-D])", output)
pred = m.group(1) if m else None # None scores as wrong
That last comment is the bug.
What actually happened
The model I was testing is a "thinking" model. It emits a long internal reasoning trace before it commits to an answer. I had a generation budget of 1024 tokens.
On easy questions it reasoned briefly, emitted Answer: B, and scored fine. On hard questions it reasoned at length, hit the token cap mid-thought, and never emitted the answer line at all.
So the harness scored every one of those as wrong.
64 of 100 questions returned no parseable answer. Zero of those were image-loading errors or crashes. They were all truncation. And the truncation was not random:
Uncertainty 0/8 answered
Counterfactual 0/3 answered
Safety-critical Planning 1/11 answered
Safety-critical Prediction 3/12 answered
Look at that distribution. The questions the model failed to answer were precisely the questions that required the most reasoning. My harness was systematically discarding the model's performance on exactly the hard subset I was trying to measure, and reporting the result as a capability ceiling.
Of the 36 it did answer, it got 86% right. The model was fine. My measurement was garbage.
The fix
Stop parsing free text. Constrain the decoding to a schema.
With Ollama:
resp = ollama.chat(
model="my-vlm:9b",
messages=[{"role": "user", "content": prompt, "images": imgs}],
format={ # enforced at decode time
"type": "object",
"properties": {"answer": {"type": "string", "enum": ["A","B","C","D"]}},
"required": ["answer"],
},
think=False,
)
The equivalent exists nearly everywhere now: response_format with a JSON schema on OpenAI-compatible endpoints, structured outputs in vLLM via guided_json, Outlines, or plain grammar-constrained sampling in llama.cpp.
The point is that the model can no longer produce an unparseable output. The constraint is applied during sampling, not checked afterwards.
Re-ran the same 100 questions: 0.70, with a 100% answer rate. The categories that had been at zero came back at 0.82 and 0.75.
The second-order trap
I hit a related version of this on a larger model in the same family, and the fix was less obvious.
That model also truncated, on 36 of 100 questions. I had a fallback that recovered an answer letter from the tail of a truncated trace, so those rows produced something. The score came out 0.67.
But the recovered answers were junk in a specific, dangerous way. Predicted answers skewed heavily toward option A (48 predictions of A against 23 in the ground truth). When the model gets cut off mid-reasoning, the letter you scrape from the tail is not a decision, it is whatever token happened to be nearby. That is position bias, and it looks exactly like a real answer to your scorer.
Rows where the reasoning actually completed scored 0.83. I re-ran just the truncated rows with a budget of 8192 instead of 3072, and the merged score came to 0.73.
So: tail-recovery is worse than a non-answer. A non-answer is visibly missing. A biased recovered answer silently contaminates your accuracy in a direction you did not choose.
Note also that 16 of those 36 questions still truncated at 8192 tokens. 0.73 remains a floor, not a measurement.
What I do now
Match the evaluator's generation budget to the training or intended-use budget. Mismatched budgets scored 31% of my rows incorrectly in one run. If you fine-tuned at 512 tokens, do not evaluate at 128.
Track truncation as a first-class metric. Log clipped_ratio and parseable_rate next to accuracy. If either moves, your accuracy number is not comparable to the previous run. I now fail a run outright if the parse-failure count is anything other than what I expect.
Distinguish infrastructure failure from poor performance. An unparseable output is a harness event, not a model event. They must be counted separately, always. A weak model still produces well-formed answers and simply scores badly.
Never let a non-answer score as wrong by default. Make it raise, or count it in its own bucket. Silently mapping "I could not read this" to "the model was incorrect" is how you get a confident, published, wrong conclusion.
Verify the harness against a known-good signal before trusting any number it emits. An evaluation harness that has never been checked against a baseline you can sanity-check by hand is untested code that produces numbers, not a measurement instrument.
Why this matters more than it used to
Reasoning models broke an assumption that free-text answer parsing quietly relied on for years: that a model's answer appears in its output. With extended chain-of-thought, the answer arrives last, and last is exactly what a token budget truncates.
Every leaderboard comparing a thinking model against a non-thinking one under a shared token budget is, at minimum, measuring something other than what it claims. The thinking model pays for its reasoning out of the same budget that has to carry its answer.
I published 0.31 as a capability limit and had to retract it. The model was never the problem.
I write about ML evaluation, robotics data pipelines, and the ways measurement quietly fails. linkedin.com/in/rickeshnatarajan
Top comments (0)