DEV Community: Rickesh T N

Three Ways Your Training Data Lies to You (And None of Them Throw an Error)

Rickesh T N — Fri, 07 Aug 2026 00:21:27 +0000

Every failure I am about to describe produced a clean run. No exception, no stack trace, no red build. Each one produced a plausible number that I believed for longer than I should have.

That is the category of bug I have come to fear most. A crash tells you it crashed. A silently broken dataset tells you nothing at all, and your metrics will politely agree with it.

Here are three from the last year, all from my own work, all found late.

1. The dataset that was 92% one category

I had a training set of 688 records for a multi-category vision-language task. Thirteen categories. Reasonable size for a fine-tune, already used in a completed training run whose results I had written up.

While preparing a stratified split, I joined the records back against the source annotations and actually counted the categories.

630 of 688 were a single category: scene captions. Zero examples of traffic signals. Zero of planning. Zero of uncertainty. Several categories the evaluation explicitly measured had no representation in training at all.

The previous fine-tune had shown gains on some of those very categories. I had interpreted this as the model learning the task. The real explanation was duller and more useful: the model had learned the answer format from caption supervision, and format alignment alone was enough to move a multiple-choice score. Nothing category-specific had been learned, because nothing category-specific had been shown.

The root cause was upstream and boring. The conversion script I inherited only rewrote file paths and dropped records with missing frames. It faithfully preserved a caption-only selection made further up the chain. It had no opinion about balance because nobody had asked it to have one.

What I changed: the composition of a training set is now an artifact I generate and inspect before any run, not a property I assume. A category histogram takes seconds. I had not looked, for months.

2. The 18-hour run that converged perfectly to nothing

Large model, QLoRA, multi-GPU, 8 camera views per sample. Roughly a full day of compute.

The loss curve was beautiful. It fell from 19.4 to 15.7 over the first 26 steps, then to 0.078 by step 51, then flattened near 0.02 and stayed there. Token accuracy reached 0.99. Gradient norms decayed smoothly. Nothing in the training telemetry looked wrong.

The task metric was 0.10.

The cause: I was supervising on free-text answers, median 11 words, phrased like "One should keep to the right side of the road and drive slowly." The evaluation was four-option multiple choice, scored on the index of the chosen option.

The model learned, quickly and correctly, to reproduce the style of the training answers. That is genuinely what the loss was asking for, and it maxed it out in about 50 steps. It was never asked to select an option, so it never learned to.

Training and evaluation were measuring different tasks. Both were internally consistent. Neither could detect the other's disagreement.

What I changed: before a long run starts, I now write down what the eval measures and what the loss optimizes, in one line each, and check they describe the same thing. When loss collapses to near zero within a few dozen steps, I treat that as an alarm rather than a success. Genuine learning on a hard task does not look like that.

3. The dataset that was 40% smaller than its file size claimed

Assessing a large public robotics corpus for adoption. The headline: 3.91 TB compressed, 4.65 TB extracted, hundreds of shards, dozens of buildings, thousands of hours of teleoperated manipulation.

The number that mattered was buried in the paper's method section. Only about 60% of the raw data converted into reliable 3D flow annotations. The rest failed depth estimation, camera pose optimization, or point tracking. Then a further filter kept only trajectories with actual robot-object contact and real object motion.

Nominal size: thousands of hours. Training-ready size after both filters: roughly 500 hours.

Every plan built on the first number was wrong by a factor of several. Storage estimates, download time, compute budget, and most importantly the question of whether the corpus was even large enough for what we wanted.

What I changed: I now treat yield as the primary dataset metric, not size. What fraction of raw capture survives every stage into training-ready data? For collected data this is a hard operational KPI, and it is the one that determines cost per usable hour. Bytes on disk is a storage figure. It tells you almost nothing about what you can train on.

The pattern

All three share a shape.

A pipeline stage did exactly what it was written to do. A metric moved in the direction that metric moves when things go well. And the thing I actually cared about was not being measured by anything at all.

The defenses that work for me:

Check that work happened, not just that output exists. A gate that verifies "400 rows were produced" while never checking "any row succeeded" will pass a completely dead run. I once had an evaluation return zero correct on every row for 45 minutes because a kernel was unavailable on that GPU. Zero is a valid ratio. Row counts agreed. The gate passed and the sweep moved on.

Count infrastructure failures separately, and require zero. A weak model produces well-formed output and scores badly. A broken pipeline produces nothing and scores identically. Collapsing those two into one number destroys the only signal that distinguishes them.

Emit progress with errors first. Anything running longer than a minute should print failure count before position before metric. Silence is not neutral. A job working perfectly and a job failing on every single item look exactly the same from outside when neither prints anything.

Validate splits as their own gated job. I run split validation as a separate step that must pass before training is allowed to start. It checks class balance, domain distribution across sites, and leakage between splits. It has caught things that would otherwise have cost days of GPU time and, worse, produced a believable number.

Write down what the metric would look like if the thing were broken. If you cannot answer that, you cannot tell success from failure, and you will default to reading any completed run as a successful one.

The uncomfortable part

In every one of these cases the data was the limiting factor, and in every case I spent the first stretch of debugging looking at the model.

That instinct is hard to unlearn, because the model is where the interesting work feels like it lives. But architecture, optimizer, and learning rate are all things I can inspect in a config file in about a minute. The composition of the training set is something I have to deliberately go and measure, and that extra step is exactly why it goes unchecked.

The failure mode is not carelessness. It is that broken data produces output that looks entirely normal.

I work on robotics data pipelines, model evaluation, and the operational side of making training data trustworthy. linkedin.com/in/rickeshnatarajan

Your reasoning model isn't dumb. Your parser is throwing away its best answers.

Rickesh T N — Fri, 07 Aug 2026 00:19:57 +0000

I benchmarked a vision-language model and scored it at 0.31.

The real number was 0.70. Same model, same weights, same hardware, same 100 questions. The only thing that changed was how I read its output.

I had already written up the 0.31 as a capability finding and concluded the model was unsuitable. That conclusion was wrong, and the failure was entirely in my harness. Here is the mistake, because I doubt I am the only one making it.

The setup

I was evaluating a batch of open-weight and frontier models on a multiple-choice benchmark: multi-view driving scenes, four options per question, one correct answer. Standard stuff. The prompt asked for reasoning followed by a final line, Answer: X.

My scoring code did the obvious thing:

m = re.search(r"Answer:\s*([A-D])", output)
pred = m.group(1) if m else None   # None scores as wrong

That last comment is the bug.

What actually happened

The model I was testing is a "thinking" model. It emits a long internal reasoning trace before it commits to an answer. I had a generation budget of 1024 tokens.

On easy questions it reasoned briefly, emitted Answer: B, and scored fine. On hard questions it reasoned at length, hit the token cap mid-thought, and never emitted the answer line at all.

So the harness scored every one of those as wrong.

64 of 100 questions returned no parseable answer. Zero of those were image-loading errors or crashes. They were all truncation. And the truncation was not random:

Uncertainty                 0/8   answered
Counterfactual              0/3   answered
Safety-critical Planning    1/11  answered
Safety-critical Prediction  3/12  answered

Look at that distribution. The questions the model failed to answer were precisely the questions that required the most reasoning. My harness was systematically discarding the model's performance on exactly the hard subset I was trying to measure, and reporting the result as a capability ceiling.

Of the 36 it did answer, it got 86% right. The model was fine. My measurement was garbage.

The fix

Stop parsing free text. Constrain the decoding to a schema.

With Ollama:

resp = ollama.chat(
    model="my-vlm:9b",
    messages=[{"role": "user", "content": prompt, "images": imgs}],
    format={                       # enforced at decode time
        "type": "object",
        "properties": {"answer": {"type": "string", "enum": ["A","B","C","D"]}},
        "required": ["answer"],
    },
    think=False,
)

The equivalent exists nearly everywhere now: response_format with a JSON schema on OpenAI-compatible endpoints, structured outputs in vLLM via guided_json, Outlines, or plain grammar-constrained sampling in llama.cpp.

The point is that the model can no longer produce an unparseable output. The constraint is applied during sampling, not checked afterwards.

Re-ran the same 100 questions: 0.70, with a 100% answer rate. The categories that had been at zero came back at 0.82 and 0.75.

The second-order trap

I hit a related version of this on a larger model in the same family, and the fix was less obvious.

That model also truncated, on 36 of 100 questions. I had a fallback that recovered an answer letter from the tail of a truncated trace, so those rows produced something. The score came out 0.67.

But the recovered answers were junk in a specific, dangerous way. Predicted answers skewed heavily toward option A (48 predictions of A against 23 in the ground truth). When the model gets cut off mid-reasoning, the letter you scrape from the tail is not a decision, it is whatever token happened to be nearby. That is position bias, and it looks exactly like a real answer to your scorer.

Rows where the reasoning actually completed scored 0.83. I re-ran just the truncated rows with a budget of 8192 instead of 3072, and the merged score came to 0.73.

So: tail-recovery is worse than a non-answer. A non-answer is visibly missing. A biased recovered answer silently contaminates your accuracy in a direction you did not choose.

Note also that 16 of those 36 questions still truncated at 8192 tokens. 0.73 remains a floor, not a measurement.

What I do now

Match the evaluator's generation budget to the training or intended-use budget. Mismatched budgets scored 31% of my rows incorrectly in one run. If you fine-tuned at 512 tokens, do not evaluate at 128.

Track truncation as a first-class metric. Log clipped_ratio and parseable_rate next to accuracy. If either moves, your accuracy number is not comparable to the previous run. I now fail a run outright if the parse-failure count is anything other than what I expect.

Distinguish infrastructure failure from poor performance. An unparseable output is a harness event, not a model event. They must be counted separately, always. A weak model still produces well-formed answers and simply scores badly.

Never let a non-answer score as wrong by default. Make it raise, or count it in its own bucket. Silently mapping "I could not read this" to "the model was incorrect" is how you get a confident, published, wrong conclusion.

Verify the harness against a known-good signal before trusting any number it emits. An evaluation harness that has never been checked against a baseline you can sanity-check by hand is untested code that produces numbers, not a measurement instrument.

Why this matters more than it used to

Reasoning models broke an assumption that free-text answer parsing quietly relied on for years: that a model's answer appears in its output. With extended chain-of-thought, the answer arrives last, and last is exactly what a token budget truncates.

Every leaderboard comparing a thinking model against a non-thinking one under a shared token budget is, at minimum, measuring something other than what it claims. The thinking model pays for its reasoning out of the same budget that has to carry its answer.

I published 0.31 as a capability limit and had to retract it. The model was never the problem.

I write about ML evaluation, robotics data pipelines, and the ways measurement quietly fails. linkedin.com/in/rickeshnatarajan