Every failure I am about to describe produced a clean run. No exception, no stack trace, no red build. Each one produced a plausible number that I believed for longer than I should have.
That is the category of bug I have come to fear most. A crash tells you it crashed. A silently broken dataset tells you nothing at all, and your metrics will politely agree with it.
Here are three from the last year, all from my own work, all found late.
1. The dataset that was 92% one category
I had a training set of 688 records for a multi-category vision-language task. Thirteen categories. Reasonable size for a fine-tune, already used in a completed training run whose results I had written up.
While preparing a stratified split, I joined the records back against the source annotations and actually counted the categories.
630 of 688 were a single category: scene captions. Zero examples of traffic signals. Zero of planning. Zero of uncertainty. Several categories the evaluation explicitly measured had no representation in training at all.
The previous fine-tune had shown gains on some of those very categories. I had interpreted this as the model learning the task. The real explanation was duller and more useful: the model had learned the answer format from caption supervision, and format alignment alone was enough to move a multiple-choice score. Nothing category-specific had been learned, because nothing category-specific had been shown.
The root cause was upstream and boring. The conversion script I inherited only rewrote file paths and dropped records with missing frames. It faithfully preserved a caption-only selection made further up the chain. It had no opinion about balance because nobody had asked it to have one.
What I changed: the composition of a training set is now an artifact I generate and inspect before any run, not a property I assume. A category histogram takes seconds. I had not looked, for months.
2. The 18-hour run that converged perfectly to nothing
Large model, QLoRA, multi-GPU, 8 camera views per sample. Roughly a full day of compute.
The loss curve was beautiful. It fell from 19.4 to 15.7 over the first 26 steps, then to 0.078 by step 51, then flattened near 0.02 and stayed there. Token accuracy reached 0.99. Gradient norms decayed smoothly. Nothing in the training telemetry looked wrong.
The task metric was 0.10.
The cause: I was supervising on free-text answers, median 11 words, phrased like "One should keep to the right side of the road and drive slowly." The evaluation was four-option multiple choice, scored on the index of the chosen option.
The model learned, quickly and correctly, to reproduce the style of the training answers. That is genuinely what the loss was asking for, and it maxed it out in about 50 steps. It was never asked to select an option, so it never learned to.
Training and evaluation were measuring different tasks. Both were internally consistent. Neither could detect the other's disagreement.
What I changed: before a long run starts, I now write down what the eval measures and what the loss optimizes, in one line each, and check they describe the same thing. When loss collapses to near zero within a few dozen steps, I treat that as an alarm rather than a success. Genuine learning on a hard task does not look like that.
3. The dataset that was 40% smaller than its file size claimed
Assessing a large public robotics corpus for adoption. The headline: 3.91 TB compressed, 4.65 TB extracted, hundreds of shards, dozens of buildings, thousands of hours of teleoperated manipulation.
The number that mattered was buried in the paper's method section. Only about 60% of the raw data converted into reliable 3D flow annotations. The rest failed depth estimation, camera pose optimization, or point tracking. Then a further filter kept only trajectories with actual robot-object contact and real object motion.
Nominal size: thousands of hours. Training-ready size after both filters: roughly 500 hours.
Every plan built on the first number was wrong by a factor of several. Storage estimates, download time, compute budget, and most importantly the question of whether the corpus was even large enough for what we wanted.
What I changed: I now treat yield as the primary dataset metric, not size. What fraction of raw capture survives every stage into training-ready data? For collected data this is a hard operational KPI, and it is the one that determines cost per usable hour. Bytes on disk is a storage figure. It tells you almost nothing about what you can train on.
The pattern
All three share a shape.
A pipeline stage did exactly what it was written to do. A metric moved in the direction that metric moves when things go well. And the thing I actually cared about was not being measured by anything at all.
The defenses that work for me:
Check that work happened, not just that output exists. A gate that verifies "400 rows were produced" while never checking "any row succeeded" will pass a completely dead run. I once had an evaluation return zero correct on every row for 45 minutes because a kernel was unavailable on that GPU. Zero is a valid ratio. Row counts agreed. The gate passed and the sweep moved on.
Count infrastructure failures separately, and require zero. A weak model produces well-formed output and scores badly. A broken pipeline produces nothing and scores identically. Collapsing those two into one number destroys the only signal that distinguishes them.
Emit progress with errors first. Anything running longer than a minute should print failure count before position before metric. Silence is not neutral. A job working perfectly and a job failing on every single item look exactly the same from outside when neither prints anything.
Validate splits as their own gated job. I run split validation as a separate step that must pass before training is allowed to start. It checks class balance, domain distribution across sites, and leakage between splits. It has caught things that would otherwise have cost days of GPU time and, worse, produced a believable number.
Write down what the metric would look like if the thing were broken. If you cannot answer that, you cannot tell success from failure, and you will default to reading any completed run as a successful one.
The uncomfortable part
In every one of these cases the data was the limiting factor, and in every case I spent the first stretch of debugging looking at the model.
That instinct is hard to unlearn, because the model is where the interesting work feels like it lives. But architecture, optimizer, and learning rate are all things I can inspect in a config file in about a minute. The composition of the training set is something I have to deliberately go and measure, and that extra step is exactly why it goes unchecked.
The failure mode is not carelessness. It is that broken data produces output that looks entirely normal.
I work on robotics data pipelines, model evaluation, and the operational side of making training data trustworthy. linkedin.com/in/rickeshnatarajan
Top comments (2)
Example 2 is the one I'd bring up in a post-mortem. We've run into the same thing in extraction work: loss on boundary prediction bottoms out fast, but the metric is full-span F1 with type labels, and those two just don't move together. Your practice of writing what the loss optimizes vs what the eval measures in one line is the clearest forcing function I've seen for catching this before you've spent the compute. Honestly the thing that should've caught ours faster is a short eval pass on a few labeled examples before committing to any run over an hour.
The quiet failure mode is metric substitution. The run is optimizing a legible proxy, then everyone starts treating the proxy as the task because it produced a clean curve. I like the histogram rule. I would add one more artifact before every run, a tiny table of target metric, training loss, and the exact assumption linking them.