I spent months training a 730M model with no tool that could tell me it was already dead

Panagiotis Gkilis — Mon, 03 Aug 2026 12:16:57 +0000

I spent months trying to train a 730M-parameter TTS model on my own hardware. It wouldn't converge, and nothing in my stack would tell me why — not the loss curve, not TensorBoard, not the checkpoints. Every tool I had showed me numbers. None of them would say this run is already dead, stop paying for it.

That situation isn't rare. A fine-tune quietly goes to NaN after step 300 and trains on garbage the rest of the night. A run trains at learning-rate-zero, updating nothing, for six hours. A dataset carries a few hundred broken rows you find at the end. Every time the pattern is identical: the run looks fine, and nothing tells you otherwise until the hours are spent.

So I built the linter I needed.

pip install trainproof
trainproof doctor .   # discovers and judges every training log it finds

MIT, and the interesting part isn't the rules — it's the constraint.

No ML judging ML

There is no model inside trainproof. No "87% confidence this looks unhealthy." Every verdict is a deterministic rule that fires or doesn't, and every finding prints the number it fired on.

That's not minimalism for its own sake. A reliability tool that hallucinates is worse than no tool, because now you don't trust your own alarms — and an alarm you don't trust is just noise. When trainproof can't be sure, it is required to say so rather than guess.

That requirement has a concrete shape. Severity and exit code are separate axes:

Severity	Meaning	Exit
`FAIL`	Your run is broken	1
`WARN`	Something is worth your attention	0
`NOT-CHECKED`	I could not judge this	2
`PASS`	Checked, and fine	0

NOT-CHECKED is the one that matters. A checking tool can say two very different sentences — "your run failed" and "I couldn't read your file" — and a tool that can't tell them apart is lying to your CI in a quiet, expensive way. An unreadable log reports no verdict rather than inventing a failure.

Validating the rules: controlled fault injection

You validate a detector by feeding it faults whose answer you already know, then measuring what it catches. One Qwen2.5-3B QLoRA on an RTX 5080, run six ways, three seeds each — 18 runs:

healthy — the control
lr_hot — learning rate 100× too high
lr_zero — learning rate at zero
fp16_nan — an fp16 overflow that NaNs the loss
bad_labels — labels shuffled into pure noise
overfit — 64 samples, many epochs, with a held-out eval set to prove it

Four of the five failures got caught. lr_hot spiked gradient norm to 2,650 — about 4,900× its own median, flagged in seconds. NaN and flatline were trivial.

One beat every loss rule I had.

The one that fooled it

bad_labels is a dataset that cannot be learned — the labels no longer correspond to the inputs. That run reduced its loss by 69.8% (18.9 → 5.7).

On its own curve it looked like textbook-healthy training. A clean downward slope. It was learning nothing; it was memorizing the statistics of noise, which any sufficiently large network will happily do.

Be precise about the failure, because you can run this yourself: bad_labels doesn't come back clean, it comes back WARN — and the only rule that fires is TP-GRAD-SPIKE, an unrelated gradient blip. Not one loss rule fired. The tool noticed something twitched and completely missed that the dataset was noise.

The lesson is bigger than the tool: a loss curve going down is not evidence that anything real is being learned.

I wrote that straight into the README rather than papering over it. It's also why there's a compare mode — put the run next to a known-good baseline and the relative floor gives it away instantly. One run in isolation lies. Two runs side by side tell the truth.

Then I pointed it at my own real work

Injected faults are a lab. Eventually I ran the shipped rules against real fine-tunes I had actually paid for, on my own GPU. Two results, both in the repo under evidence/:

Real run	Steps	Verdict
Coqui XTTS v2	125,000	FAIL — `TP-DIVERGE`, `TP-THROUGHPUT`
Fish Speech LoRA (Lightning)	2,049	WARN — `TP-OVERFIT`

TP-OVERFIT fires when eval-loss climbs past 1.2× its own minimum while train-loss keeps falling. It warns rather than fails, because early stopping is your call — but it means your best checkpoint has already gone by, and if you're keeping only the last one, you kept the wrong one.

The XTTS run is logged twice, by two entirely separate readers — Coqui's text log and its TensorBoard event file, same run. They produce the same verdict and the same rule set, which is the check I care about most: agreement between independent readers is how you find out your parser is inventing things.

The part I'd defend hardest: it was wrong, and I shipped the fix

Running against real logs also made trainproof report a false FAIL on a perfectly healthy run.

TP-ZERO-GRAD fired whenever every finite gradient norm was exactly 0.0, and reported a severed backward graph. Coqui writes avg_grad_norm as 0.0 when gradient clipping is off. So a healthy 125,000-step XTTS fine-tune whose loss reached 0.017 was reported FAIL by my own tool.

The fix is a piece of reasoning, not a threshold tweak: a run cannot both learn and receive no gradient. The check now stands down when the loss improved beyond a minimum threshold — and records why it stood down as an explicit skip, rather than silently not running.

No test caught that. A real run did. That's the entire argument for shipping evidence logs in the repo instead of only fixtures.

What it actually does

Every stage exits with a status code, so it drops into CI as-is:

Before a GPU-second — preflight and env. Dataset and tokenizer: malformed JSONL with the line number, empty rows, duplicates, missing eos_token, pad == eos. Then the environment: does the training entrypoint import (in a subprocess, so a segfaulting extension can't take the linter with it), is the checkpoint intact, is there RAM and disk.
During — the guardian. A one-line Hugging Face callback. Warns by default; opt into stop_on_fail and it aborts a doomed run itself. It killed a diverging demo run at step 20 of 300.
After — epoch and doctor. Diverged, flatlined, NaN'd, spiked, overfitting.
Against a baseline — compare. The relative-floor rules that catch shuffled labels.

Reads what you already produce: Hugging Face trainer_state.json, Coqui, TensorBoard event files, JSONL, CSV. Nothing to instrument.

One note on the TensorBoard reader, since it's the part I enjoyed most: it's written from the wire format — TFRecord framing plus the Event / Summary / TensorProto fields it needs. It imports no tensorflow, no tensorboard, no protobuf, no torch. It's validated byte-exact against EventAccumulator on a real 2,049-step Lightning run: all 13 tags, all point counts, all values. And a truncated event file — the normal state of a killed run — is read up to the cut instead of raising, because the killed run is the run most in need of judging.

Checkpoints are read without unpickling

torch.load executes arbitrary code by design. That's the reason torch 2.6 flipped weights_only to True.

A tool whose job is to tell you a checkpoint is safe to use, which must run that checkpoint to find out, is not a safety tool. So trainproof reads .pt / .pth / .ckpt as the ZIP archives they are — entry table, storage count, CRC, straight from the archive directory. It distinguishes missing, zero-byte, truncated, CRC-corrupt, and legacy pre-1.6 pickle, which it reports as NOT-CHECKED, because refusing to unpickle is correct behaviour rather than an error.

The truncated case is the one that earns its keep. A save killed mid-write leaves a ZIP header with no central directory, and zipfile.is_zipfile() returns False for it — so the most common real checkpoint failure was being reported as "not a checkpoint at all." That's the difference between resuming from your previous checkpoint and hunting for a file that was never written.

Where it is now

17 releases. 84 stable rule IDs, 230 tests, schema_version 3. Rule IDs are stable and the JSON schema has a written contract in CONTRACTS.md — what each exit code means, when the format may change, that verdicts won't silently drift.

Writing that document found a bug I had already shipped: doctor --baseline printed [FAIL] findings and still exited 0, because the exit code was computed from single-run verdicts and never consulted the comparison. The terminal had been telling the truth; the exit code had been lying. No test caught it, because no test had ever written down which of the two was authoritative.

Every verdict the tool produces on its example runs is frozen in 38 golden snapshots. A rule that silently stops firing fails the build — and so does one that starts firing spuriously.

Try it

pip install trainproof
trainproof doctor .

Repo (with the fault-injection logs, the real 125k-step evidence, and every locked verdict): github.com/Mormolykos/trainproof
PyPI: pypi.org/project/trainproof
Full write-up: ai.bedvibe.studio/trainproof
Sibling: ttsproof — failure-mode QA for text-to-speech
More of what I've built: tts.bedvibe.studio/portfolio
MIT.

If you fine-tune, you've hit at least one of these. I'd genuinely like to know which failure mode has burned your GPU hours — if a deterministic check would have caught it, open an issue and it goes in, with credit.

DEV Community: Panagiotis Gkilis