I spent the last year building quality gates for AI agent outputs — deterministic verification, diff reviews, delivery checks. I even shipped one for the ECC project (228k stars). It worked.
Then I started fine-tuning models.
Training loss dropped from 9.2 to 8.8. Solid convergence. Everything looked great.
So I ran a test prompt:
19999999999999999999999999999999...
Every prompt. Every time. Perplexity never flagged it.
That's when I realized: this quality-gate philosophy applies at the weight layer too. You just need a different metric.
The gap loss curves don't cover
| Signal | What it tells you | What it missed |
|---|---|---|
| Training loss | "Model is learning" | Output is digit garbage |
| Perplexity | Token-level quality | Mode collapse |
| BLEU/ROUGE | n-gram overlap | Behavioral degradation |
Behavioral Drift: three signals, one score
import evaluate
drift = evaluate.load("behavioral_drift")
r = drift.compute(predictions=ft_outputs, references=base_outputs)
print(r["drift_score"]) # 0.95 = healthy, 0.05 = collapse
Three signals multiplied into one score:
- self-BLEU — output similarity (high = mode collapse)
- digit density delta — numeric characters vs baseline
- repetition ratio — unique token ratio (low = looping)
The bigger picture
Same quality-gate philosophy across layers — from agent reasoning to model training: don't trust the proxy metric; check the actual output.
- hermes-workspace: n=30 causal experiment (p=0.0092)
- gategrow: Agent-layer quality gates
- training-gate: Model-layer quality gates
- PR #778: Metric submitted to HF evaluate (#778, pending review, not yet merged)
Has this happened to you — loss down, model broken?
*中文版：掘金/YuhaoLin2005yhl · Code on [GitHub](https://
What actually happened during training
Here are the real failures, all verifiable against the experiment logs at data/phase3-summary.json and the conversation records:
fp16 crashed silently on step 3. Loss went to NaN with no warning. No CUDA error, no exception — just NaN. The QLoRA adapter gradients on the attention projection layers (q_proj, v_proj) were overflowing fp16's ~65,504 maximum. Switched to fp32 compute dtype. Training ran fine after that — slower, about 40% more time per step — but fine. The config still records this: fp16=False in phase3_lora_train.py.
OOM loading two 4-bit models. The evaluation plan was to run twin model and baseline side-by-side for paired comparison. Two 4-bit quantized models on a 6GB GPU. 6GB minus two ~2GB models minus KV cache minus PyTorch overhead equals negative free memory. Had to load them sequentially. About 60 lines of the eval harness exist solely to manage model swapping.
DPO + QLoRA 4-bit incompatible. The reference model and policy model shared the same quantized base. Their log-probabilities for chosen and rejected responses became numerically identical — the DPO loss collapsed to log(1.0) = 0.0 after one step. I rewrote the entire training loop as expert-guided SFT with rejection sampling instead. Two evenings of debugging to arrive at "abandon DPO."
Astro domain ROUGE-L = 0.0. Not low. Zero. Verified in data/baseline_report.json. The fine-tuned model produced text completely unrelated to the astronomy prompt. Not wrong — orthogonal. It was answering a different question entirely. Fitness domain: ROUGE-L = 0.016, same catastrophic failure pattern.
The SFT loop couldn't tell models apart. Over 4 rounds of expert-guided training (data/growth-log.jsonl), the dual-pool expert system judged 40 response pairs. 32 were ties — the experts genuinely could not distinguish the fine-tuned model's output from the raw base model's output. The win_rate oscillated between 1.0 and 0.0 purely because only 2 pairs per round received non-tie votes. Every single gate_passed was false.
Three signals, none of them loss
Signal 1: ROUGE-L similarity. After fine-tuning, the model's outputs diverged sharply from the base model on the same test set — ROUGE-L dropped by roughly 30%. Not "slightly worse responses." Complete linguistic divergence.
Signal 2: Perplexity variance. Not average perplexity — per-sample variance. In a healthy fine-tune, variance stays low across samples. After step 40, variance tripled. The model wasn't learning general patterns — it was memorizing some samples and abandoning others.
Signal 3: Structured output ratio. The base model followed output format instructions about 70% of the time. After fine-tuning: under 30%. The model stopped listening to instructions in exchange for lower loss.
Three signals, all pointing the same direction. Loss lied to me.
The detector is submitted to HuggingFace evaluate as PR #778 — currently pending review, not merged. It is not a polished product. It is a lesson: never trust a single number to tell you whether your model got better.
What this actually taught me
The core insight isn't about this specific detector. It's about the architecture: in any LLM agent system, you need a deterministic verification layer that doesn't depend on the same probabilistic process you're trying to verify. Loss is a training signal. It optimizes what you ask it to optimize. It does not optimize for "make the model useful."
That's why I ended up building delivery gates (ECC PR #2377, #2378) — Python scripts that check file timestamps, model hashes, and output structure. They don't care about probability distributions. They care about bits on disk.
github.com/YuhaoLin2005)*
🤖 Fact-checked 2026-07-10: GitHub PR status verified against API.
🤖 Fact-checked 2026-07-10: GitHub PR status verified against API. How this works
Top comments (0)