Every content pipeline eventually hits the same moment: something is rejecting good work, or something is letting garbage through, and you don't know which. We hit it hard. Our QA rail was killing roughly six out of every ten pieces that made it that far, and the easy story -- "the critic is broken, loosen the gate" -- was sitting right there, ready to be believed.
We didn't believe it. We audited instead.
The Baseline
Here's what the pipeline looks like day to day: a writer model (gemma-4-31B) drafts, a critic model (phi4:14b) reviews, and anything scoring under our qa_final_score_threshold of 80 gets one shot at a rewrite before it's killed for good. Over a recent 30-day window, we published 41 pieces at an average version score of 78.5, with 4 sitting in an approval queue. In the same window, 72 pieces got rejected -- 57 outright, 15 after already failing a rewrite attempt. That's a roughly 62% rejection rate on anything that reached QA at all.
That number alone doesn't tell you if the critic is doing its job. It tells you the critic is doing something. The question was whether that something was catching real defects or just being trigger-happy.
Running the Audit
So we pulled all 72 rejects and went through them one at a time, attributing each to whichever gate actually killed it. The breakdown:
| Cause | Count |
|---|---|
| Critic hard-FAIL (only gate that failed) | 20 |
| Programmatic validator FAIL | 29 |
| Brand-fabrication FAIL | 10 |
| No hard fail at all | remainder |
The critic alone only accounted for 20 of the 72. Most of the carnage was coming from the programmatic validator and brand-fabrication checks -- mechanical gates, not the model doing judgment calls.
Where the Rejections Actually Came From
Digging into the programmatic validator failures turned up something upstream of QA entirely: reasoning tokens from models like glm-4.7-5090 and gemma-4:31b were leaking into finished output, corrupting titles and getting flagged as broken structure -- not bad writing. We fixed that at the source with a strip_reasoning_artifacts function rather than telling the critic to look the other way. That's the difference between patching a symptom and fixing the actual defect.
Once we isolated those cases, the picture on the critic's own 20 hard-fails held up. We went through each one manually. The verdict: the critic is exonerated. Phi4 was catching real defects -- thin content, broken claims, structural problems -- almost every time. Our false-positive hypothesis was wrong.
The Critic Is Doing Its Job -- the Problem Sits Upstream
This matters more than it sounds like it should, because the instinct when a gate is expensive is to weaken the gate. That instinct is almost always backwards. If a critic model is rejecting 60% of what it sees and the rejections are justified, loosening the threshold doesn't fix your pipeline -- it just moves the mess further downstream, onto readers.
Fresh-eyes review matters here too. Ours runs as a separate model from the writer specifically so it isn't grading its own homework, the same principle we've written about in how AI amplifies existing engineering practices -- a QA layer that shares blind spots with the thing it's checking isn't a QA layer, it's theater.
The lesson from this audit wasn't "add more automation." It was narrower: audit before you tune. A critic that looks broken because your reject rate is high might just be honest. Find the actual upstream cause -- in our case, leaking reasoning tokens -- before you touch the one gate that's doing exactly what you built it to do.
Top comments (0)