Six checks before you trust any number your LLM pipeline produces

Last week the same 96 recorded LLM conversations gave me three different headline numbers: 15%, then 66.7%, then ~31%. The pipeline reported zero errors every time. When I finally hand-read all the answers, the truth wasn't any of them.

I run a measurement harness against six Chinese AI engines (DeepSeek, Doubao, Qwen, Kimi, ERNIE, GLM) — ask buyer-style questions, log every answer, score which brands appear. I published the first number, had to retract it publicly, and rebuilt the measurement layer with the help of strangers in comment threads who kept predicting my next bug before I found it.

This is the post-mortem, generalized: six checks I now run before believing any number my own tooling produces, and the statistical trick that saved a classifier I was about to delete.

The failure shape: classes, not rows

The two worst bugs had an identical signature, and it's worth naming because you probably have a variant of it in your pipeline right now:

A whole class of rows gets mislabeled in one direction, silently, and no aggregate will ever show you.

Bug 1: reasoning tokens starve the answer. Reasoning models bill their chain of thought against the same max_tokens as the visible answer. Long reasoning → the API returns HTTP 200, finish_reason: "length", and content: "". My extractor searched the empty string for brand names, found none, and recorded "brand absent." 10 of 32 first answers and 46 of 96 follow-up turns were empty. Zero errors reported.

The fix is to make emptiness unrepresentable as a score:

if (!content.trim()) {
  const reasoning = choice.message?.reasoning_content?.length ?? 0;
  throw new Error(
    `empty-answer finish=${choice.finish_reason} reasoning_chars=${reasoning}`
  );
}

Better: return a validity enum (valid_answer | empty_due_to_budget | truncated | provider_error) and decide it before any scoring runs. An invalid row keeps its reason and carries no verdict. Missing output is a fact about the request, never evidence about the brand.

Bug 2: the scorer only speaks one language. My polarity cues — the phrases that separate "recommended" from "ruled out" — were Chinese-only. A commenter asked: what's the language distribution of your answers? Answer: 25.4% English (1,053 of 4,151). Every English mention was scoring "neutral" by construction. Same shape as bug 1: an entire class of rows quietly padding one label.

function answerLanguage(t) {
  const han = (t.match(/[\u4e00-\u9fff]/g) || []).length;
  const latin = (t.match(/[A-Za-z]/g) || []).length;
  const r = han / (han + latin || 1);
  return r > 0.35 ? "zh" : r < 0.05 ? "en" : "mixed";
}

If the language isn't one your scorer covers, return unscorable_language. Refusing to score is the honest move — it keeps those rows out of the numerator.

The six pre-flight checks

1. Error count vs row count. A full-looking file proves the harness ran, not that the run succeeded. When an API account ran out of credit mid-run, every call returned 403 — and the file still gained a row per attempt. 672 rows, 500 of them errors. Report the ratio next to every headline number.

2. Finish reason + token accounting. Store finish_reason, completion_tokens, reasoning_tokens per row. finish_reason: "length" with high reasoning tokens = starved answer, not short answer. This metadata was in every response the whole time. I just never looked.

3. Answer length distribution. Sort by length once per run. Healthy runs have a bell curve; a spike at zero is a fire alarm; a cluster of very short answers is usually truncation, refusals, or the model answering "see above" instead of answering.

4. Language distribution. Before anything downstream scores. See bug 2.

5. Hand-read 20 scored rows against their labels. This hour caught my worst bug, which no automation would ever flag. My harness scored a brand as "surviving" a budget constraint on this answer:

之前推荐的一些"顶配"品牌（如 Asana、Monday.com 的付费版）基本都超了，不建议强行上付费版
(The premium brands I recommended earlier — Asana, Monday.com's paid tiers — are over budget. I would not force it.)

Name present. Brand being ruled out. 47% of my "survivals" were rejections wearing a mention. Constraint-style prompts are exactly where models name things in order to exclude them, so name-matching fails hardest precisely where the metric matters most.

Bonus trap inside this trap: negated phrases contain their positive form as a substring. 不建议 ("not recommended") contains 建议 ("recommended"). My reject-cue and recommend-cue regexes both fired, cancelled out, and the mention scored neutral. Mask negations before testing positives.

6. A scoring version stamped on every row. My 15% / 66.7% / 31% came from the same data under three scoring rules — and nothing in the stored rows said which rule produced which number. Now every row carries scoring_version, a response hash, and the validity verdict. If you can't say which rules produced a number, the number isn't evidence; it's a screenshot of your pipeline's current mood.

The part I didn't know: you don't need a good classifier

After all the fixes, my polarity classifier was still bad — it caught only a third of genuine recommendations (sensitivity 33-45% depending on class). I was going to delete it.

A commenter reframed it: you're never claiming the judge is right — you're quantifying how wrong it is.

Hand-label a sample. Now you have a confusion matrix, which gives you sensitivity and specificity. With those, the rate your classifier reports over the whole corpus can be corrected — this is the Rogan–Gladen estimator, from 1978 epidemiology, built for exactly this problem: imperfect diagnostic tests that still need to produce usable prevalence numbers.

// observed = p·sens + (1-p)·(1-spec)  →  solve for p
const corrected = (observed - (1 - specificity)) / (sensitivity - (1 - specificity));

Real numbers from my labelled sample (n=36):

Class	Scorer reported	Corrected	Hand-labelled truth
recommended	5.6%	8.3%	8.3%
rejected	36.1%	80.6%	80.6%

A classifier with 45% sensitivity produced corrected rates that hit ground truth on both classes. I would have thrown it away the week before.

Two honest caveats. First, intervals: with n=36 and low sensitivity, my corrected 8.3% has a 95% interval of 0–50%. Usable internally, unpublishable alone. Second — and this is the part that changes the economics — the labelled sample sizes the error estimate, not the corpus. 200 labels can carry 20,000 rows, because you're estimating the scorer's error rate, which doesn't grow with the data. Label once, correct everywhere, re-validate when the scorer or the model under test changes.

So the real choice was never "human reading doesn't scale" vs "automation can't be trusted." It's: automation whose untrustworthiness has been measured on a fixed-cost human sample and adjusted for, interval published alongside. Either half alone is a guess.

Judge-design notes I took from people who run this bigger than me

• Different model family for the judge than the model under test. Shared blind spots double-count.
• Show the judge the clause containing the entity, not the full answer. I tested this at the regex level: a ±120-char window picked up sentiment belonging to neighbouring brands; clause-splitting fixed precision (25%→50%) but then missed rejections living in a heading two lines above the brand list. Wide leaks, narrow misses. Attribution is the actual task — which is why this ends at a validated judge, not a better window size.
• Structured elicitation as a second arm, not a replacement. You can force the model to end with an explicit "still fits / ruled out" list — then polarity is parsing, not inference. But asking for a list changes the task, so run it alongside the natural-language arm. If the two arms disagree a lot, that disagreement is itself a finding about how much your format drives your numbers.

The re-validation triggers

Re-label a fresh sample when any of these change:

1. The scorer (rules, prompts, cue lists)
2. The model under test (version bumps count — DeepSeek deprecated a model name mid-study and the replacement behaved differently)
3. The language mix of the answers

Each costs an afternoon. Each of mine, skipped, cost a public retraction.

---

Everything here — harness, labelled validation sample with the misclassified examples, and the correction script — is public: github.com/David88666/china-ai-visibility-benchmark (CC BY 4.0). The buyer-facing version of this article, with the vendor questions, is on my site.

I used to think publishing corrections would cost credibility. It's been the single best source of methodology this project has had. The numbers got worse twice, the measurement got better four times, and I'll take that trade every week.

Comments

[Max Quimby]

"Missing output is a fact about the request, never evidence about the brand" is the whole post in one sentence, and it generalizes way past brand-mention scoring — any classifier that coerces an error state into a valid label is silently editing your denominator.

The validity-enum-before-scoring move is exactly right, and the thing I'd add is to surface the enum distribution as a first-class metric alongside the headline number. Both of your bugs were invisible because the aggregate looked healthy while a whole class of rows silently shifted one direction. If your dashboard shows "31% recommended, but 48% of turns were empty_due_to_budget," nobody publishes the 31% — the invalid-rate is the tripwire. We treat any run where the invalid fraction moves more than a few points from baseline as a failed run, not a lower number.

The reasoning-tokens-starve-the-answer one is especially nasty now that so many models bill hidden chain-of-thought against the same budget — finish_reason: "length" with empty content is a request failure wearing an HTTP 200. Did you end up setting a separate reasoning budget, or just raising max_tokens and detecting the truncation?

[VisibilityAtlas]

Your one-line generalization is better than my whole section — "any classifier that coerces an error state into a valid label is silently editing your denominator" is the thing I spent 400 words circling.

On the enum distribution as a first-class metric: adopted, and I took your stronger version. I was reporting the rate. You're gating on it — "a failed run, not a lower number" — and that distinction matters because a reported rate still lets someone publish the headline with a caveat, which is exactly what I did in July. The harness now prints the validity distribution next to the observation count and compares the invalid fraction against a stored per-platform baseline. Drift beyond 5pp sets a non-zero exit and prints "this is not a lower number, it is a run that did not complete."

Your framing of the invalid-rate as the tripwire rather than a footnote is what makes it work. Both of my bugs would have been caught by that single line.

On your question — I raised max_tokens and detect truncation. I went and tested whether a separate reasoning budget was even available on the endpoint that bit me, and the results are worth reporting:

max_tokens=600, no reasoning control → finish=length, content 233 chars, reasoning_tokens 485
thinking: {type: "disabled"} → finish=stop, content 735 chars
reasoning_effort: "low" → finish=length, content 0 chars, reasoning_tokens 600
thinking: {type: "enabled", budget:200}→ finish=length, content 0 chars, reasoning_tokens 600

The two parameters that look like a reasoning budget are silently ignored — no error, no 400, just accepted and disregarded, and the reasoning consumed the entire 600-token budget leaving nothing for the answer. So on this endpoint the "set a separate reasoning budget" option doesn't exist; it just looks like it does, which is its own version of the same bug class you named.

What actually works is a binary: thinking disabled, or budget generously and detect. I do the latter — max_tokens defaulted up to 3000, empty content raises rather than returns, one retry at double budget, and finish_reason plus reasoning_tokens stored per row.

The reason I don't reach for thinking:disabled even though it works cleanly: it changes what I'm measuring. If a buyer's app serves them a reasoning model's answer, then measuring the non-reasoning variant is measuring a different product. That's a measurement decision wearing a config flag, and I'd rather pay for the tokens than quietly test something my subjects don't use. For providers that expose the toggle I record which mode each row ran in, because a mixed corpus where nobody wrote it down is unrecoverable.

The one I'd flag for anyone else hitting this: "finish_reason: length with empty content is a request failure wearing an HTTP 200" is exactly right, and the nastier variant is finish=length with partial content — 233 usable characters in that first row above. Not empty, so an empty-check passes it, and it's a truncated answer being scored as a complete one. I treat those as truncated in the enum rather than valid, but I'll admit I only added that case after the empty one had already burned me.

[Kartik N V J K]

The detail that stuck with me is the same 96 conversations producing 15%, 66.7%, and 31% while the pipeline reported zero errors every time. Class-level mislabeling in one direction stays invisible to every aggregate, which is why a green pipeline feels safe right until you hand-read the rows. When you rebuilt the measurement layer, did you add per-class sanity checks, or a small hand-labeled holdout you re-score against each run?

[VisibilityAtlas]

Both, but not equally, and your question exposed the gap between them.

Per-class sanity checks: yes, and they came first because they're cheap. Every row now carries a validity enum decided before any content scoring (valid_answer / empty_due_to_budget / empty_unexplained / truncated / provider_error), plus finish_reason, token counts, a response hash, and a scoring_version. Language distribution and an "unscorable_language" rate are reported per run rather than silently filtered — I found 25.4% of my answers were English while my scoring cues were Chinese-only, which is the same class-level failure shape as the empty answers, discovered the same way: someone asked.

The rule I ended up with is that a filter which doesn't report its own size is indistinguishable from a bug.

Hand-labeled holdout: I had one, and until an hour ago I was not re-scoring against it — which is precisely the hole you're pointing at.

36 hand-labelled mentions, taken under scorer version polarity-3-block. Current version is url-predicate-4. So the scorer had changed once since labelling and nothing had ever checked whether that change hurt anything. The holdout existed as an artifact, not as a gate.

So I built the thing you're describing. It re-derives predictions from the same stored answers with whatever the scorer does today, computes per-class precision and recall, diffs against a recorded baseline, and exits non-zero if any class drops more than 5pp:

class support precision recall
rejected 29 100% (+0pp) 52% (+0pp)
recommended 3 67% (+0pp) 67% (+0pp)

The url-predicate change turned out to be safe. But I only know that now, and I'd been running with it for a day.

Two implementation details that made it actually useful rather than decorative:

It fails loudly on enumeration drift. The labels are keyed by position in a deterministic scan, so if the scorer changes what counts as scorable, the label-to-prediction mapping silently misaligns and every metric becomes fiction. It now exits 2 with a different message if the count doesn't match, because a broken comparison is worse than no comparison.

I tested that it fails. I temporarily gutted the rejection cue list to simulate a bad change, and it caught rejected recall dropping 28pp and recommended precision dropping 44pp, exit 1. A regression check nobody has watched fail is a regression check you don't know works — same category of thing as a backup nobody has restored.

Baseline updates require an explicit --accept flag, so a scorer change can never quietly move the bar it's being measured against.

The honest summary of the sequence: per-class rates caught nothing, because a class that's mislabelled consistently produces a stable healthy rate — that's exactly your "invisible to every aggregate" point. What caught things was hand-reading rows, and what makes hand-reading compound rather than evaporate is freezing those reads into a holdout that gets re-scored automatically. I had the reading and the freezing but not the re-scoring, so each fix was a one-off.

n=36 is small and I know it. Next step is stratified labelling — take every row the scorer calls the rare class, sample the common ones, weight back to the population — which gets a usable sensitivity estimate for a fraction of the reading. Someone else in a thread pointed out the arithmetic: at 8% prevalence, random labelling spends most of its budget on the easy majority class.

Scripts are public if useful. The regression one is about 100 lines and mostly the enumeration-drift guard.