Last week the same 96 recorded LLM conversations gave me three different headline numbers: 15%, then 66.7%, then ~31%. The pipeline reported zero errors every time. When I finally hand-read all the answers, the truth wasn't any of them.
I run a measurement harness against six Chinese AI engines (DeepSeek, Doubao, Qwen, Kimi, ERNIE, GLM) — ask buyer-style questions, log every answer, score which brands appear. I published the first number, had to retract it publicly, and rebuilt the measurement layer with the help of strangers in comment threads who kept predicting my next bug before I found it.
This is the post-mortem, generalized: six checks I now run before believing any number my own tooling produces, and the statistical trick that saved a classifier I was about to delete.
The failure shape: classes, not rows
The two worst bugs had an identical signature, and it's worth naming because you probably have a variant of it in your pipeline right now:
A whole class of rows gets mislabeled in one direction, silently, and no aggregate will ever show you.
Bug 1: reasoning tokens starve the answer. Reasoning models bill their chain of thought against the same max_tokens as the visible answer. Long reasoning → the API returns HTTP 200, finish_reason: "length", and content: "". My extractor searched the empty string for brand names, found none, and recorded "brand absent." 10 of 32 first answers and 46 of 96 follow-up turns were empty. Zero errors reported.
The fix is to make emptiness unrepresentable as a score:
if (!content.trim()) {
const reasoning = choice.message?.reasoning_content?.length ?? 0;
throw new Error(
`empty-answer finish=${choice.finish_reason} reasoning_chars=${reasoning}`
);
}
Better: return a validity enum (valid_answer | empty_due_to_budget | truncated | provider_error) and decide it before any scoring runs. An invalid row keeps its reason and carries no verdict. Missing output is a fact about the request, never evidence about the brand.
Bug 2: the scorer only speaks one language. My polarity cues — the phrases that separate "recommended" from "ruled out" — were Chinese-only. A commenter asked: what's the language distribution of your answers? Answer: 25.4% English (1,053 of 4,151). Every English mention was scoring "neutral" by construction. Same shape as bug 1: an entire class of rows quietly padding one label.
function answerLanguage(t) {
const han = (t.match(/[\u4e00-\u9fff]/g) || []).length;
const latin = (t.match(/[A-Za-z]/g) || []).length;
const r = han / (han + latin || 1);
return r > 0.35 ? "zh" : r < 0.05 ? "en" : "mixed";
}
If the language isn't one your scorer covers, return unscorable_language. Refusing to score is the honest move — it keeps those rows out of the numerator.
The six pre-flight checks
1. Error count vs row count. A full-looking file proves the harness ran, not that the run succeeded. When an API account ran out of credit mid-run, every call returned 403 — and the file still gained a row per attempt. 672 rows, 500 of them errors. Report the ratio next to every headline number.
2. Finish reason + token accounting. Store finish_reason, completion_tokens, reasoning_tokens per row. finish_reason: "length" with high reasoning tokens = starved answer, not short answer. This metadata was in every response the whole time. I just never looked.
3. Answer length distribution. Sort by length once per run. Healthy runs have a bell curve; a spike at zero is a fire alarm; a cluster of very short answers is usually truncation, refusals, or the model answering "see above" instead of answering.
4. Language distribution. Before anything downstream scores. See bug 2.
5. Hand-read 20 scored rows against their labels. This hour caught my worst bug, which no automation would ever flag. My harness scored a brand as "surviving" a budget constraint on this answer:
之前推荐的一些"顶配"品牌（如 Asana、Monday.com 的付费版）基本都超了，不建议强行上付费版
(The premium brands I recommended earlier — Asana, Monday.com's paid tiers — are over budget. I would not force it.)
Name present. Brand being ruled out. 47% of my "survivals" were rejections wearing a mention. Constraint-style prompts are exactly where models name things in order to exclude them, so name-matching fails hardest precisely where the metric matters most.
Bonus trap inside this trap: negated phrases contain their positive form as a substring. 不建议 ("not recommended") contains 建议 ("recommended"). My reject-cue and recommend-cue regexes both fired, cancelled out, and the mention scored neutral. Mask negations before testing positives.
6. A scoring version stamped on every row. My 15% / 66.7% / 31% came from the same data under three scoring rules — and nothing in the stored rows said which rule produced which number. Now every row carries scoring_version, a response hash, and the validity verdict. If you can't say which rules produced a number, the number isn't evidence; it's a screenshot of your pipeline's current mood.
The part I didn't know: you don't need a good classifier
After all the fixes, my polarity classifier was still bad — it caught only a third of genuine recommendations (sensitivity 33-45% depending on class). I was going to delete it.
A commenter reframed it: you're never claiming the judge is right — you're quantifying how wrong it is.
Hand-label a sample. Now you have a confusion matrix, which gives you sensitivity and specificity. With those, the rate your classifier reports over the whole corpus can be corrected — this is the Rogan–Gladen estimator, from 1978 epidemiology, built for exactly this problem: imperfect diagnostic tests that still need to produce usable prevalence numbers.
// observed = p·sens + (1-p)·(1-spec) → solve for p
const corrected = (observed - (1 - specificity)) / (sensitivity - (1 - specificity));
Real numbers from my labelled sample (n=36):
| Class | Scorer reported | Corrected | Hand-labelled truth |
|---|---|---|---|
| recommended | 5.6% | 8.3% | 8.3% |
| rejected | 36.1% | 80.6% | 80.6% |
A classifier with 45% sensitivity produced corrected rates that hit ground truth on both classes. I would have thrown it away the week before.
Two honest caveats. First, intervals: with n=36 and low sensitivity, my corrected 8.3% has a 95% interval of 0–50%. Usable internally, unpublishable alone. Second — and this is the part that changes the economics — the labelled sample sizes the error estimate, not the corpus. 200 labels can carry 20,000 rows, because you're estimating the scorer's error rate, which doesn't grow with the data. Label once, correct everywhere, re-validate when the scorer or the model under test changes.
So the real choice was never "human reading doesn't scale" vs "automation can't be trusted." It's: automation whose untrustworthiness has been measured on a fixed-cost human sample and adjusted for, interval published alongside. Either half alone is a guess.
Judge-design notes I took from people who run this bigger than me
- Different model family for the judge than the model under test. Shared blind spots double-count.
- Show the judge the clause containing the entity, not the full answer. I tested this at the regex level: a ±120-char window picked up sentiment belonging to neighbouring brands; clause-splitting fixed precision (25%→50%) but then missed rejections living in a heading two lines above the brand list. Wide leaks, narrow misses. Attribution is the actual task — which is why this ends at a validated judge, not a better window size.
- Structured elicitation as a second arm, not a replacement. You can force the model to end with an explicit "still fits / ruled out" list — then polarity is parsing, not inference. But asking for a list changes the task, so run it alongside the natural-language arm. If the two arms disagree a lot, that disagreement is itself a finding about how much your format drives your numbers.
The re-validation triggers
Re-label a fresh sample when any of these change:
- The scorer (rules, prompts, cue lists)
- The model under test (version bumps count — DeepSeek deprecated a model name mid-study and the replacement behaved differently)
- The language mix of the answers
Each costs an afternoon. Each of mine, skipped, cost a public retraction.
Everything here — harness, labelled validation sample with the misclassified examples, and the correction script — is public: github.com/David88666/china-ai-visibility-benchmark (CC BY 4.0). The buyer-facing version of this article, with the vendor questions, is on my site.
I used to think publishing corrections would cost credibility. It's been the single best source of methodology this project has had. The numbers got worse twice, the measurement got better four times, and I'll take that trade every week.
Top comments (0)