I rewrote the same detector three times in one day. Each fix introduced a new way of being wrong.

I wrote a post last week about an LLM fabricating a company's official website. Then I built a detector to find more cases at scale.

I have now rewritten that detector three times in one day. Each version fixed the previous version's bug and introduced a new way of mislabelling the same data. The counts went 1 → 134 → 41 → 12.

The interesting part isn't the final number. It's that all three bugs were the same bug wearing different clothes, and I had published an article about that exact bug in between versions two and three.

Version 1: blind to agreement

The detector's job: find domains an LLM calls a brand's "official website" when the brand doesn't own that domain.

// v1
if (engineA.saysOfficial(domain) && engineB.saysNoSuchSite(domain)) {
  flag(domain);
}

It found one real case, which I published. A commenter pointed out the hole:

The disagreement is confirmation rather than the trigger. Any URL an engine calls official can be checked against the brand's owned domain list on its own, single engine, no cross run needed. And you'd catch the case where every engine names the same wrong domain, which is the worse version.

He was right on all counts, and the third one is the serious one. A rule that fires on inconsistency is structurally blind to correlated error — which is the dangerous kind, because models trained on overlapping corpora fail together, and that's exactly when a wrong answer reaches every user at once.

My one published finding had survived only because one engine out of six happened to dissent. Drop that engine from the panel and v1 returns nothing.

Version 2: 134 findings, mostly wrong

The fix seemed obvious. I had ground truth sitting unused — every brand has a list of domains it owns. Check the claim directly:

// v2
for (const url of urlsIn(answer)) {
  const h = hostname(url);
  if (ownedDomains(brand).some(o => h === o || h.endsWith(`.${o}`))) continue;
  if (officialClaimNear(answer, url.index)) flag({ brand, domain: h, engine });
}

134 domains flagged, up from 1. I started drafting this post with a table of the top nine.

Then I looked at the rest of the output instead of the head of it. Domains like airtable-official.com, asana-vip.net, wrike-sales.net, basecampchina.net. Names that pattern-match to fabricated examples, not to real channels.

They were. Here's a representative flagged passage:

检查域名：官方主站是 wrike.com。任何其他域名（如 wrike-official.com、wrike-sales.net）都需要警惕。
(Check the domain: the official site is wrike.com. Be wary of any other domain, such as wrike-official.com, wrike-sales.net.)

The engine is warning users about fake sites. My detector saw a domain near the words "official website" and recorded it as a claim. I was scoring correct model behaviour as a hallucination.

Sampling the flagged set: notion-cn.com, which I had been about to publish as my most alarming new discovery, appeared in a warning context 86% of the time.

Assertion, warning, and denial are three states. I had two.

Version 3: still attributing to the wrong domain

So I added the third state, reusing the polarity logic I'd built three days earlier for brand mentions:

if (negationCues.test(tightClause(answer, i)))  return "denied";
if (warningCues.test(block(answer, i, 260)))    return "warned";
if (assertCues.test(clause(answer, i)))         return "asserted";
return "referenced";

41 flagged. Better. Still wrong, and wrong in a way I should have predicted, because I had written an article about it that week.

Look at that Wrike passage again. The phrase 官方主站 ("official main site") appears roughly thirty characters before wrike-sales.net. Any proximity-based rule attributes it to the nearest domain. But the cue belongs to wrike.com — the domain named before it. The sentence structure is: [real domain, asserted] then [fake domains, warned].

Proximity is not attribution. This is the identical failure I'd published about days earlier, in which a keyword window around a brand name picked up sentiment belonging to a neighbouring brand. Same bug, different entity type, and I walked straight into it.

Version 4: attribution, finally

The fix is to require that no other domain sits between the assertion cue and the domain being classified:

const clause = tightClauseAround(a, index);
const m = ASSERT_CUES.exec(clause);
if (!m) return "referenced";

// Attribution, not proximity: if another domain sits between the cue and this one,
// the cue belongs to that one.
const span = clause.slice(
  Math.min(m.index, hereIdx),
  Math.max(m.index + m[0].length, hereIdx)
);
const domainsBetween = (span.match(DOMAIN_RE) || []).length;
return domainsBetween > 1 ? "referenced" : "asserted";

Final counts over the same 4,023 stored answers:

12 domains asserted as official and not owned
173 domains appearing ONLY in warning contexts

That second number is the one worth sitting with. 173 cases where the engines were doing exactly the right thing — telling buyers to be careful about lookalike domains — and version 2 of my detector counted every one of them as a hallucinated official channel.

If I'd shipped v2, I'd have published a story about LLMs inventing 134 fake official websites. The true story is that they invented a handful and warned about many more, which is close to the opposite.

What I take from this

Three bugs, one shape. Each version failed because I treated nearby text as text about this thing. Windows around brand names, clauses around URLs, blocks around domains — same mistake at three granularities. Entity-level attribution is the actual problem, and no amount of tuning window size solves it. I now assume any new extractor I write has this bug until I've specifically checked.

Reading the head of the output is not reading the output. I drafted a version of this post based on the first fifty lines. The tail was where the story fell apart. Sorting by count and reading the top is how you confirm what you expect.

A detection rule needs a false-positive class, not just a true-positive class. v2 had no concept of "correct behaviour that resembles the thing I'm looking for." Once I named that class, it turned out to be 93% of what I'd flagged.

Three questions I'd now ask of any extractor before trusting its counts:

1. Does it distinguish the model asserting X, warning about X, and denying X? Those look nearly identical in text and mean opposite things.
2. When it finds a cue near an entity, what guarantees the cue is about that entity rather than the one next to it?
3. If the thing you're detecting has a legitimate lookalike — correct behaviour that pattern-matches to the failure — is that lookalike represented in your test cases?

I got all three wrong in a single day, on a codebase whose entire purpose is catching this category of error. The one thing that worked was a stranger asking whether my rule handled a case I hadn't considered, and then me reading the raw output instead of the summary.

---

Harness, scoring module, and labelled validation sets are public under CC BY 4.0: github.com/David88666/china-ai-visibility-benchmark

Comments

[Alex Shev]

This is the part people underestimate about detectors: every fix changes the failure surface. I like keeping a tiny regression set for the old mistake, the new mistake, and the tempting overcorrection.

[VisibilityAtlas]

Built it, and the third category earned its place within about ten minutes of existing.

First thing it exposed, before a single case ran: my existing regression set covered mentionPolarity — and the detector I rewrote four times in one day had no regression coverage at all. Coverage had accumulated where labelling was easy, not where the failure surface was moving. That's backwards, and I'd never have seen it without being made to enumerate per-detector.

Then the overcorrection cases. Writing them is a strange exercise, because you're encoding a fix you deliberately didn't make. Mine:

Citation parser: after a bare domain got counted as a citation because the brand is literally named Monday.com, the tempting fix is "stop counting bare domains." That's a third of all citations gone, and it silently reverts to a narrower contract by accident rather than by decision. Fixture asserts asana.com in prose still counts.
Domain-claim detector: after it scored engines warning about fake domains as engines asserting them (134 false findings), the tempting fix is "if any caution language appears, treat every domain as warned." Caution language is ambient in Chinese commercial answers — that rule deletes the single most important finding in the dataset.
Polarity: after a rejection hidden in a heading was missed, the tempting fix is to widen the context window. Widen it far enough and every brand named anywhere near any negation reads as rejected.
I tested that they can fail rather than assuming: applied the bare-domain overcorrection for real, watched that fixture go red, reverted. A negative assertion nobody has watched fire isn't one.

The thing I'd add to your three. One fixture failed on first run — a genuine assertion sitting just after an unrelated warning gets scored as warned. I nearly "fixed" it. Then I wrote the competing case and found the fix breaks the far more common pattern: a warning cue that has to reach backwards past a list of example domains. That's the pattern that produced the 134 false findings in the first place.

Then I checked real data: the domain in question appears six times in my corpus and is scored 3 asserted, 0 warned. The blind spot was only reachable by a case I constructed.

So it became a fourth category — accepted limitation — asserted against current behaviour, carrying the paired case that shows what fixing it costs and the date I verified it against real data. If someone later "fixes" it, this fails and shows them the bill:

1 known blind spot(s) held open on purpose:
dcs-limit-assertion-inside-warning-window
kept because: adding attribution to the warning branch breaks the
list-of-fakes case, which is far more common and produced 134 false
findings. Under-reporting an assertion is a missed finding;
over-reporting one is a public accusation against a domain owner.
Old mistake and new mistake are memory. Overcorrection is a decision you made once and would otherwise forget. The fourth is a decision you made knowing it's wrong in a specific way — which is the one most likely to get quietly undone by someone acting in good faith, including me in three weeks.

11 fixtures, first gate in the chain. Thanks — this was the highest ratio of insight to effort of anything I've added this month.

[Alex Shev]

That is exactly the kind of failure a regression category should expose. The useful signal is not only that the detector was wrong, but that the test suite had a blind spot shaped like the rewrite. I would probably keep that as a named coverage rule so future fixes have to prove they did not just move the same weakness somewhere else.

[VisibilityAtlas]

That is the remaining gap.

The 11 fixtures prove that the cases I registered still hold. They do not prove that every detector which should be protected is registered, or that each detector still has all three required failure categories.

Right now, a detector could lose its overcorrection case and the suite could still pass because another detector has one. A new detector could also exist outside the fixture registry entirely.

So I’m turning this into a named completeness rule:

• every protected detector must be registered;
• every registered detector needs an old-mistake, new-mistake and overcorrection fixture;
• an accepted limitation must name its paired case, cost and real-data verification date;
• the runner and fixture registries must match;
• a detector-version change without a corresponding coverage-contract update must fail.

That separates two questions I had collapsed:

“Do the fixtures pass?”

and

“Does the fixture set cover the failure surfaces we claim to protect?”

The first protects known behaviour. The second protects where attention goes.

[Alex Shev]

That is a good way to frame it. A regression category is more valuable than a single fix because it changes future review. The best detector work I have seen leaves behind a small corpus of "things this must never miss again," even if the detector itself stays simple.

[Julian Neagu]

This is the part of building detectors that people underestimate. A lot of bugs are not in the model output, they are in how we interpret the output. I’ve hit the same thing with AI tools where the parser became the weak link.