I rewrote the same detector three times in one day. Each fix introduced a new way of being wrong.

I wrote a post last week about an LLM fabricating a company's official website. Then I built a detector to find more cases at scale.

I have now rewritten that detector three times in one day. Each version fixed the previous version's bug and introduced a new way of mislabelling the same data. The counts went 1 → 134 → 41 → 12.

The interesting part isn't the final number. It's that all three bugs were the same bug wearing different clothes, and I had published an article about that exact bug in between versions two and three.

Version 1: blind to agreement

The detector's job: find domains an LLM calls a brand's "official website" when the brand doesn't own that domain.

// v1
if (engineA.saysOfficial(domain) && engineB.saysNoSuchSite(domain)) {
  flag(domain);
}

It found one real case, which I published. A commenter pointed out the hole:

The disagreement is confirmation rather than the trigger. Any URL an engine calls official can be checked against the brand's owned domain list on its own, single engine, no cross run needed. And you'd catch the case where every engine names the same wrong domain, which is the worse version.

He was right on all counts, and the third one is the serious one. A rule that fires on inconsistency is structurally blind to correlated error — which is the dangerous kind, because models trained on overlapping corpora fail together, and that's exactly when a wrong answer reaches every user at once.

My one published finding had survived only because one engine out of six happened to dissent. Drop that engine from the panel and v1 returns nothing.

Version 2: 134 findings, mostly wrong

The fix seemed obvious. I had ground truth sitting unused — every brand has a list of domains it owns. Check the claim directly:

// v2
for (const url of urlsIn(answer)) {
  const h = hostname(url);
  if (ownedDomains(brand).some(o => h === o || h.endsWith(`.${o}`))) continue;
  if (officialClaimNear(answer, url.index)) flag({ brand, domain: h, engine });
}

134 domains flagged, up from 1. I started drafting this post with a table of the top nine.

Then I looked at the rest of the output instead of the head of it. Domains like airtable-official.com, asana-vip.net, wrike-sales.net, basecampchina.net. Names that pattern-match to fabricated examples, not to real channels.

They were. Here's a representative flagged passage:

检查域名：官方主站是 wrike.com。任何其他域名（如 wrike-official.com、wrike-sales.net）都需要警惕。
(Check the domain: the official site is wrike.com. Be wary of any other domain, such as wrike-official.com, wrike-sales.net.)

The engine is warning users about fake sites. My detector saw a domain near the words "official website" and recorded it as a claim. I was scoring correct model behaviour as a hallucination.

Sampling the flagged set: notion-cn.com, which I had been about to publish as my most alarming new discovery, appeared in a warning context 86% of the time.

Assertion, warning, and denial are three states. I had two.

Version 3: still attributing to the wrong domain

So I added the third state, reusing the polarity logic I'd built three days earlier for brand mentions:

if (negationCues.test(tightClause(answer, i)))  return "denied";
if (warningCues.test(block(answer, i, 260)))    return "warned";
if (assertCues.test(clause(answer, i)))         return "asserted";
return "referenced";

41 flagged. Better. Still wrong, and wrong in a way I should have predicted, because I had written an article about it that week.

Look at that Wrike passage again. The phrase 官方主站 ("official main site") appears roughly thirty characters before wrike-sales.net. Any proximity-based rule attributes it to the nearest domain. But the cue belongs to wrike.com — the domain named before it. The sentence structure is: [real domain, asserted] then [fake domains, warned].

Proximity is not attribution. This is the identical failure I'd published about days earlier, in which a keyword window around a brand name picked up sentiment belonging to a neighbouring brand. Same bug, different entity type, and I walked straight into it.

Version 4: attribution, finally

The fix is to require that no other domain sits between the assertion cue and the domain being classified:

const clause = tightClauseAround(a, index);
const m = ASSERT_CUES.exec(clause);
if (!m) return "referenced";

// Attribution, not proximity: if another domain sits between the cue and this one,
// the cue belongs to that one.
const span = clause.slice(
  Math.min(m.index, hereIdx),
  Math.max(m.index + m[0].length, hereIdx)
);
const domainsBetween = (span.match(DOMAIN_RE) || []).length;
return domainsBetween > 1 ? "referenced" : "asserted";

Final counts over the same 4,023 stored answers:

12 domains asserted as official and not owned
173 domains appearing ONLY in warning contexts

That second number is the one worth sitting with. 173 cases where the engines were doing exactly the right thing — telling buyers to be careful about lookalike domains — and version 2 of my detector counted every one of them as a hallucinated official channel.

If I'd shipped v2, I'd have published a story about LLMs inventing 134 fake official websites. The true story is that they invented a handful and warned about many more, which is close to the opposite.

What I take from this

Three bugs, one shape. Each version failed because I treated nearby text as text about this thing. Windows around brand names, clauses around URLs, blocks around domains — same mistake at three granularities. Entity-level attribution is the actual problem, and no amount of tuning window size solves it. I now assume any new extractor I write has this bug until I've specifically checked.

Reading the head of the output is not reading the output. I drafted a version of this post based on the first fifty lines. The tail was where the story fell apart. Sorting by count and reading the top is how you confirm what you expect.

A detection rule needs a false-positive class, not just a true-positive class. v2 had no concept of "correct behaviour that resembles the thing I'm looking for." Once I named that class, it turned out to be 93% of what I'd flagged.

Three questions I'd now ask of any extractor before trusting its counts:

1. Does it distinguish the model asserting X, warning about X, and denying X? Those look nearly identical in text and mean opposite things.
2. When it finds a cue near an entity, what guarantees the cue is about that entity rather than the one next to it?
3. If the thing you're detecting has a legitimate lookalike — correct behaviour that pattern-matches to the failure — is that lookalike represented in your test cases?

I got all three wrong in a single day, on a codebase whose entire purpose is catching this category of error. The one thing that worked was a stranger asking whether my rule handled a case I hadn't considered, and then me reading the raw output instead of the summary.

---

Harness, scoring module, and labelled validation sets are public under CC BY 4.0: github.com/David88666/china-ai-visibility-benchmark

Comments

[Alex Shev]

This is the part people underestimate about detectors: every fix changes the failure surface. I like keeping a tiny regression set for the old mistake, the new mistake, and the tempting overcorrection.