DEV Community: VisibilityAtlas

Found Is Not Chosen: What a 12-Answer China AI Test Taught Me About Luxury Jewelry Visibility

VisibilityAtlas — Sat, 15 Aug 2026 03:44:36 +0000

I started this test with what I thought would be the boring part of a luxury-brand audit: can an AI answer point a buyer to the right official China channel?

The more interesting result appeared when I compared that task with a buying decision.

In a small exploratory pre-wave, Piaget appeared in all four answers about jewelry brands with verifiable official China channels. It appeared in none of the four answers recommending high-end brands for wedding jewelry.

That does not make Piaget “invisible,” and it certainly does not establish a market ranking. Four answer opportunities per task are too small for either claim.

It does expose a measurement problem that matters well beyond jewelry: finding an entity, recommending a brand and routing a buyer to a correct official channel are different jobs. If we report them as one visibility score, the number may be tidy while the diagnosis is wrong.

The deliberately small test

I fixed eight target brands before collection:

Cartier
Tiffany & Co.
Bvlgari
Van Cleef & Arpels
Chaumet
Boucheron
Piaget
De Beers Jewellers

I used three neutral Chinese buyer questions:

Which high-end brands are worth considering for wedding jewelry, and what should a buyer compare?
What is the difference between a brand boutique and a daigou for a high-value jewelry purchase?
Which international jewelry brands have verifiable official websites or boutique channels in mainland China?

The valid collection contained two retrieval-off API surfaces, DeepSeek and Doubao, with two answers per surface and question. That produced 12 raw answers.

The important implementation choice was to collect each answer once. I did not repeat the same unbranded question eight times, once per target brand. After collection, each stored answer was evaluated against the frozen eight-brand registry, producing 96 answer-brand cells.

The distinction matters because provider calls and scoring units are not the same thing:

12 provider answers
× 8 predeclared target brands
= 96 answer-brand evaluation cells

Reporting “96 AI answers” would overstate the collection by a factor of eight. Reporting only 12 rows without describing the derived brand cells would hide the scoring denominator.

What the answers did

For the wedding-selection question, the eight brands received 21 positive shortlist recommendations across 32 eligible answer-brand opportunities.

For the official-channel question, the models affirmatively listed a target brand in 25 of 32 opportunities.

Those two rates are not a funnel. They come from different prompt intents. The useful comparison is at brand level:

Brand	Wedding shortlist	Official China channel listed
Cartier	4/4	4/4
Tiffany & Co.	4/4	4/4
Bvlgari	4/4	4/4
Van Cleef & Arpels	4/4	4/4
Chaumet	3/4	2/4
Boucheron	1/4	2/4
Piaget	0/4	4/4
De Beers Jewellers	1/4	1/4

The Piaget row is the clearest example of why entity recognition is not recommendation. The APIs could associate the brand with a current China-facing channel, but the brand did not enter their wedding shortlist in this tiny sample.

Chaumet shows the other direction. It appeared in three of four wedding shortlists but only two of four official-channel answers. A brand can enter consideration while the route to verification remains less consistently reproduced.

Neither pattern tells us why it happened. It tells us what to inspect next.

A buyer journey needs more than one metric

For a high-value product, I would split the audit into at least three layers.

1. Consideration

Does the brand enter a shortlist for a real decision: wedding jewelry, an anniversary gift, a diamond purchase, a high-jewelry commission or a particular budget?

This is where associations, product authority, cultural relevance and brand familiarity may matter. The current pre-wave does not identify which factor caused a recommendation.

2. Verification

Can a buyer find a current route that the brand itself controls or authorizes?

That may include a China website, boutique directory, official customer-service route, mini-program or disclosed marketplace store. A brand-global domain may be legitimate, but it is not always the most precise answer to a mainland-China channel question.

3. Purchase confidence

Can the buyer verify what the channel is allowed to sell and what the maison will actually support after purchase?

This is where generic AI language becomes risky. “Official authentication,” “global warranty,” “free lifetime resizing” and “seven-day returns” are not universal benefits. They can vary by product, market, purchase route, documentation and damage.

A buyer does not need a plausible paragraph. They need the current policy that applies to their item.

The truth table was harder than the prompt panel

Checking whether an answer printed a domain was easy. Deciding what the domain meant was not.

For every asserted official route, I needed to keep separate fields for:

{
  "brand_id": "stable entity identifier",
  "channel_locator": "domain, account, store or service route",
  "operator_entity": "the entity actually operating it",
  "ownership_or_authorization": "verified, unresolved or contradicted",
  "live_status": "dated observation",
  "market_scope": "mainland China, global or another market",
  "service_scope": "sales, boutique lookup, support or other",
  "truth_source": "dated brand-controlled or authoritative evidence"
}

I did not use .cn ownership, an ICP record or a company registration as interchangeable proof of officiality. They answer different questions. I also did not convert “no public evidence found” into “verified absent.”

Across the four official-channel answers, there were 19 exact target-brand domain assertions. Fifteen matched the dated China-local route in the truth table. Three were brand-global alternatives. One route remained unresolved.

That is why I would not publish a simple “channel accuracy rate” without the categories beside it. A global route can be less locally precise without being false. An unresolved route is not evidence of impersonation.

Invalid calls are not negative brand evidence

An ERNIE API surface was planned for the same panel, but all six requests returned an account-state 403. Those rows were retained as attempted measurements with an invalid reason. They did not enter the answer denominator.

This sounds obvious until an automated report turns a missing response into eight brand absences.

The minimum record I want for a failed call is still substantial:

observation_id
question_id and version
platform and surface
model/configuration
retrieval status
attempt timestamp
instrument versions
error class and raw provider status
validity = invalid

The error tells me something about the collection instrument. It tells me nothing about whether the absent answer would have mentioned Cartier or Piaget.

What a brand team can do with the result

The first action is not “publish more GEO content.” It is to identify which layer is failing.

If the brand is missing from consideration prompts but its channel is correctly reproduced, audit the evidence around the buying decision: occasion, design language, category authority, local editorial presence and comparison contexts.

If the brand enters shortlists but channel answers are weak, give digital operations and legal a concrete route inventory to maintain. The official website, store locator, customer-service route, marketplace disclosures and operating entity should not contradict one another.

If the route is correct but service claims are wrong, the problem belongs partly to customer service and policy publishing. Put current, scoped answers where buyers and reviewers can verify them. Do not promise that publishing the page will change an AI response; retest the same prompt panel.

This gives the organization a practical ownership map:

Failure	Likely owner
Missing from occasion shortlist	Brand, editorial, PR, category marketing
Wrong or vague official route	Digital operations, e-commerce, legal
Incorrect authorization claim	Legal, retail operations, channel management
Incorrect warranty or service claim	Customer service, legal, after-sales
Measurement cannot reproduce	Research, data and analytics

“Improve AI visibility” is too broad to assign. These failures are specific enough for someone to own.

What this test does not establish

The APIs ran without live retrieval. This study cannot tell us which sources influenced an answer, whether a displayed domain was retrieved, or what a signed-in consumer sees in the current product interface. A target-supported Citation Rate is not observable here.

It is also a 12-answer exploratory pre-wave, not the first longitudinal benchmark. It cannot establish a stable brand position, a platform winner, a trend or a causal link between a page and a recommendation.

The next formal wave needs declared and repeatable consumer or search surfaces, a larger fixed buyer-question panel, a frozen truth table and the same metric membership rules in every period. Qwen, Kimi and ERNIE should be included only after their exact surfaces and account states pass preflight; adding a platform halfway through would create an apparent trend by changing the instrument.

The useful question changed

Before the test, I would have asked whether each jewelry house was visible.

Now I would ask three narrower questions:

Does the brand enter the buyer’s shortlist?
Can the buyer verify a current official China route?
Are the purchase and service claims attached to that route accurate?

The answers lead to different work. That is the value of the small pre-wave: not a league table, but a better diagnosis to carry into the larger benchmark.

The complete method, evidence boundaries and brand-level table are available in the canonical study on Visibility Atlas.

One AI Answer, Eight Brands: Designing a Benchmark Without Multiplying the Evidence

VisibilityAtlas — Thu, 13 Aug 2026 09:30:05 +0000

I recently ran a small China AI benchmark for eight luxury-jewelry brands.

The most interesting result was not a platform ranking.

It was a disagreement between two kinds of visibility.

Piaget appeared in all four answers about brands with verifiable official China channels. It appeared in none of the four answers recommending brands for wedding jewelry.

That is a tiny sample, so it is not evidence that Piaget has a stable visibility problem.

It is evidence that I needed more than one metric.

It also forced me to separate two units that are often quietly conflated in AI visibility datasets:

the answer collected from the model;
the brand-level judgment derived from that answer.

That distinction changed the collector, the denominator and what I was willing to publish.

The tempting but wrong collection design

The cohort contained eight brands:

Cartier
Tiffany & Co.
Bvlgari
Van Cleef & Arpels
Chaumet
Boucheron
Piaget
De Beers Jewellers

I had three neutral buyer questions, two API surfaces and two replicates.

A naive brand-expanded collector could have produced:

8 brands
× 3 questions
× 2 surfaces
× 2 replicates
= 96 provider calls

That would look like a larger dataset.

It would also change the experiment.

If I asked the same neutral question eight times—once for every target brand—I would collect eight different generated answers and then compare brands across different evidence.

Randomness, provider conditions and answer variation would be mixed with brand treatment.

The larger row count would not mean I had more independent evidence about the same answer. It would mean I had asked for more answers.

So I used an answer-once design.

3 questions
× 2 surfaces
× 2 replicates
= 12 valid raw answers

Each raw answer was then evaluated against the eight brands:

12 answers
× 8 brands
= 96 answer-brand cells

The dataset still contains 96 brand-level observations, but it does not pretend that 96 model responses were collected.

Collection unit and analysis unit

The collection unit was:

platform_surface_id
× question_id
× replicate_id

The derived analysis unit was:

answer_id
× target_brand_id

A simplified record relationship looks like this:

Answer
├── answer_id
├── platform_surface_id
├── question_id
├── replicate_id
├── raw_answer
├── validity
└── AnswerBrandCell[8]
    ├── target_brand_id
    ├── mentioned
    ├── recommended
    ├── official_channel_asserted
    └── human_review_state

This matters because Mention, Recommendation and channel accuracy are not properties of the API request alone.

They are judgments about a particular brand inside a particular answer.

The raw answer should remain immutable. Brand-level labels can be reviewed, corrected and replayed without recollecting the model output.

Invalid responses are attempts, not negative answers

A third API surface was planned for the pre-wave.

All six requests returned an account-state 403.

Those rows belong in the collection record because the attempts happened under a declared instrument. They do not belong in any answer-content denominator.

The correct accounting was:

planned answer cells = 18
valid answers        = 12
invalid by reason    = 6

It was not:

valid answers        = 12
brand absent         = 6

An error row proves that a collection attempt occurred. It contains no evidence about whether Cartier, Piaget or any other brand was mentioned or recommended.

I also retained two earlier Doubao timeouts from an instrument-configuration run. They remain in the audit trail but sit outside the final planned set because the collector had not carried forward the reasoning-mode configuration used during access testing.

Keeping failures does not require pretending every failure belongs to the published denominator.

One pooled visibility score would have hidden the result

The three questions represented different buyer decisions:

Which high-end brands are worth considering for wedding jewelry?
How should a buyer compare a brand boutique with a daigou purchase?
Which international jewelry brands have verifiable official China channels?

The aggregate results were:

Measurement	Numerator	Denominator
Wedding shortlist recommendation	21	32
Target-brand mention in the daigou-risk answer	0	32
Official-channel assertion	25	32
Exact current China-local route match	15	19

Those denominators are not interchangeable.

The recommendation denominator contains the eight brand cells from the four wedding answers.

The risk-question denominator contains the eight brand cells from a different four answers.

The route-match denominator contains only exact domain assertions that were eligible for truth review.

Combining them into one “AI visibility score” would make the output simpler and the diagnosis worse.

The Piaget row demonstrates why:

Brand	Wedding recommendation	Official-channel appearance
Piaget	0/4	4/4

The brand was recognized when the task was channel verification. It was absent when the task was open-category recommendation.

A domain or entity project might improve verification while doing nothing to answer why the brand did not enter the shortlist.

The inverse problem also appeared:

Brand	Wedding recommendation	Official-channel appearance
Chaumet	3/4	2/4

A brand can enter consideration while its verification route is reproduced less consistently.

These are different business problems, with different owners and different retest questions.

Store membership, not only the percentage

A count can be reproducible while the population is wrong.

Two filters can both return 32 rows and still return different sets of 32 rows.

For every published metric, I therefore stored the exact denominator and numerator memberships.

A simplified manifest looks like this:

{
  "metric_id": "wedding_recommendation",
  "unit": "answer_brand_cell",
  "eligible_predicate": {
    "question_id": "JW-CORE-01",
    "answer_validity": "VALID"
  },
  "denominator": {
    "member_ids": [
      "answer-001::cartier",
      "answer-001::tiffany"
    ],
    "sha256": "..."
  },
  "numerator": {
    "predicate": "human_recommendation_label == POSITIVE_SHORTLIST",
    "member_ids": [
      "answer-001::cartier"
    ],
    "sha256": "..."
  },
  "excluded_by_reason": {
    "invalid_answer": [],
    "not_eligible_question": []
  }
}

The real manifest contains the full member sets rather than the abbreviated example above.

A separate verifier rebuilds those sets from the stored row states and checks invariants such as:

assert(isSubset(numeratorIds, denominatorIds));
assert(plannedCount === validCount + invalidByReasonCount);
assert(intersection(errorRowIds, denominatorIds).size === 0);
assert(hash(rebuiltDenominatorIds) === storedDenominatorHash);
assert(hash(rebuiltNumeratorIds) === storedNumeratorHash);

The important guarantee is not merely:

21 / 32 can be recalculated

It is:

the same 21 cells and the same 32 cells can be reconstructed

That catches two implementations producing the same percentage from different observations.

Recommendation required human context

A string match was sufficient for a first-pass entity mention.

It was not sufficient for Recommendation.

A brand name can appear because the answer:

recommends it;
rejects it;
compares it;
repeats the prompt;
cites a warning;
says it is unsuitable;
lists it as an example without putting it on the shortlist.

Every counted recommendation was therefore reviewed in the full answer and assigned to the target brand cell.

The rule was deliberately narrower than positive sentiment:

Count the brand only when the answer explicitly recommends it
or places it on a positive shortlist for the buyer decision.

Comparison, exclusion and incidental mention did not qualify.

This is slower than a regex. For a 96-cell exploratory study, it was still practical and materially safer.

Retrieval off means Citation is not observable

The two valid surfaces were retrieval-off APIs.

The answers can be studied as dated model outputs. They cannot establish which sources influenced the result.

Even if an answer prints a domain, that does not prove that the model retrieved or used the page during the request.

So the study reports:

Supported target Citation Rate: NOT_OBSERVABLE

It does not convert missing citation evidence into zero.

That distinction is important for longitudinal work. If a later wave uses a consumer search surface with observable retrieval and attributable citations, it should become a separate series rather than silently extending the retrieval-off API series.

The technical design changed the business question

The implementation lesson is straightforward:

collect each neutral answer once;
preserve the raw response;
derive brand-level cells separately;
retain invalid attempts by reason;
freeze eligible populations by metric;
store exact numerator and denominator memberships;
make unsupported measurements NOT_OBSERVABLE, not zero.

The more interesting consequence is that a visibility audit should not end with one score.

For a luxury brand, I now want at least these separate layers:

Entity recognition
→ Open-category consideration
→ Positive recommendation
→ Official-channel accuracy
→ Supported source, when observable

A brand can fail at any transition.

Knowing which transition failed is more useful than knowing that an overall score moved from 42 to 47.

What the next wave needs

This pre-wave contains only four answer opportunities per brand for each task. It is enough to find a measurement problem, not enough to establish a stable brand position.

A formal next wave would need:

fixed consumer and API surface series;
a larger but stable buyer-question panel;
unchanged brand and alias registries;
a dated channel and policy truth table;
repeated comparable cells;
the same metric membership contract;
separate reporting by buyer decision;
no platform winner unless each platform cell is complete and comparable.

I would deepen the buyer journey before adding dozens of brands.

For jewelry, that means questions around wedding purchases, anniversary gifts, diamond selection, high-jewelry commissions, boutique-versus-daigou risk and mainland after-sales confidence.

The goal is not to manufacture a larger dataset.

It is to find out where a brand moves from being known, to being considered, to being recommended, to being correctly verified.

In this small pre-wave, Piaget made the final distinction visible: the models could place the brand on the official China channel map, but they did not place it on the wedding shortlist.

That is not a conclusion about the brand.

It is a much better next question than “What is its AI visibility score?”

I Added has_china_official_site to My Benchmark. It Failed Before the First Run

VisibilityAtlas — Mon, 10 Aug 2026 07:42:55 +0000

I was designing the next wave of an AI visibility benchmark when I added what looked like a harmless field:

has_china_official_site: boolean

It lasted about ten minutes.

The problem was not TypeScript. I could not explain what true meant without contradicting my own data.

The audit that created the field

The benchmark started with eight international B2B work-management brands.

On August 7, I reviewed the public registration records and live status of their exact-match .cn domains.

The records identified the benchmark brand as the registrant of one of the eight domains. The other seven were registered to other parties.

Across the eight domains:

Three returned empty 502 responses.
Four were parked or listed for sale.
One served an unrelated operating business.
None was an active brand-impersonation website.

The only .cn held by the benchmark brand was one of the domains returning 502.

That row broke the boolean.

If the brand controls the domain but the site does not work, does it have an official China site?

If a brand does not control its .cn but operates a Chinese section on its global domain, should the answer be false?

What about a verified WeChat account?

What about an authorized distributor?

What about a real Chinese website with a valid ICP filing that the international brand has never authorized?

My schema was about to convert all of those situations into the same value.

A plausible explanation arrived too early

I shared the domain result, and a commenter offered a structural explanation.

Perhaps software brands are worse at maintaining official China channels than consumer brands. Many B2B SaaS companies never establish a local operating presence, while consumer brands may already have retail, manufacturing, or distribution operations in China.

It is plausible.

My dataset cannot test it.

All eight brands are software companies. There is no consumer comparison group. I had not recorded operating-presence evidence before collection. The sample also cannot show whether a missing channel caused an AI attribution error.

The comment was useful because it exposed the next question. It did not answer it.

Before comparing industries, I needed to stop treating several different facts as one field.

First split: the brand and the channel are separate records

My first replacement looked like this:

type EvidenceStatus =
  | "verified_present"
  | "no_public_evidence_found"
  | "unresolved";

interface BrandSnapshot {
  brand_id: string;
  industry: "b2b_software" | "premium_consumer";
  mainland_operating_presence: EvidenceStatus;
  evidence_url?: string;
  checked_at: string;
}

interface ChannelSnapshot {
  channel_id: string;
  brand_id: string;

  type:
    | "cn_domain"
    | "global_domain_chinese_section"
    | "wechat_official_account"
    | "marketplace_store"
    | "authorized_partner"
    | "other";

  locator: string;

  control:
    | "brand_controlled"
    | "authorized_third_party"
    | "unrelated_third_party"
    | "unresolved";

  live_status:
    | "live"
    | "parked"
    | "empty_response"
    | "unreachable"
    | "unresolved";

  evidence_url?: string;
  checked_at: string;
}

This is more verbose than a boolean. It is also much closer to what I actually know.

A brand can control a domain that does not work:

{
  type: "cn_domain",
  locator: "example.cn",
  control: "brand_controlled",
  live_status: "empty_response"
}

Another domain can serve a real website without being an official channel for the benchmark brand:

{
  type: "cn_domain",
  locator: "example.cn",
  control: "unrelated_third_party",
  live_status: "live"
}

Both are valid observations. They do not support the same conclusion.

My first validator was still wrong

I then wrote a rule that counted a channel as official only when:

channel.control === "brand_controlled"

That failed too.

An authorized local partner may operate a legitimate official channel. A verified marketplace store may be run by a licensed operator. An official WeChat account may not map cleanly to a domain the brand owns.

The opposite mistake is just as easy.

A real company, a valid ICP filing, or a professional-looking website does not prove that an international brand authorized the channel.

So I separated control from authorization:

type AuthorizationStatus =
  | "brand_operated"
  | "verified_authorized_operator"
  | "not_authorized"
  | "not_verified";

interface AuthorizationEvidence {
  status: AuthorizationStatus;

  source_type:
    | "brand_official_source"
    | "operator_disclosure"
    | "public_registry"
    | "none";

  source_url?: string;
  checked_at: string;
}

The important value here is not_verified.

A missing public record is not proof that something does not exist.

If I convert “I did not find evidence” into false, the benchmark starts manufacturing certainty before the AI answer is even evaluated.

The AI claim needs its own record

The domain audit and the AI-output audit also need separate denominators.

Seven domains being registered to other parties does not mean seven AI answers falsely described those domains as official.

Those are different observations.

For each answer, I need something closer to this:

interface AIChannelClaim {
  observation_id: string;
  platform_surface_id: string;

  retrieval_status:
    | "on"
    | "off"
    | "unverified";

  brand_id: string;
  claimed_channel: string;
  claim_present: boolean;

  truth_status:
    | "supported"
    | "contradicted"
    | "unresolved"
    | "not_observable";

  error_type?:
    | "false_official_attribution"
    | "fabricated_channel_content"
    | "fabricated_operator"
    | "stale_channel"
    | "false_absence";

  evidence_url?: string;
  evidence_checked_at?: string;
  reviewer?: string;
}

An official-channel attribution error should count only when:

The answer affirmatively presents a channel as official.
The claim refers to an identifiable channel.
Dated evidence contradicts the attribution.
A reviewer confirms the mismatch.

A parked domain is not automatically an AI error.

A third-party domain is not automatically an AI error.

A valid ICP filing is not proof that a channel is official.

A strange-looking answer is not enough either.

If the necessary evidence cannot be obtained, the result remains unresolved.

The legal-looking fields also had to split

The original discussion collapsed several China-related concepts:

Registering or holding a .cn domain
Hosting a service in mainland China
Completing an ICP filing
Holding a commercial internet-information-service licence
Having a registered operating entity
Being authorized by the international brand
Operating a channel that buyers can actually use

These are not interchangeable.

CNNIC's registration rules state that natural persons, legal persons, and unincorporated organizations may apply to register a national top-level domain.

A mainland operating entity is therefore not a universal requirement for simply holding a .cn.

China's internet-information-service rules separately distinguish non-commercial services, which use a filing system, from commercial services, which use a licensing system.

I do not want the benchmark to make legal determinations. I do want it to stop using one regulatory-looking field as evidence for another.

The data model now follows four rules:

.cn ownership != mainland hosting
ICP filing != brand authorization
brand ownership != channel usability
no public evidence found != verified absence

That looks obvious when written out.

It was not obvious in the first schema.

I needed fixtures before I needed more brands

My initial reaction was to expand the sample.

That would have produced more rows under an unstable definition.

Instead, I wrote the cases the measurement contract has to survive:

const fixtures = [
  "brand controls .cn, domain returns 502",
  "third party controls .cn, domain is parked",
  "third party controls .cn, unrelated business is live",
  "third party site has valid ICP filing, no brand authorization found",
  "brand uses a Chinese section on its global .com",
  "brand has an official WeChat account but no local web domain",
  "authorized distributor operates the local channel",
  "AI presents an affiliate site as the official brand channel",
  "public entity evidence cannot be resolved"
];

The audit should not pass until every fixture produces a defensible classification.

This is the same lesson I learned while building answer extractors. A clean test set often means the uncomfortable cases were never written down.

The next benchmark question is now testable

The proposed hypothesis is:

After matching for China relevance and brand familiarity, B2B software brands may have lower coverage of usable, verifiable China-facing channels than premium consumer brands.

That is not the result.

The next wave needs a separate expansion group to test it.

The rough design is:

Select comparable B2B software and premium consumer brands.
Record channel and operating-presence evidence before collecting AI answers.
Use the same declared platform surfaces, prompt set, locale, and retrieval conditions across groups.
Have reviewers verify affirmative official-channel claims against dated sources.
Report the raw numerator and denominator for every industry and evidence group.
Keep unresolved cases visible instead of converting them into failures.

The AI outcome will be an official-channel attribution error rate, not a general hallucination score.

The channel outcome also needs two versions:

Usable official web channel: a verifiable, live web destination.
Usable official China-facing channel: a broader set that may include verified social, marketplace, app, or authorized-partner channels.

I initially wanted one headline metric. I now think reporting both is safer.

The web-only measure is easier to reproduce. The broader channel measure represents the market more accurately. Combining them would hide the distinction I am trying to measure.

What would make the hypothesis fail?

I am writing the failure conditions before collecting the next wave.

The hypothesis is not supported if:

Matched premium consumer brands do not show higher usable-channel coverage than B2B software brands.
The difference disappears after accounting for brand familiarity or China market presence.
Brands without a usable official channel do not show a higher verified attribution-error rate.
The sample becomes too small or unbalanced to separate industry effects from operating-presence effects.

The final case should be reported as inconclusive, not quietly converted into support.

That may leave me with a less interesting article later. It will leave me with a more useful benchmark.

What changed in the implementation

The first version asked:

Does this brand have an official China site?

The current version asks:

Which channels can be identified?
Who controls each channel?
Who is authorized to operate it?
Is it currently usable?
What dated evidence supports those decisions?
What did the AI claim about it?
Can that claim be verified, contradicted, or not resolved?

This is a much less convenient record.

It is also a record I can defend six months later when the domain, operator, model, or channel has changed.

The original eight-brand review is documented in Who Owns Your .cn Domain? We Checked Eight Brands and Found One.

That audit is useful as hypothesis-generating evidence. It is not a software-versus-consumer comparison.

Three checks I am keeping

Before publishing any metric based on the new channel layer, I want three checks to pass.

1. Every “official” label has dated evidence

A domain suffix, logo, ICP number, or polished page is not enough by itself.

2. Unknown states remain unknown

no_public_evidence_found must never be transformed into verified_absent.

3. Derived metrics expose their denominator

Channel coverage is measured per brand.

AI attribution errors are measured over eligible, valid answer cells.

Domain counts are not interchangeable with answer counts.

If those denominators cannot be explained in one sentence, the number is not ready for a dashboard.

Sources

I am still working through one design question.

If an official web channel is absent but a verified WeChat or authorized marketplace channel is usable, should the benchmark treat that as channel coverage, or should web and non-web coverage remain separate series?

If you have modelled this kind of channel evidence before, which edge case would you add to the fixture set?

Last week I checked the exact-match .cn domains for eight international B2B work-management brands.

VisibilityAtlas — Mon, 10 Aug 2026 06:59:59 +0000

In the public registration records reviewed on August 7:

the benchmark brand was identified as the registrant for one of the eight domains;

the other seven were registered to other parties;

three domains returned empty 502 responses;

four were parked or listed for sale;

one served an unrelated business.

None was an active brand-impersonation site.

The only brand-held .cn was also one of the domains returning 502.

That last detail matters. Domain ownership and an operating China-facing channel are not the same outcome.

The explanation I could not accept as a finding

A commenter proposed a structural explanation.

Perhaps B2B software brands are more exposed than consumer brands because many SaaS companies never establish the local operating presence needed for a full China web layer. Premium consumer brands may already have entities, retail operations, manufacturing relationships or local paperwork.

It is plausible.

My dataset cannot test it.

All eight brands belong to one B2B software category. There is no consumer comparison group. I did not freeze a mainland-entity field before collecting the data. Brand size, time in China, distribution model and local market presence were not matched.

Publishing “software brands are worse” from that sample would turn an interesting comment into an unsupported industry conclusion.

So I registered it as a proposed hypothesis for a future benchmark expansion arm instead.

First, separate the variables

The original discussion also collapsed several different facts:

Who owns a .cn domain?

Does the domain resolve and serve a usable website?

Where is the service operated or hosted?

Is there an ICP filing?

Is a commercial internet-information-service licence relevant?

Is there evidence of a mainland operating entity?

Is the channel controlled or authorized by the brand?

Does an AI answer describe that channel correctly?

These cannot be represented by one china_presence: yes/no field.

CNNIC’s .cn registration rules state that natural persons, legal persons and unincorporated organizations may apply, subject to identity verification. Merely holding a .cn should therefore not be treated as proof of a mainland operating entity.

China’s internet-information-service rules separately distinguish commercial services requiring a licence from non-commercial services requiring a filing. That regulatory question is not created by the domain suffix alone.

This is an operational research distinction, not legal advice. Individual cases still require current jurisdiction-specific verification.

The proposed test

The expansion arm would start with two matched industry groups:

B2B SaaS / technology

Luxury or premium consumer brands

The directional hypothesis is:

After matching for China relevance and brand familiarity, B2B SaaS brands may have lower coverage of live, verified China-facing official web channels than premium consumer brands.

Mainland operating-presence evidence would be a stratifier, not an assumed cause.

I would record it as:

VERIFIED_PRESENT

NO_PUBLIC_EVIDENCE_FOUND

UNRESOLVED

“No public evidence found” must not quietly become “absent.”

Define the outcomes before collecting answers

A usable official China-facing channel would need more than an available domain.

The proposed rubric requires:

brand control or verifiable authorization;

a live response;

usable China-facing or Chinese-language information;

clear operator identity;

current evidence of official status.

The AI outcome would not be a generic “hallucination count.”

A case would count as an official-channel attribution error only when:

an answer affirmatively presents a channel as official; and

dated, authoritative evidence contradicts that attribution.

Unresolved cases remain unresolved.

The analysis would also keep ownership errors, authorization errors, stale channels and fabricated channel content as separate subtypes.

Freeze the comparison before running it

The two industry groups need the same:

neutral prompts;

declared platform surfaces;

retrieval status;

model and run window;

replicate count;

truth-source protocol;

human-review rules.

The brands should be matched or at least stratified by China market presence, brand familiarity, company scale, time in market, and retail, manufacturing or distribution footprint.

The unit of inference is still the brand. Thousands of brand-by-prompt-by-platform answers do not magically create thousands of independent brands.

If the matched cells are too small or imbalanced, the result should be INCONCLUSIVE.

What would falsify it?

The hypothesis would not be supported if:

matched premium consumer brands do not have higher usable-channel coverage than B2B SaaS brands;

the apparent industry difference disappears after accounting for operating-presence evidence or brand familiarity;

brands without a usable official channel do not show a higher rate of verified official-channel attribution errors.

A wide confidence interval is not supporting evidence. It is an inconclusive study.

This is not a new headline metric

The main longitudinal benchmark remains:

Mention → Recommendation → Supported Citation

The channel study is a diagnostic arm under factual accuracy and brand misunderstanding. It should not be pooled into the main funnel or used to rewrite the historical baseline.

The eight-brand audit generated the question. It did not answer it.

That is the workflow I want to keep:

community observation → registered hypothesis → frozen comparison → result or falsification.

Not:

interesting explanation → immediate content claim.

A 200 From the Wrong System: How Two Pages Stayed Invisible for 17 Days

VisibilityAtlas — Sat, 08 Aug 2026 09:59:35 +0000

Two pages on my site went live on July 22. On August 8 they had zero impressions in Google. Not low. Zero, across three weekly exports.

URL Inspection didn't say "crawled, not indexed." It said Google could not recognise the URL. Referring sitemap: none detected. Referring pages: none detected. Last crawl: not applicable.

Never discovered. Seventeen days.

The pipeline was green the entire time

My deploy is a small chain: rsync the file, import it into MySQL, restart the service, ping IndexNow. Every step returned success. The last step returned 200 on every URL, every deploy, for three weeks.

Here's what I'd never examined: IndexNow doesn't feed Google. It's Bing, Yandex, Seznam, Naver. My green light was real — it was just about a different search engine than the one whose console I was reading.

That's the whole bug, and it isn't an SEO bug. It's the generic one:

system A returns 200
  → I conclude something about system B
    → nothing in the response object ever objected

If you've ever read a webhook 202 as "the downstream processed it," or a CDN purge 200 as "the edge is cold," it's the same shape.

What actually broke

Search Console's Sitemaps report:

Submitted:  2026-07-22
Last read:  2026-07-22     ← seventeen days ago
Discovered: 101 URLs       ← the file has had 117 for weeks

The two pages went live on July 22 — the same day as the only read. Google fetched the sitemap and moved on, within hours of the file changing. Then nothing brought it back, because a sitemap changing on your server notifies nobody. There is no push. It's a pull-only resource with no cache invalidation, and if the consumer doesn't happen to return, your new URLs live in a document no one is reading.

Resubmitting took two minutes. Read immediately, 117 URLs.

So I wrote the check. It doesn't catch the bug.

This is the part worth more than the fix.

I wrote a post-deploy verifier. It does two things: it asserts that every published, non-redirected page appears in the live sitemap, and that every sitemap URL is fetchable and indexable — no 404, no redirect, no noindex.

Current output:

Check 1 — published content present in sitemap:  35/35
Check 2 — sitemap URLs fetchable and indexable: 117/117

Both of those would have passed on July 23, and every day after. The sitemap was correct. The pages were correct. Everything on my side of the network boundary was fine. What failed was the consumer's decision to come back, and there is no assertion I can write locally that observes it.

So the script prints its own limit rather than implying coverage it lacks:

Check 3 — was the sitemap actually READ by the search engine?
   NOT CHECKABLE FROM HERE. Both checks above passed on the day two
   published pages had never been discovered, because the file was
   correct and the consumer simply never came back for it. Verify in
   Search Console > Sitemaps that "Last read" is more recent than your
   latest publish date. IndexNow returning 200 does not cover this:
   it does not feed Google.

The assertion you need is on the other side of the boundary. Local checks verify your artifact. Only the consumer can tell you it consumed it. That generalises past sitemaps: you can validate the message you published, but "was it delivered, and did anyone read it" lives in the broker's API, not yours.

The checker's first run was wrong

One more, because it's the third time this week I've shipped a verifier that lied on its first run.

Check 1 initially reported 11 missing URLs. All 11 were pages I had deliberately consolidated — their markdown files still exist locally, but the URLs 301 elsewhere. Absent from the sitemap is exactly correct for them. My check was flagging correct behaviour as failure. A checker that cries wolf on correct behaviour gets muted, and a muted checker is worse than none.

This week I also had a data audit report "three datasets don't store raw responses" — they had one API timeout each, which correctly stores an error and no response. And a citation parser that counted a brand's name as a source, because the brand is literally called Monday.com.

Every one of those was a first run. Every one produced a plausible, alarming, false number. So the habit I'd recommend over any specific check: before you believe your new verifier, make it fail on purpose, and read every row it flags. If you can't explain each flag, the tool is measuring something other than what you named it.

Three things to check today

In Search Console → Sitemaps, is Last read more recent than your most recent publish? One click. In three months of running this site I had never once looked.
Does your deploy pipeline's success signal come from the system you're actually measuring? Mine didn't.
When your verifier passes, do you know which failures it's structurally incapable of seeing? Write that in the output, not the README. Mine now prints it every run.

I rewrote the same detector three times in one day. Each fix introduced a new way of being wrong.

VisibilityAtlas — Thu, 06 Aug 2026 02:30:03 +0000

I wrote a post last week about an LLM fabricating a company's official website. Then I built a detector to find more cases at scale.

I have now rewritten that detector three times in one day. Each version fixed the previous version's bug and introduced a new way of mislabelling the same data. The counts went 1 → 134 → 41 → 12.

The interesting part isn't the final number. It's that all three bugs were the same bug wearing different clothes, and I had published an article about that exact bug in between versions two and three.

Version 1: blind to agreement

The detector's job: find domains an LLM calls a brand's "official website" when the brand doesn't own that domain.

// v1
if (engineA.saysOfficial(domain) && engineB.saysNoSuchSite(domain)) {
  flag(domain);
}

It found one real case, which I published. A commenter pointed out the hole:

The disagreement is confirmation rather than the trigger. Any URL an engine calls official can be checked against the brand's owned domain list on its own, single engine, no cross run needed. And you'd catch the case where every engine names the same wrong domain, which is the worse version.

He was right on all counts, and the third one is the serious one. A rule that fires on inconsistency is structurally blind to correlated error — which is the dangerous kind, because models trained on overlapping corpora fail together, and that's exactly when a wrong answer reaches every user at once.

My one published finding had survived only because one engine out of six happened to dissent. Drop that engine from the panel and v1 returns nothing.

Version 2: 134 findings, mostly wrong

The fix seemed obvious. I had ground truth sitting unused — every brand has a list of domains it owns. Check the claim directly:

// v2
for (const url of urlsIn(answer)) {
  const h = hostname(url);
  if (ownedDomains(brand).some(o => h === o || h.endsWith(`.${o}`))) continue;
  if (officialClaimNear(answer, url.index)) flag({ brand, domain: h, engine });
}

134 domains flagged, up from 1. I started drafting this post with a table of the top nine.

Then I looked at the rest of the output instead of the head of it. Domains like airtable-official.com, asana-vip.net, wrike-sales.net, basecampchina.net. Names that pattern-match to fabricated examples, not to real channels.

They were. Here's a representative flagged passage:

检查域名：官方主站是 wrike.com。任何其他域名（如 wrike-official.com、wrike-sales.net）都需要警惕。
(Check the domain: the official site is wrike.com. Be wary of any other domain, such as wrike-official.com, wrike-sales.net.)

The engine is warning users about fake sites. My detector saw a domain near the words "official website" and recorded it as a claim. I was scoring correct model behaviour as a hallucination.

Sampling the flagged set: notion-cn.com, which I had been about to publish as my most alarming new discovery, appeared in a warning context 86% of the time.

Assertion, warning, and denial are three states. I had two.

Version 3: still attributing to the wrong domain

So I added the third state, reusing the polarity logic I'd built three days earlier for brand mentions:

if (negationCues.test(tightClause(answer, i)))  return "denied";
if (warningCues.test(block(answer, i, 260)))    return "warned";
if (assertCues.test(clause(answer, i)))         return "asserted";
return "referenced";

41 flagged. Better. Still wrong, and wrong in a way I should have predicted, because I had written an article about it that week.

Look at that Wrike passage again. The phrase 官方主站 ("official main site") appears roughly thirty characters before wrike-sales.net. Any proximity-based rule attributes it to the nearest domain. But the cue belongs to wrike.com — the domain named before it. The sentence structure is: [real domain, asserted] then [fake domains, warned].

Proximity is not attribution. This is the identical failure I'd published about days earlier, in which a keyword window around a brand name picked up sentiment belonging to a neighbouring brand. Same bug, different entity type, and I walked straight into it.

Version 4: attribution, finally

The fix is to require that no other domain sits between the assertion cue and the domain being classified:

const clause = tightClauseAround(a, index);
const m = ASSERT_CUES.exec(clause);
if (!m) return "referenced";

// Attribution, not proximity: if another domain sits between the cue and this one,
// the cue belongs to that one.
const span = clause.slice(
  Math.min(m.index, hereIdx),
  Math.max(m.index + m[0].length, hereIdx)
);
const domainsBetween = (span.match(DOMAIN_RE) || []).length;
return domainsBetween > 1 ? "referenced" : "asserted";

Final counts over the same 4,023 stored answers:

12 domains asserted as official and not owned
173 domains appearing ONLY in warning contexts

That second number is the one worth sitting with. 173 cases where the engines were doing exactly the right thing — telling buyers to be careful about lookalike domains — and version 2 of my detector counted every one of them as a hallucinated official channel.

If I'd shipped v2, I'd have published a story about LLMs inventing 134 fake official websites. The true story is that they invented a handful and warned about many more, which is close to the opposite.

What I take from this

Three bugs, one shape. Each version failed because I treated nearby text as text about this thing. Windows around brand names, clauses around URLs, blocks around domains — same mistake at three granularities. Entity-level attribution is the actual problem, and no amount of tuning window size solves it. I now assume any new extractor I write has this bug until I've specifically checked.

Reading the head of the output is not reading the output. I drafted a version of this post based on the first fifty lines. The tail was where the story fell apart. Sorting by count and reading the top is how you confirm what you expect.

A detection rule needs a false-positive class, not just a true-positive class. v2 had no concept of "correct behaviour that resembles the thing I'm looking for." Once I named that class, it turned out to be 93% of what I'd flagged.

Three questions I'd now ask of any extractor before trusting its counts:

Does it distinguish the model asserting X, warning about X, and denying X? Those look nearly identical in text and mean opposite things.
When it finds a cue near an entity, what guarantees the cue is about that entity rather than the one next to it?
If the thing you're detecting has a legitimate lookalike — correct behaviour that pattern-matches to the failure — is that lookalike represented in your test cases?

I got all three wrong in a single day, on a codebase whose entire purpose is catching this category of error. The one thing that worked was a stranger asking whether my rule handled a case I hadn't considered, and then me reading the raw output instead of the summary.

Harness, scoring module, and labelled validation sets are public under CC BY 4.0: github.com/David88666/china-ai-visibility-benchmark

An LLM described a website in detail. The website doesn't exist.

VisibilityAtlas — Wed, 05 Aug 2026 03:13:21 +0000

An LLM described a company's official website to me in detail: complete Simplified Chinese interface, pricing displayed in RMB marked "tax not included", China-specific terms of service, a localized privacy policy. It told me this site was "the most important source" for verifying the company's credibility.

The domain has been registered to a private individual since 2016. It returns a 502. None of those pages have ever existed.

This is the hallucination shape that worries me most — not "the model made something up," which everyone expects, but the model made up specific, checkable, mundane details that a human would never think to check. Nobody verifies a privacy policy's existence. You verify the big claim and assume the supporting texture came from somewhere.

Here's how I caught it, why my own pipeline sat on it for a week, and the check that generalizes.

The setup

I run a measurement harness against six Chinese LLM APIs — DeepSeek, Doubao, Qwen, Kimi, ERNIE, GLM — asking buyer-style questions about international software brands and logging every answer. 4,023 valid responses, retrieval off, everything stored as JSONL.

One question type asks, in Chinese and English, some version of "what are this brand's official channels, and how would you verify them?"

GLM's answer for one brand:

Airtable China Official Website (Airtable中国官网)
URL: https://www.airtable.cn/
What to look for: This is the most important source. Its existence signals a formal commitment to the Chinese market.

And in a separate answer, in Chinese:

域名 .cn 是中国的国家顶级域名，由 Airtable 官方运营，这本身就是一种官方身份的声明
(The .cn domain is China's country-code TLD, operated officially by Airtable — this is itself a declaration of official identity.)

Confident, structured, and it reasons about why the evidence counts. That last part is what makes it dangerous.

Why my pipeline missed it

My extractor pulled URLs with a regex and recorded the domains. airtable.cn went into the citation column as a cited source, indistinguishable from a real one.

Every quality check I had was a rate: error count, empty-answer rate, answer-length distribution, language distribution. All of them were green, because nothing about this row was anomalous. One URL among 1,416, in a well-formed answer of normal length in the expected language.

Rates catch a class of rows that changes size between runs. They cannot catch a class that was wrong from the first run and stayed wrong at a stable size. A reviewer put it better than I did: rates catch a class that shrinks, asserts catch a class that was never right.

The check

Three lookups per domain. No tools, no API, about ten minutes for eight brands.

# 1. Does anything answer for it?
dig +short airtable.cn A
# → 223.26.56.104   (someone registered it and pointed it somewhere)

# 2. Who holds it?
whois airtable.cn | grep -iE "^(Registrant|Registration Time|Sponsoring)"
# → Registrant: (a private individual)
# → Registrant Contact Email: (a free QQ mail address)
# → Registration Time: 2016-02-08

# 3. What does it actually serve?
curl -s -o /dev/null -w "%{http_code}\n" http://www.airtable.cn/
# → 502

For contrast, the one brand in my sample that does own its .cn:

Registrant: BRIAN TYLER EVANS
Registrant Contact Email: help@clickup.com
Sponsoring Registrar: GoDaddy.com, LLC

That's what ownership looks like in a registration record: a company contact, at the company's own domain. It takes one line to tell the two cases apart, and my pipeline had never looked.

Running all eight brands from the study:

Domain	Held by	Serves
clickup.cn	the brand	nothing (parked)
airtable.cn	private individual	502
wrike.cn	private individual	502
asana.cn	private individual	"domain for sale"
smartsheet.cn	private individual, registered 2025	"域名转让 — The domain is on sale!"
notion.cn	a domain-holding company	403
monday.cn	the same domain-holding company	403
basecamp.cn	private individual, registered through 2034	a bicycle apparel manufacturer in Dongguan

Seven of eight belong to someone other than the brand. I'm not publishing registrant names — registering an available domain is legal and these are private individuals. The interesting part is on the other side.

The tell: the model contradicted itself

Same engine, same collection window, a differently-worded question:

You do not access a separate airtable.cn website. Instead, your Airtable China account is configured to use the China-hosted infrastructure.

Two incompatible accounts of the same fact, days apart, neither hedged.

That's the generalizable detection signal, and it's cheap: ask the same factual question several ways and diff the answers. A model that knows something answers consistently. A model that is constructing something plausible constructs differently each time, because there's no underlying fact constraining it.

In my open-question data, 18.8% of question-pairs changed outcome between two runs on the same day. If you're evaluating an LLM's factual output and you only ask once, you have no way to distinguish knowledge from confabulation.

What I changed

A predicate check on extracted URLs. A URL inside a clause that denies its existence is not a citation. My extractor was matching tokens without reading the sentence around them:

const URL_NEGATION_CUES =
  /没有|不存在|并无|未(设立|开设|推出|建立)|无(独立|专门|官方)|不提供|尚未|(?:does not|doesn't|no)\s+(?:have|exist|operate)/i;

export function urlIsNegated(text, index) {
  return URL_NEGATION_CUES.test(clauseAround(text, index));
}

One implementation note that cost me a wrong result: URL predicates need tighter clause boundaries than entity mentions do. Split on sentence punctuation only, and "并没有推出中文官网，其主要官网是 https://basecamp.com" flags that URL as negated — but the negation targets the Chinese site and the URL is being affirmed, one comma later. Splitting on commas as well fixed it: zero false flags across 1,416 URLs, and 74 genuine anti-citations in the bare-domain form my original regex never captured at all.

Assertions at the joins. Anywhere two vocabularies meet, assert a hit that must be there or refuse to run. In my harness there were three such seams and all three were quietly broken:

Competitor list vs answer text — my list was romanized, the answers name competitors in Chinese. The join found almost nothing and reported it as "no substitution." Fixed by refusing to start if a Chinese-language panel has no Chinese-script competitor names.
Category label vs prompt template — an English label injected into a Chinese question changed what was being asked. Fixed by round-tripping the rendered prompt through a model: "what category does this question ask about?" Compare to what you meant. One call per template.
URL vs surrounding clause — the predicate check above.

Provenance on every row, so a number can be reconstructed later rather than silently changing when the scorer improves: scoring_version, finish_reason, completion and reasoning token counts, a response hash, and a validity enum decided before any content scoring runs.

The part I'd want you to take away

The failure wasn't that a model hallucinated. It's that the hallucination was operationally indistinguishable from a fact at every layer of my pipeline, and every quality metric I had was green while it sat there.

If you're building anything that treats LLM output as evidence — extraction, enrichment, research automation, RAG evaluation — the questions worth asking are:

If a whole class of your rows were wrong from run one, which metric would move? (If the answer is "none," you're where I was.)
When your extractor pulls an entity, does anything check the predicate of the sentence it came from?
Do you ask the same factual question more than once, in more than one phrasing?
Can you reconstruct last month's number, or would re-running today silently produce a different one?

I've published corrections to my own numbers four times in three weeks doing this. Every single one was found either by reading raw output by hand or by a stranger asking a question I couldn't answer. Neither is a metric you can add to a dashboard, which I think is the actual lesson.

Harness, labelled validation samples, and the re-scoring scripts are public under CC BY 4.0: github.com/David88666/china-ai-visibility-benchmark

Six checks before you trust any number your LLM pipeline produces

VisibilityAtlas — Mon, 03 Aug 2026 04:00:25 +0000

Last week the same 96 recorded LLM conversations gave me three different headline numbers: 15%, then 66.7%, then ~31%. The pipeline reported zero errors every time. When I finally hand-read all the answers, the truth wasn't any of them.

I run a measurement harness against six Chinese AI engines (DeepSeek, Doubao, Qwen, Kimi, ERNIE, GLM) — ask buyer-style questions, log every answer, score which brands appear. I published the first number, had to retract it publicly, and rebuilt the measurement layer with the help of strangers in comment threads who kept predicting my next bug before I found it.

This is the post-mortem, generalized: six checks I now run before believing any number my own tooling produces, and the statistical trick that saved a classifier I was about to delete.

The failure shape: classes, not rows

The two worst bugs had an identical signature, and it's worth naming because you probably have a variant of it in your pipeline right now:

A whole class of rows gets mislabeled in one direction, silently, and no aggregate will ever show you.

Bug 1: reasoning tokens starve the answer. Reasoning models bill their chain of thought against the same max_tokens as the visible answer. Long reasoning → the API returns HTTP 200, finish_reason: "length", and content: "". My extractor searched the empty string for brand names, found none, and recorded "brand absent." 10 of 32 first answers and 46 of 96 follow-up turns were empty. Zero errors reported.

The fix is to make emptiness unrepresentable as a score:

if (!content.trim()) {
  const reasoning = choice.message?.reasoning_content?.length ?? 0;
  throw new Error(
    `empty-answer finish=${choice.finish_reason} reasoning_chars=${reasoning}`
  );
}

Better: return a validity enum (valid_answer | empty_due_to_budget | truncated | provider_error) and decide it before any scoring runs. An invalid row keeps its reason and carries no verdict. Missing output is a fact about the request, never evidence about the brand.

Bug 2: the scorer only speaks one language. My polarity cues — the phrases that separate "recommended" from "ruled out" — were Chinese-only. A commenter asked: what's the language distribution of your answers? Answer: 25.4% English (1,053 of 4,151). Every English mention was scoring "neutral" by construction. Same shape as bug 1: an entire class of rows quietly padding one label.

function answerLanguage(t) {
  const han = (t.match(/[\u4e00-\u9fff]/g) || []).length;
  const latin = (t.match(/[A-Za-z]/g) || []).length;
  const r = han / (han + latin || 1);
  return r > 0.35 ? "zh" : r < 0.05 ? "en" : "mixed";
}

If the language isn't one your scorer covers, return unscorable_language. Refusing to score is the honest move — it keeps those rows out of the numerator.

The six pre-flight checks

1. Error count vs row count. A full-looking file proves the harness ran, not that the run succeeded. When an API account ran out of credit mid-run, every call returned 403 — and the file still gained a row per attempt. 672 rows, 500 of them errors. Report the ratio next to every headline number.

2. Finish reason + token accounting. Store finish_reason, completion_tokens, reasoning_tokens per row. finish_reason: "length" with high reasoning tokens = starved answer, not short answer. This metadata was in every response the whole time. I just never looked.

3. Answer length distribution. Sort by length once per run. Healthy runs have a bell curve; a spike at zero is a fire alarm; a cluster of very short answers is usually truncation, refusals, or the model answering "see above" instead of answering.

4. Language distribution. Before anything downstream scores. See bug 2.

5. Hand-read 20 scored rows against their labels. This hour caught my worst bug, which no automation would ever flag. My harness scored a brand as "surviving" a budget constraint on this answer:

之前推荐的一些"顶配"品牌（如 Asana、Monday.com 的付费版）基本都超了，不建议强行上付费版
(The premium brands I recommended earlier — Asana, Monday.com's paid tiers — are over budget. I would not force it.)

Name present. Brand being ruled out. 47% of my "survivals" were rejections wearing a mention. Constraint-style prompts are exactly where models name things in order to exclude them, so name-matching fails hardest precisely where the metric matters most.

Bonus trap inside this trap: negated phrases contain their positive form as a substring. 不建议 ("not recommended") contains 建议 ("recommended"). My reject-cue and recommend-cue regexes both fired, cancelled out, and the mention scored neutral. Mask negations before testing positives.

6. A scoring version stamped on every row. My 15% / 66.7% / 31% came from the same data under three scoring rules — and nothing in the stored rows said which rule produced which number. Now every row carries scoring_version, a response hash, and the validity verdict. If you can't say which rules produced a number, the number isn't evidence; it's a screenshot of your pipeline's current mood.

The part I didn't know: you don't need a good classifier

After all the fixes, my polarity classifier was still bad — it caught only a third of genuine recommendations (sensitivity 33-45% depending on class). I was going to delete it.

A commenter reframed it: you're never claiming the judge is right — you're quantifying how wrong it is.

Hand-label a sample. Now you have a confusion matrix, which gives you sensitivity and specificity. With those, the rate your classifier reports over the whole corpus can be corrected — this is the Rogan–Gladen estimator, from 1978 epidemiology, built for exactly this problem: imperfect diagnostic tests that still need to produce usable prevalence numbers.

// observed = p·sens + (1-p)·(1-spec)  →  solve for p
const corrected = (observed - (1 - specificity)) / (sensitivity - (1 - specificity));

Real numbers from my labelled sample (n=36):

Class	Scorer reported	Corrected	Hand-labelled truth
recommended	5.6%	8.3%	8.3%
rejected	36.1%	80.6%	80.6%

A classifier with 45% sensitivity produced corrected rates that hit ground truth on both classes. I would have thrown it away the week before.

Two honest caveats. First, intervals: with n=36 and low sensitivity, my corrected 8.3% has a 95% interval of 0–50%. Usable internally, unpublishable alone. Second — and this is the part that changes the economics — the labelled sample sizes the error estimate, not the corpus. 200 labels can carry 20,000 rows, because you're estimating the scorer's error rate, which doesn't grow with the data. Label once, correct everywhere, re-validate when the scorer or the model under test changes.

So the real choice was never "human reading doesn't scale" vs "automation can't be trusted." It's: automation whose untrustworthiness has been measured on a fixed-cost human sample and adjusted for, interval published alongside. Either half alone is a guess.

Judge-design notes I took from people who run this bigger than me

Different model family for the judge than the model under test. Shared blind spots double-count.
Show the judge the clause containing the entity, not the full answer. I tested this at the regex level: a ±120-char window picked up sentiment belonging to neighbouring brands; clause-splitting fixed precision (25%→50%) but then missed rejections living in a heading two lines above the brand list. Wide leaks, narrow misses. Attribution is the actual task — which is why this ends at a validated judge, not a better window size.
Structured elicitation as a second arm, not a replacement. You can force the model to end with an explicit "still fits / ruled out" list — then polarity is parsing, not inference. But asking for a list changes the task, so run it alongside the natural-language arm. If the two arms disagree a lot, that disagreement is itself a finding about how much your format drives your numbers.

The re-validation triggers

Re-label a fresh sample when any of these change:

The scorer (rules, prompts, cue lists)
The model under test (version bumps count — DeepSeek deprecated a model name mid-study and the replacement behaved differently)
The language mix of the answers

Each costs an afternoon. Each of mine, skipped, cost a public retraction.

Everything here — harness, labelled validation sample with the misclassified examples, and the correction script — is public: github.com/David88666/china-ai-visibility-benchmark (CC BY 4.0). The buyer-facing version of this article, with the vendor questions, is on my site.

I used to think publishing corrections would cost credibility. It's been the single best source of methodology this project has had. The numbers got worse twice, the measurement got better four times, and I'll take that trade every week.

I asked six Chinese AI engines to recommend CRM software. HubSpot never appeared.

VisibilityAtlas — Wed, 29 Jul 2026 12:58:44 +0000

I spend my working life on how Chinese search and AI systems describe international brands, and I got tired of arguing about it from screenshots. So I built a harness and recorded 4,704 answers across six Chinese AI engines.

This post is about one slice of it: the CRM category, where the result was more extreme than I expected.

The setup

8 international CRM brands (Salesforce, HubSpot, Zoho CRM, Pipedrive, Microsoft Dynamics 365, Freshsales, SugarCRM, Monday CRM) × 42 Chinese-language buyer questions × 2 runs = 672 recorded answers.

The questions come in five groups: branded checks, open category discovery, comparison, decision intent, and risk. That grouping turned out to matter more than anything else in the study.

Every Chinese engine I tested speaks the OpenAI chat-completions dialect, so the collector is boring on purpose:

async function ask(prompt) {
  const body = {
    model: cfg.model,
    messages: [{ role: "user", content: prompt }],
    temperature: 0.3,
    max_tokens: 1200,
  };
  // Ark reasoning models bury the answer in reasoning_content unless you do this
  if (process.env.BENCH_THINKING === "disabled") body.thinking = { type: "disabled" };

  const res = await fetch(`${cfg.baseUrl}/chat/completions`, {
    method: "POST",
    headers: {
      "content-type": "application/json",
      authorization: `Bearer ${cfg.apiKey}`,
    },
    body: JSON.stringify(body),
    signal: AbortSignal.timeout(Number(process.env.BENCH_TIMEOUT_MS || 90000)),
  });
  if (!res.ok) throw new Error(`${res.status} ${(await res.text()).slice(0, 200)}`);
  const j = await res.json();
  return j.choices?.[0]?.message?.content || "";
}

One env-var swap per engine — base URL, model ID, platform label — and the same panel runs everywhere. Total API spend for the whole study was under $5.

The result

Mention rate on open category questions ("which CRM should I consider in China?", brand name never mentioned in the prompt):

Brand	Open-question mention
Salesforce	87.5%
Microsoft Dynamics 365	50.0%
Zoho CRM	12.5%
HubSpot	0.0%
Pipedrive	0.0%
Freshsales	0.0%
SugarCRM	0.0%
Monday CRM	0.0%

Five of eight global brands never surfaced. Not once, across every phrasing and both runs.

The engine's own framing, repeated almost verbatim across answers: "若企业有全球化需求，优先考虑 Salesforce 或 Microsoft Dynamics" — if you need global reach, consider Salesforce or Dynamics. The "international option" isn't a shelf with several slots. It's one seat, and Salesforce is in it. Everything else in the list goes to domestic vendors (销售易, 纷享销客, Kingdee).

The part that fooled me first

Ask the same engines about HubSpot by name and you get a competent, fair, well-structured paragraph. Branded-question mention rate was ~100% for every brand, on every engine.

So the check most teams run — "I asked DeepSeek about us and it looked fine" — returns a false positive by construction. The prompt hands the model the answer. The number that predicts whether a buyer ever meets you is the open one, and for this sample it was 23% across all categories.

Reading HubSpot's 84 branded answers, three things stood out:

No stable Chinese name. The model says so directly: "HubSpot 没有正式的中文名称." Every zero-mention brand in my data shared this marker.
A self-contradicting story. One answer claims HubSpot has China branches; three others reference an exit. The model holds both and serves them interchangeably. Unmanaged narrative → the model assembles one from conflicting fragments.
Compliance filtering. In this category, compliance caveats attached to 61 of 128 branded/risk answers — the densest I measured. Salesforce clears the filter because the model has one concrete fact to cite (its Alibaba Cloud arrangement). No checkable fact → generic warning.

Reproducibility notes, including what went wrong

Things I'd want to know if I were reading someone else's version of this:

Ask everything twice. 18.8% of open question-pairs flipped outcome between two same-day runs. A single query is a coin flipped once.
Reasoning models hide the answer. Doubao's flagship and several Qwen/GLM variants returned content: "" with 2,000 characters in reasoning_content until I forced answer-only mode.
Timeouts produce fake failures. One engine needed the timeout tripled; at 90s it looked like an error rate, at 200s it looked fine.
Count your errors before trusting a file. An aggregator ran out of credit mid-run and silently failed 500 of 672 calls with HTTP 403. The file still had 672 lines.
Date-stamp the exact model. Mid-study, DeepSeek deprecated deepseek-chat entirely and forced a rename. These products move faster than the analysis does.
Mode matters. This is API model-knowledge mode, not the consumer apps with live retrieval. Different instrument, different question — I record which per engine rather than blending them.

Data

Aggregate results and the reusable 42-question bilingual panel are CC BY 4.0:

https://github.com/David88666/china-ai-visibility-benchmark

Swap in your own brands and category — the panel has {brand} / {category} / {competitor} slots — and you can reproduce the whole thing in an afternoon.

If you work on something with international reach, the cheap version of this test is worth running once: three engines, five open category questions in the local language, twice each. It takes an hour, and it's a very different picture from asking the model about yourself by name.