I Added has_china_official_site to My Benchmark. It Failed Before the First Run

I was designing the next wave of an AI visibility benchmark when I added what looked like a harmless field:

has_china_official_site: boolean

It lasted about ten minutes.

The problem was not TypeScript. I could not explain what true meant without contradicting my own data.

The audit that created the field

The benchmark started with eight international B2B work-management brands.

On August 7, I reviewed the public registration records and live status of their exact-match .cn domains.

The records identified the benchmark brand as the registrant of one of the eight domains. The other seven were registered to other parties.

Across the eight domains:

• Three returned empty 502 responses.
• Four were parked or listed for sale.
• One served an unrelated operating business.
• None was an active brand-impersonation website.

The only .cn held by the benchmark brand was one of the domains returning 502.

That row broke the boolean.

If the brand controls the domain but the site does not work, does it have an official China site?

If a brand does not control its .cn but operates a Chinese section on its global domain, should the answer be false?

What about a verified WeChat account?

What about an authorized distributor?

What about a real Chinese website with a valid ICP filing that the international brand has never authorized?

My schema was about to convert all of those situations into the same value.

A plausible explanation arrived too early

I shared the domain result, and a commenter offered a structural explanation.

Perhaps software brands are worse at maintaining official China channels than consumer brands. Many B2B SaaS companies never establish a local operating presence, while consumer brands may already have retail, manufacturing, or distribution operations in China.

It is plausible.

My dataset cannot test it.

All eight brands are software companies. There is no consumer comparison group. I had not recorded operating-presence evidence before collection. The sample also cannot show whether a missing channel caused an AI attribution error.

The comment was useful because it exposed the next question. It did not answer it.

Before comparing industries, I needed to stop treating several different facts as one field.

First split: the brand and the channel are separate records

My first replacement looked like this:

type EvidenceStatus =
  | "verified_present"
  | "no_public_evidence_found"
  | "unresolved";

interface BrandSnapshot {
  brand_id: string;
  industry: "b2b_software" | "premium_consumer";
  mainland_operating_presence: EvidenceStatus;
  evidence_url?: string;
  checked_at: string;
}

interface ChannelSnapshot {
  channel_id: string;
  brand_id: string;

  type:
    | "cn_domain"
    | "global_domain_chinese_section"
    | "wechat_official_account"
    | "marketplace_store"
    | "authorized_partner"
    | "other";

  locator: string;

  control:
    | "brand_controlled"
    | "authorized_third_party"
    | "unrelated_third_party"
    | "unresolved";

  live_status:
    | "live"
    | "parked"
    | "empty_response"
    | "unreachable"
    | "unresolved";

  evidence_url?: string;
  checked_at: string;
}

This is more verbose than a boolean. It is also much closer to what I actually know.

A brand can control a domain that does not work:

{
  type: "cn_domain",
  locator: "example.cn",
  control: "brand_controlled",
  live_status: "empty_response"
}

Another domain can serve a real website without being an official channel for the benchmark brand:

{
  type: "cn_domain",
  locator: "example.cn",
  control: "unrelated_third_party",
  live_status: "live"
}

Both are valid observations. They do not support the same conclusion.

My first validator was still wrong

I then wrote a rule that counted a channel as official only when:

channel.control === "brand_controlled"

That failed too.

An authorized local partner may operate a legitimate official channel. A verified marketplace store may be run by a licensed operator. An official WeChat account may not map cleanly to a domain the brand owns.

The opposite mistake is just as easy.

A real company, a valid ICP filing, or a professional-looking website does not prove that an international brand authorized the channel.

So I separated control from authorization:

type AuthorizationStatus =
  | "brand_operated"
  | "verified_authorized_operator"
  | "not_authorized"
  | "not_verified";

interface AuthorizationEvidence {
  status: AuthorizationStatus;

  source_type:
    | "brand_official_source"
    | "operator_disclosure"
    | "public_registry"
    | "none";

  source_url?: string;
  checked_at: string;
}

The important value here is not_verified.

A missing public record is not proof that something does not exist.

If I convert “I did not find evidence” into false, the benchmark starts manufacturing certainty before the AI answer is even evaluated.

The AI claim needs its own record

The domain audit and the AI-output audit also need separate denominators.

Seven domains being registered to other parties does not mean seven AI answers falsely described those domains as official.

Those are different observations.

For each answer, I need something closer to this:

interface AIChannelClaim {
  observation_id: string;
  platform_surface_id: string;

  retrieval_status:
    | "on"
    | "off"
    | "unverified";

  brand_id: string;
  claimed_channel: string;
  claim_present: boolean;

  truth_status:
    | "supported"
    | "contradicted"
    | "unresolved"
    | "not_observable";

  error_type?:
    | "false_official_attribution"
    | "fabricated_channel_content"
    | "fabricated_operator"
    | "stale_channel"
    | "false_absence";

  evidence_url?: string;
  evidence_checked_at?: string;
  reviewer?: string;
}

An official-channel attribution error should count only when:

1. The answer affirmatively presents a channel as official.
2. The claim refers to an identifiable channel.
3. Dated evidence contradicts the attribution.
4. A reviewer confirms the mismatch.

A parked domain is not automatically an AI error.

A third-party domain is not automatically an AI error.

A valid ICP filing is not proof that a channel is official.

A strange-looking answer is not enough either.

If the necessary evidence cannot be obtained, the result remains unresolved.

The legal-looking fields also had to split

The original discussion collapsed several China-related concepts:

• Registering or holding a .cn domain
• Hosting a service in mainland China
• Completing an ICP filing
• Holding a commercial internet-information-service licence
• Having a registered operating entity
• Being authorized by the international brand
• Operating a channel that buyers can actually use

These are not interchangeable.

CNNIC's registration rules state that natural persons, legal persons, and unincorporated organizations may apply to register a national top-level domain.

A mainland operating entity is therefore not a universal requirement for simply holding a .cn.

China's internet-information-service rules separately distinguish non-commercial services, which use a filing system, from commercial services, which use a licensing system.

I do not want the benchmark to make legal determinations. I do want it to stop using one regulatory-looking field as evidence for another.

The data model now follows four rules:

.cn ownership != mainland hosting
ICP filing != brand authorization
brand ownership != channel usability
no public evidence found != verified absence

That looks obvious when written out.

It was not obvious in the first schema.

I needed fixtures before I needed more brands

My initial reaction was to expand the sample.

That would have produced more rows under an unstable definition.

Instead, I wrote the cases the measurement contract has to survive:

const fixtures = [
  "brand controls .cn, domain returns 502",
  "third party controls .cn, domain is parked",
  "third party controls .cn, unrelated business is live",
  "third party site has valid ICP filing, no brand authorization found",
  "brand uses a Chinese section on its global .com",
  "brand has an official WeChat account but no local web domain",
  "authorized distributor operates the local channel",
  "AI presents an affiliate site as the official brand channel",
  "public entity evidence cannot be resolved"
];

The audit should not pass until every fixture produces a defensible classification.

This is the same lesson I learned while building answer extractors. A clean test set often means the uncomfortable cases were never written down.

The next benchmark question is now testable

The proposed hypothesis is:

After matching for China relevance and brand familiarity, B2B software brands may have lower coverage of usable, verifiable China-facing channels than premium consumer brands.

That is not the result.

The next wave needs a separate expansion group to test it.

The rough design is:

1. Select comparable B2B software and premium consumer brands.
2. Record channel and operating-presence evidence before collecting AI answers.
3. Use the same declared platform surfaces, prompt set, locale, and retrieval conditions across groups.
4. Have reviewers verify affirmative official-channel claims against dated sources.
5. Report the raw numerator and denominator for every industry and evidence group.
6. Keep unresolved cases visible instead of converting them into failures.

The AI outcome will be an official-channel attribution error rate, not a general hallucination score.

The channel outcome also needs two versions:

• Usable official web channel: a verifiable, live web destination.
• Usable official China-facing channel: a broader set that may include verified social, marketplace, app, or authorized-partner channels.

I initially wanted one headline metric. I now think reporting both is safer.

The web-only measure is easier to reproduce. The broader channel measure represents the market more accurately. Combining them would hide the distinction I am trying to measure.

What would make the hypothesis fail?

I am writing the failure conditions before collecting the next wave.

The hypothesis is not supported if:

• Matched premium consumer brands do not show higher usable-channel coverage than B2B software brands.
• The difference disappears after accounting for brand familiarity or China market presence.
• Brands without a usable official channel do not show a higher verified attribution-error rate.
• The sample becomes too small or unbalanced to separate industry effects from operating-presence effects.

The final case should be reported as inconclusive, not quietly converted into support.

That may leave me with a less interesting article later. It will leave me with a more useful benchmark.

What changed in the implementation

The first version asked:

Does this brand have an official China site?

The current version asks:

Which channels can be identified?
Who controls each channel?
Who is authorized to operate it?
Is it currently usable?
What dated evidence supports those decisions?
What did the AI claim about it?
Can that claim be verified, contradicted, or not resolved?

This is a much less convenient record.

It is also a record I can defend six months later when the domain, operator, model, or channel has changed.

The original eight-brand review is documented in Who Owns Your .cn Domain? We Checked Eight Brands and Found One.

That audit is useful as hypothesis-generating evidence. It is not a software-versus-consumer comparison.

Three checks I am keeping

Before publishing any metric based on the new channel layer, I want three checks to pass.

1. Every “official” label has dated evidence

A domain suffix, logo, ICP number, or polished page is not enough by itself.

2. Unknown states remain unknown

no_public_evidence_found must never be transformed into verified_absent.

3. Derived metrics expose their denominator

Channel coverage is measured per brand.

AI attribution errors are measured over eligible, valid answer cells.

Domain counts are not interchangeable with answer counts.

If those denominators cannot be explained in one sentence, the number is not ready for a dashboard.

Sources

• CNNIC: National Top-Level Domain Registration Rules
• MIIT: Measures for the Administration of Internet Information Services
• MIIT: Measures for the Administration of Non-Commercial Internet Information Service Filings

I am still working through one design question.

If an official web channel is absent but a verified WeChat or authorized marketplace channel is usable, should the benchmark treat that as channel coverage, or should web and non-web coverage remain separate series?

If you have modelled this kind of channel evidence before, which edge case would you add to the fixture set?