DEV Community

Cover image for I asked six Chinese AI engines to recommend CRM software. HubSpot never appeared.
VisibilityAtlas
VisibilityAtlas

Posted on • Originally published at visibilityatlas.com

I asked six Chinese AI engines to recommend CRM software. HubSpot never appeared.

#ai #datascience #opensource #api

I spend my working life on how Chinese search and AI systems describe international brands, and I got tired of arguing about it from screenshots. So I built a harness and recorded 4,704 answers across six Chinese AI engines.

This post is about one slice of it: the CRM category, where the result was more extreme than I expected.

The setup

8 international CRM brands (Salesforce, HubSpot, Zoho CRM, Pipedrive, Microsoft Dynamics 365, Freshsales, SugarCRM, Monday CRM) × 42 Chinese-language buyer questions × 2 runs = 672 recorded answers.

The questions come in five groups: branded checks, open category discovery, comparison, decision intent, and risk. That grouping turned out to matter more than anything else in the study.

Every Chinese engine I tested speaks the OpenAI chat-completions dialect, so the collector is boring on purpose:

async function ask(prompt) {
  const body = {
    model: cfg.model,
    messages: [{ role: "user", content: prompt }],
    temperature: 0.3,
    max_tokens: 1200,
  };
  // Ark reasoning models bury the answer in reasoning_content unless you do this
  if (process.env.BENCH_THINKING === "disabled") body.thinking = { type: "disabled" };

  const res = await fetch(`${cfg.baseUrl}/chat/completions`, {
    method: "POST",
    headers: {
      "content-type": "application/json",
      authorization: `Bearer ${cfg.apiKey}`,
    },
    body: JSON.stringify(body),
    signal: AbortSignal.timeout(Number(process.env.BENCH_TIMEOUT_MS || 90000)),
  });
  if (!res.ok) throw new Error(`${res.status} ${(await res.text()).slice(0, 200)}`);
  const j = await res.json();
  return j.choices?.[0]?.message?.content || "";
}
Enter fullscreen mode Exit fullscreen mode

One env-var swap per engine — base URL, model ID, platform label — and the same panel runs everywhere. Total API spend for the whole study was under $5.

The result

Mention rate on open category questions ("which CRM should I consider in China?", brand name never mentioned in the prompt):

Brand Open-question mention
Salesforce 87.5%
Microsoft Dynamics 365 50.0%
Zoho CRM 12.5%
HubSpot 0.0%
Pipedrive 0.0%
Freshsales 0.0%
SugarCRM 0.0%
Monday CRM 0.0%

Five of eight global brands never surfaced. Not once, across every phrasing and both runs.

The engine's own framing, repeated almost verbatim across answers: "若企业有全球化需求，优先考虑 Salesforce 或 Microsoft Dynamics" — if you need global reach, consider Salesforce or Dynamics. The "international option" isn't a shelf with several slots. It's one seat, and Salesforce is in it. Everything else in the list goes to domestic vendors (销售易, 纷享销客, Kingdee).

The part that fooled me first

Ask the same engines about HubSpot by name and you get a competent, fair, well-structured paragraph. Branded-question mention rate was ~100% for every brand, on every engine.

So the check most teams run — "I asked DeepSeek about us and it looked fine" — returns a false positive by construction. The prompt hands the model the answer. The number that predicts whether a buyer ever meets you is the open one, and for this sample it was 23% across all categories.

Reading HubSpot's 84 branded answers, three things stood out:

  • No stable Chinese name. The model says so directly: "HubSpot 没有正式的中文名称." Every zero-mention brand in my data shared this marker.
  • A self-contradicting story. One answer claims HubSpot has China branches; three others reference an exit. The model holds both and serves them interchangeably. Unmanaged narrative → the model assembles one from conflicting fragments.
  • Compliance filtering. In this category, compliance caveats attached to 61 of 128 branded/risk answers — the densest I measured. Salesforce clears the filter because the model has one concrete fact to cite (its Alibaba Cloud arrangement). No checkable fact → generic warning.

Reproducibility notes, including what went wrong

Things I'd want to know if I were reading someone else's version of this:

  • Ask everything twice. 18.8% of open question-pairs flipped outcome between two same-day runs. A single query is a coin flipped once.
  • Reasoning models hide the answer. Doubao's flagship and several Qwen/GLM variants returned content: "" with 2,000 characters in reasoning_content until I forced answer-only mode.
  • Timeouts produce fake failures. One engine needed the timeout tripled; at 90s it looked like an error rate, at 200s it looked fine.
  • Count your errors before trusting a file. An aggregator ran out of credit mid-run and silently failed 500 of 672 calls with HTTP 403. The file still had 672 lines.
  • Date-stamp the exact model. Mid-study, DeepSeek deprecated deepseek-chat entirely and forced a rename. These products move faster than the analysis does.
  • Mode matters. This is API model-knowledge mode, not the consumer apps with live retrieval. Different instrument, different question — I record which per engine rather than blending them.

Data

Aggregate results and the reusable 42-question bilingual panel are CC BY 4.0:

https://github.com/David88666/china-ai-visibility-benchmark

Swap in your own brands and category — the panel has {brand} / {category} / {competitor} slots — and you can reproduce the whole thing in an afternoon.

If you work on something with international reach, the cheap version of this test is worth running once: three engines, five open category questions in the local language, twice each. It takes an hour, and it's a very different picture from asking the model about yourself by name.

Top comments (0)