DEV Community: Bobb Kim

Keep ChatGPT UI observations out of your API denominator

Bobb Kim — Thu, 17 Sep 2026 08:01:44 +0000

A buyer-facing question produced a useful puzzle, not a benchmark result:

what is the best aeo/geo agency for korean beauty products?

On 14 September 2026, we recorded one answer in the consumer ChatGPT UI. On 16 September, we ran the same question five independent times through the Responses API with web_search, gpt-5.5, tool_choice: required, store: false, no prior-response reference, and no brand supplied. The user location was not fixed.

The important implementation detail is simple: these are two collection surfaces. They must remain in two denominators. A UI answer is not run 1 of an API batch, so this is not a “six-run success rate.”

That distinction preserves the interesting observation instead of overstating it. The UI named five agencies. In the later API batch, only Agency B from those original five was named in all five runs; the other four were not named in that batch. A newly appearing agency was also named 5/5, so Agency B was not the only API repeat. SearchD was absent from both the initial five and the five API answers.

The recorded comparison

In SearchD’s review, Domain Rating (DR) was looked up on 16 September. It is a proxy metric, not a complete measure of authority. The table is a source-case record, not a ranking model.

Agency	Position in the 14 Sep UI answer	DR on 16 Sep	Named in five independent API runs
Agency A	1	7	0/5
Agency B	2	0.7	5/5
Agency C	3	47	0/5
Agency D	4	0.8	0/5
Agency E	5	76	0/5
SearchD	not listed	0.1	0/5

The displayed-citation record adds a separate fact: Agency B’s domain appeared among citations in 5/5 API runs. The available record says its service page was the most commonly cited URL. It does not establish that one exact URL was cited in all five runs.

What can we say? A low-DR agency persisted across this particular five-run API batch while higher-DR initial UI names did not. What cannot we say? That DR causes, prevents, or predicts AI recommendations; that the API reproduces the consumer UI; or that a page change will create a recommendation.

First, investigate the usual explanations

In SearchD’s review, the source case checked four plausible explanations before inspecting page content:

Backlinks: the recorded lookup reported zero dofollow referring domains.
US rankings: the recorded lookup reported zero US organic keywords.
Industry roundups: the agency was absent from three checked category roundups.
Track record: the source reported a January 2026 founding date.

Those are limited checks, not proof that every authority signal or explanation has been eliminated. They are still valuable because they explain why the next step was page inspection rather than another generic SEO claim.

What the inspection found

SearchD’s inspection offers three candidate explanations for why a site might be easy for a model or reviewer to describe. They are observations about information structure, not retrieval traces.

Accessible equivalent: compare a file that is mostly a Markdown link list with a file that puts company facts in labeled, line-level fields. The right-hand structure makes legal name, founder roles, founding date, location, languages, contact, and services directly extractable.

Source caption: Original editorial illustration by Liam Hwang for the SearchD team case, 17 September 2026. It compares information structures only; it does not demonstrate model retrieval or an llms.txt effect.

1. An entity-readable file, not merely a link list

In SearchD’s page review, the inspected file contrasted a sitemap-like llms.txt with an entity-oriented version. The latter surfaced legal name, both founders and roles, founding date, location, working languages, contact address, and services as explicit attributes. The source reported a 3,272-word summary and a 16,291-word full version, with an update date inside the file.

Those lengths are descriptive facts, not an ideal word-count target, and publishing llms.txt is not proof that a model consumed it. The practical lesson is narrower: when a reviewer needs to answer “who is this company?” a labeled fact is less ambiguous than a marketing paragraph or a list of links.

2. Identity questions and regulated-service limits were answered directly

SearchD’s page review found that the category page reportedly used seven FAQ topics. Two are especially useful for avoiding inference: whether the company is a Korean marketing agency, and whether it handles FDA, customs, tax, or legal work. A separate source observation stated that an email/SMS service was paused.

Accessible equivalent: seven FAQ topics are grouped into sales questions and two classification questions. One identifies what the company is; another states what regulated work it does not do. A separate notice identifies a paused service.

Source caption: Original source-case transcription of the FAQ topics and service boundaries. This is not a product screenshot and does not measure a recommendation effect.

FAQ markup itself is not a demonstrated cause. The underlying visible answers are useful because they let a reader distinguish identity, scope, and exclusions without inventing missing facts.

3. The absence of named case studies was structured, not hidden

SearchD’s review found that the site reportedly separated four claim types: founder experience in prior roles, deliberately withheld former-client identities, illustrative deliverables, and measured results. The case did not disclose named client case studies.

This is not evidence of capability or outcomes. It is an example of a site making the limits of its evidence legible. For an assistant, reviewer, or procurement team, “illustrative deliverable” is a safer fact than an unlabeled example presented like proof.

The counterintuitive detail

The inspected site used “AEO” zero times. The model’s reported descriptions, such as entity consistency, citation-friendly content, and off-site corroboration, were not present as phrases either. Meanwhile, SearchD uses that vocabulary on its own site but was not named in these observations.

That makes “answerability beat vocabulary” a reasonable case hypothesis, not a keyword rule. The pages inspected are not the entire web, and nothing here establishes keyword determinism.

Turn the case into a content-quality run

For a Korean beauty brand selling in the US, SearchD’s team translates the abstract pattern into buyer questions:

Accessible equivalent: four checks ask whether a brand states its Korean origin, gives precise market and product facts, names exclusions, and exposes visible HTML that agrees with structured data.

Source caption: Original SearchD four-check framework from the source case. These are information-quality checks, not a guarantee of citations or recommendations.

Origin: Does body text say whether the brand is Korean? Brand origin is not the same as manufacturing location.
Market version: Can a shopper determine the US formula or packaging version, shipping and authorized-retailer status, and product-specific availability?
Negative boundaries: Does the page explicitly say what it does not sell, ship, claim, or provide? Do not imply FDA, customs, tax, or legal services where none exist.
Consistency: Does visible HTML say the same thing as the relevant structured data?

Here is an illustrative record schema. It is deliberately a planning artifact, not a client result:

{
  "entity": "Example K-beauty Brand",
  "fact": "The US retail formula for Product X is fragrance-free.",
  "source_url": "https://example.com/us/product-x",
  "source_owner": "Regulatory and product teams",
  "visible_html_verified": true,
  "structured_data_verified": true,
  "publish_change": "Add the market-version statement to the US product page.",
  "retest": "Repeat the same separated-surface collection after publication."
}

A practical run has five fields: owner, fact, source, publish, retest. The owner validates the fact. The source points to evidence. The page receives an explicit statement. The change is published through normal review. Then we rerun the protocol without pooling UI and API observations.

Minimal checks before interpreting a batch

type Surface = "consumer_ui" | "responses_api";

type Observation = {
  surface: Surface;
  observed_at: string;
  prompt: string;
  run_id: string;
  named_entities: string[];
  citation_domains: string[];
  citation_urls: string[];
  notes: string[];
};

function assertComparable(batch: Observation[]) {
  if (new Set(batch.map(x => x.surface)).size !== 1) {
    throw new Error("Aggregate one surface at a time");
  }
  if (new Set(batch.map(x => x.prompt)).size !== 1) {
    throw new Error("Keep the prompt fixed within the batch");
  }
}

SearchD’s measurement notes keep raw responses privately if they contain sensitive data. Publicly report a compact ledger: prompt, date, surface, model and tools where known, run count, entity recurrence, citation-domain recurrence, exact-URL uncertainty, and unfixed variables such as location.

The next measure is not “did the site win?” It is whether the audited facts are accurate, visible, structurally consistent, and answerable under repeatable collection conditions. This case observed no uplift, traffic, sales, or causal effect. It gives us a disciplined way to document the next test.

Source and disclosure

Based on Liam Hwang’s original SearchD case, published 17 September 2026. The illustrations are Liam’s original public editorial figures; text and examples are adapted for this channel.

Bobb Kim is an AI Engineer who works with SearchD. Prepared with AI assistance. This is our team’s research, not an independent agency endorsement or client-results report.

Debugging US K-beauty AI Answers: Missing Mentions vs. Wrong Product Facts

Bobb Kim — Wed, 16 Sep 2026 04:04:33 +0000

Three months of English content and a flat answer engine optimization (AEO) score do not identify a defect. Neither does strong Korean Google visibility. A US assistant might omit the product, confuse its identity, or name it while repeating the Korean formula. Those need different work queues.

Start with observable answers and displayed sources, not a presumed ranking or complete retrieval trace. Liam Hwang’s original SearchD source-map article motivates comparing source sets across assistants. That qualitative observation is not evidence about these K-beauty scenarios or their outcomes.

Freeze the questions, separate the defects

For SearchD’s source-gap diagnosis, keep exact question wording, assistant, date, requested market, language, and session conditions. “Which Korean moisturizer should I buy in the US?” tests discovery. “Which formula is sold in the US?” tests accuracy. Do not merge their results into one score. Keep the panel composition fixed: changing the question mix can move a headline score even when every underlying answer stays the same. Record exclusions by engine, too.

For intermittent mentions, repeat unchanged questions in comparable clean sessions before attributing a change to an edit. Record the interface and model version when exposed; requested US context is not proof of the assistant’s actual localization.

The examples below are illustrative, derived from buyer questions, not customer records, measured results, or an official platform schema.

Store capture state beside the approved product record

This illustrative record makes SearchD’s distinction between missing evidence and an absent mention explicit; it is not a production schema.

{
  "question_id": "us-formula-01",
  "question": "Which version of Example Brand Daily Cream is sold in the US?",
  "assistant": "assistant-name",
  "observed_at": "2026-09-15T00:00:00Z",
  "requested_market": "US",
  "language": "en-US",
  "capture_status": "failed",
  "failure_reason": "timeout",
  "answer_text": null,
  "named_entities": null,
  "displayed_sources": null,
  "sources_capture_status": "not_collected",
  "product_record": {
    "brand_id": "example-korean-brand",
    "brand_origin": "KR",
    "official_domain": "example.com",
    "product_id": "daily-cream-us",
    "market": "US",
    "formula_revision": null,
    "effective_from": null,
    "approved_claims": null,
    "authorized_retailers": null,
    "approval_status": "pending",
    "approved_by": null,
    "approved_at": null
  }
}

The product record becomes the comparison reference only after approval. Replace placeholders with versioned records and evidence links; never infer US facts from the Korean page. The client’s compliance owner approves formula, SPF, certification, and regulatory wording. Pending fields cannot establish that an answer is wrong.

Use ok only for a complete answer capture. Store failed, blocked, or partial separately, with reasons and retained fragments. A timeout is not an absent mention. An empty source list means no sources were displayed only when collection was complete; otherwise use null and a collection status.

Compare pages before domains

For a SearchD source comparison, preserve full displayed URLs and the answer sentences attached to them. Host overlap is a secondary diagnostic, not a measure of factual agreement:

from urllib.parse import urlparse


def hosts(record):
    if (record["capture_status"] != "ok"
            or record["sources_capture_status"] != "complete"):
        return None
    urls = record.get("displayed_sources")
    if urls is None:
        return None
    return {
        urlparse(url).hostname.lower()
        for url in urls
        if urlparse(url).hostname
    }


def jaccard(a, b):
    if a is None or b is None:
        return None
    union = a | b
    return len(a & b) / len(union) if union else None


def named_rate(records, target_id):
    eligible = [r for r in records
                if r["capture_status"] == "ok"
                and r.get("named_entities") is not None]
    if not eligible:
        return None
    return sum(target_id in r["named_entities"]
               for r in eligible) / len(eligible)

Here, named_entities contains reviewed, normalized IDs, not raw string matches. A namesake US company must not count as the Korean brand. Two empty source sets return no overlap score, rather than suggesting perfect evidence agreement. Report eligible and excluded capture counts alongside any rate.

Route observations to the right owner

SearchD can ship approved changes on brand-owned pages; external corrections remain requests to the page owner. The diagnostic cases below keep those responsibilities separate.

Absent product, plenty of English content. First inspect US page accessibility: response status, crawl directives, indexing eligibility, canonical targets, internal links, and whether product facts exist in readable HTML. Compare the US page with the Korean page, without assuming Korean search rankings transfer. A failed access check is actionable; passing it does not guarantee citation.

Accurate translation, ambiguous identity. Direct translation can omit market context. Give each answer passage an explicit brand, product, US variant, and effective date. Align the official domain, Korean origin, and product identifiers across owned pages. Distinguish the unrelated US namesake in factual copy where needed; do not count its mentions as wins.

Named product, wrong formula. Compare the exact answer claim against the approved US record and inspect the displayed page. A Korean-market source suggests a market-selection mismatch; an outdated US retailer listing suggests stale facts. Without displayed support, mark the source of the error unknown, rather than inventing a retrieval explanation.

Old retailer facts. Save the listing URL, dated excerpt, conflicting approved record, proposed correction, and page owner. Correct brand-owned pages directly after approval. Send retailer or distributor owners a documented correction request, then verify the public listing. Request submission is not acceptance, and acceptance is not proof that assistants changed.

Illustrative engineering workflow, not a customer-result chart. JSON and Python remain editable text above.

Keep measurement and delivery separate

To evaluate SearchD’s work, track named entities, displayed citations, factual accuracy, and referrals as separate events. A displayed citation is neither a visit nor the assistant’s complete retrieval history. Maintain a shipping log and an observation log; compare repeated answers per assistant without treating timing as causation.

Disclosed SearchD scope

I work with SearchD, a San Francisco-based AEO agency for Korean consumer brands selling in the US. Its published method covers fixed buyer-question measurement, approved English buyer-question pages, and technical fixes shipped on brand-owned properties. Retailers control their listings; editors control independent coverage. It can document correction requests, not guarantee acceptance or AI inclusion. Its public service method specifies 40 questions across five engines. These scenarios demonstrate diagnosis, not verified client outcomes.