I was trying to turn a small luxury-jewelry GEO test into work a brand team could actually assign.
The dataset looked larger than it was.
I had eight brands, three neutral Chinese buyer questions, two retrieval-off API surfaces, and two answers per question and surface. That produced 12 independent valid answers.
Each answer was then evaluated against the same eight-brand registry. The resulting table contained 96 answer-brand cells.
That structure is useful. It is also easy to describe badly.
Twelve answers multiplied by eight brands creates 96 classifications. It does not create 96 independent model answers.
If I lead with n=96, the dashboard hides the actual sample.
The result that exposed the reporting problem
Piaget appeared in all four answers asking which brands had verifiable official China channels.
It appeared in none of the four answers recommending high-end brands for wedding jewelry.
Four opportunities per task are not enough for a brand ranking. They are enough to show that the two questions diagnose different things.
One asks:
Can the system identify a current, verifiable route for this brand?
The other asks:
Does the brand enter consideration for this occasion?
A brand can pass one and fail the other.
Why I stopped trying to produce one AI visibility score
Recommendation presence and route accuracy fail independently.
A brand can be recommended while the answer gives a vague, outdated, or unresolved route. That is a handoff problem.
A brand can have a clear official route and still be absent from a buyer shortlist. That is a consideration problem.
Combining them into one percentage makes the number easier to present and the work harder to assign.
I now use a four-state journey view:
- Recommended, route clear.
- Recommended, route unclear.
- Not recommended, route clear.
- Not recommended, route unclear.
The matrix is only a summary. Both denominators remain visible underneath it.
The truth registry should not live inside the answer table
Model answers and official facts change on different schedules.
An answer records what one declared surface said at one time. A truth registry records whether a route, operator, authorization, or service policy was verifiable on that date.
If I overwrite the truth fields inside an old benchmark row whenever a brand changes its website or store locator, I rewrite history.
The safer design is:
Keep the registry separate.
Give every route or service claim a stable claim ID.
Version the registry.
Connect each answer judgment to the registry snapshot used on the test date.
That separates three cases:
The answer was wrong on the day it was collected.
The answer was right then, but the official route later changed.
The public evidence was never clear enough to decide.
The last one stays unresolved. Missing public evidence is not proof that a route is fake or unauthorized.
Each failure now has an owner
Missing from the shortlist goes to brand, category, and editorial marketing.
Recommended but routed badly goes to digital operations and ecommerce.
Unclear authorization goes to legal and channel management.
Wrong repair, warranty, or resizing information goes to client service.
An invalid response scored as absence, an unreproducible percentage, or an unverified retrieval state goes back to research and analytics before anyone changes content.
That is a better deliverable than “AI visibility is 62.”
The four evidence views I keep separate
The project uses four different sources of evidence, and none can replace another.
Question provenance tells me which buyer uncertainty produced the prompt. Community discussions are useful here, but they are not a prevalence survey.
The answer benchmark tells me what fixed, declared surfaces returned. Retrieval-off API output does not describe every consumer product experience.
The truth registry checks routes, operators, authorization, and service claims. It does not measure shortlist consideration.
Search and analytics measure exposure and the next onsite action. They cannot prove that one community thread or AI answer caused a visit or sale.
They can meet in a briefing. They should not share a denominator.
What changes in the next wave
The next version needs more prompts around the decision itself:
Wedding style.
Budget.
Stone choice.
After-sales confidence.
Boutique-versus-daigou risk.
Repair, resizing, and documentation.
It also needs declared consumer or search surfaces. An API returning 200 after accepting a search flag is not proof that retrieval ran.
Every report will show:
Independent answer count.
Derived answer-brand cells.
Eligible denominator for each claim.
Invalid attempts by reason.
Truth-registry version.
Exact test date and surface.
I am not trying to make the small pre-wave look bigger. I am trying to make the next decision harder to misunderstand.
The practical lesson is simple.
“Improve AI visibility” is too broad to assign.
“Enter this shortlist,” “correct this route,” “clarify this authorization claim,” and “fix this service statement” are work.
Top comments (0)