Checking whether an AI assistant mentions your company once is interesting, but it is not a measurement system. Answers vary between models, sessions, dates, and prompt wording. A useful benchmark needs a stable set of questions, a consistent scoring method, and a record of what changed.
This article describes a lightweight approach that a small team can run without an enterprise monitoring platform.
1. Start with decisions, not keywords
Traditional rank tracking starts with search terms. AI-search monitoring should start with the decisions a potential customer asks an assistant to help make.
Build prompts from four groups:
- Category discovery: “What are the best ways to solve X?”
- Provider discovery: “Which companies help with X in Finland?”
- Comparison: “What is the difference between approach A and approach B?”
- Trust and validation: “How can I evaluate whether a provider is credible?”
Keep the set small enough to run consistently. Twenty carefully selected prompts are more useful than two hundred prompts that change every month.
2. Freeze a golden prompt set
A golden prompt set is a versioned list of prompts that stays stable between measurement rounds. Each row should include at least:
prompt_id, journey_stage, prompt_text, language, market, expected_entities
Do not silently rewrite prompts after a run. If a prompt needs to change, create a new version. Otherwise, an apparent visibility improvement may only be the result of easier wording.
3. Capture more than mentions
A binary “mentioned / not mentioned” score misses most of the useful information. For each response, record:
run_date
model
prompt_id
brand_mentioned
mention_position
recommendation_strength
linked_or_cited
cited_domains
answer_summary
Recommendation strength can use a simple scale:
- 0: not mentioned
- 1: mentioned incidentally
- 2: included as a relevant option
- 3: clearly recommended or used as a primary example
Keep the raw answer as evidence. Scores alone are difficult to audit later.
4. Separate visibility from citation
These are related but different outcomes:
- Visibility: the brand or entity appears in the answer.
- Citation: the assistant links to or names a source that supports the answer.
A company can be visible without its own website being cited. A website can also be cited while the brand is not recommended. Track both rates separately.
For a prompt set with N prompts:
mention_rate = prompts_with_brand_mention / N
citation_rate = prompts_citing_owned_domain / N
recommendation_rate = prompts_with_strength_2_or_3 / N
These simple metrics are easy to explain and hard to manipulate.
5. Control the test conditions
Perfect repeatability is not possible, but avoid unnecessary variation:
- Run the same prompts in the same order.
- Record the model and date.
- Use a fresh conversation for each prompt.
- Keep language and market context consistent.
- Avoid follow-up questions in the benchmark run.
- Run on a regular schedule, such as once a month.
The goal is not laboratory certainty. The goal is enough consistency to distinguish a durable trend from a lucky answer.
6. Review the evidence behind changes
When a score moves, inspect the responses before drawing conclusions.
Ask:
- Did the assistant discover a new page or third-party source?
- Did a competitor disappear rather than our brand improve?
- Did the answer cite stronger evidence than last month?
- Did the model misunderstand the category?
- Is the change visible across several prompts or only one?
This review turns monitoring into an action list. For example, repeated category confusion suggests clearer entity and service descriptions. Missing evidence may suggest publishing a primary source, case study, methodology, or data page.
7. Report trends, not isolated wins
A useful monthly report can fit on one page:
- Mention rate
- Citation rate
- Strong recommendation rate
- Most frequently cited domains
- Prompts with the largest positive and negative changes
- Three evidence-backed actions for the next month
Avoid presenting a single flattering answer as proof of authority. A stronger signal is repeated appearance across relevant prompts, models, and measurement rounds.
The practical takeaway
AI-search visibility is noisy, but it is measurable. The key is to freeze the questions, preserve the raw evidence, score consistently, and investigate why the answers changed.
I use this type of repeatable measurement thinking while developing AEOvara, a Finnish AI-search visibility and AEO project. The method itself is deliberately tool-independent: a spreadsheet and disciplined process are enough to begin.
Top comments (2)
The golden-set and raw-answer rules are solid. One portfolio-level metric I’d add is the zero-appearance rate, because the mean can hide how sparse the results are. In a 284-brand Korean DTC scan we ran across 50 AI shopping questions per brand, 65.5% of brands had zero appearances and the mean was only 0.648 out of 50. Reporting only the average mention rate would make that landscape look less discontinuous than it was. For repeated runs, would you also keep two or three identical executions per prompt and report the share of prompts that are stable across runs? That seems useful for separating a durable movement from model variance before the monthly comparison.
AI search benchmarks need repeatability more than drama. Same prompts, same surfaces, same capture rules, and a clear way to separate ranking drift from measurement noise.