Your dashboard says a brand appeared in 25% of AI answers. Before interpreting that number, ask a less exciting question: what counted as an answer?
If a test failed, timed out or never reached the intended question, recording a zero turns a collection problem into an apparent visibility problem.
Here is a practical way to keep those outcomes separate.
1. Separate absence from missing information
A completed answer that does not mention the brand is a valid non-mention.
An interrupted session with no assessable answer is missing data.
Store them differently:
-
false: the outcome was assessed and absent. -
true: the outcome was assessed and present. -
null: the outcome could not be assessed.
This distinction should apply separately to brand mentions, owned-domain citations and explicit recommendations. An answer might be scorable for mentions while its citations remain inaccessible.
2. Define completeness before seeing the result
Write down what makes an observation usable before collecting answers.
For example: the system answered the intended question, and the relevant output was preserved for review.
Record refusals separately. Whether they belong in a particular denominator depends on what the metric is intended to measure. Preserve them and disclose the rule.
Do not decide that an answer is “incomplete” simply because the brand is missing.
3. Report coverage beside performance
Consider this hypothetical batch—not an actual client result:
- Eight attempts were planned.
- Six produced complete, scorable answers.
- Two of those six mentioned the brand.
- Two attempts failed.
The mention rate among complete answers is 2/6, or 33.3%.
Collection coverage is 6/8, or 75%.
Reporting only 2/8 treats failures as known non-mentions. Reporting only 33.3% hides the missing observations.
Show both. Also explain if failures cluster around particular questions, because the available answers may not represent the planned test set.
4. Preserve reruns
A rerun should create another observation, not erase the first attempt.
Set a rule in advance: for example, retry a technical failure once, but do not rerun a valid answer simply because it omitted the brand.
Selecting the most flattering answer from several attempts measures something different from following a fixed test procedure.
5. Test the scoring rules
Check these edge cases before trusting your dashboard:
- A third-party source mentions the brand, but the brand’s website is not cited.
- The brand appears in a negative comparison.
- A similarly named business appears.
- A domain is printed as plain text without being used as a supporting source.
- An interrupted run produces no assessable answer.
A mention, citation and recommendation should not automatically receive the same label.
Make the score auditable
Preserve the raw response, prompt version, timestamp, visible model label, interface and retrieval settings when observable. Record “unknown” instead of guessing.
The useful output is not just a percentage. It is a percentage another reviewer can reconstruct.
I work on AEOvara, a Finnish AEO and AI-search visibility project. This proposed checklist extends my earlier repeatable benchmark guide with data-quality checks. It does not establish that any content change causes more AI recommendations.
What edge case would you add to the scoring tests?
Top comments (0)