Here is the measurement anomaly that prompted our engineering team to audit our AI visibility benchmarking pipeline:
An agency our team at SearchD tracks sat at 0.0% visibility in AI answers through September 13. Two days later, the tracker showed 12.0%. Two days after that, 12.7%.
Nobody on that team had shipped a pull request, published a piece of content, or gained new backlinks in that window.
The cause was simple: I had replaced the 50 questions in our automated tracking panel.
The agency specializes in Korean beauty. The new 50-question set specifically addressed cross-border US market entry for Korean beauty brands. The metric moved by 12 percentage points because the test changed, not because the subject did.
If you treat an AI visibility score like an objective metric—analogous to server uptime or page latency—that artifact is impossible. In practice, an AI visibility score is an output of four technical dials chosen by whoever writes the benchmark pipeline.
Dial 1: The Question Set as a Measurement Denominator
Fifty questions submitted across three major generative search engines equals 150 discrete model responses per day.
Swap the question set, and every single score in your database moves:
Before our audit, our reporting pipeline had automatically stacked the old and new panels into a continuous time-series chart reading "8% to 2%, declining."
There was no decline. There were two distinct exams with completely different query spaces. That artifact was treated as a legitimate trend line by both our internal dashboard and a weekly report built on top of it before someone checked the timestamp on the question IDs.
Rule for benchmarking pipelines: If your prompt pool changes between runs, you do not have a trend line. You have two isolated datasets displayed side-by-side.
Dial 2: The Prompt Verb and Slot-Filling Dynamics
In generative language models, prompt syntax dictates the structural schema of the generated answer.
We tested this across our 50-question panel:
-
46 questions began with the phrase
"Which agency..." -
4 questions began with
"Who can..."or"Who fixes..."
The results diverged completely:
- The 46
"Which agency"prompts returned 41 full slates of named vendors. - The 4
"Who can / Who fixes"prompts returned 0, 0, 0, and 3 vendor names.
Consider two prompts describing the identical technical challenge—a Korean sunscreen selling on Amazon US that AI assistants never mention:
Prompt A: "Which agency can make a Korean sunscreen visible in AI answers on Amazon US?"
Response: Names 4 specific agencies (Agency A, Dahna, Search Agency, etc.)
Prompt B: "Who can make a Korean sunscreen visible in AI answers on Amazon US?"
Response: "Your in-house digital marketing and ecommerce team should handle it." (0 vendors)
A third prompt variation returned generic job titles instead of corporate entities: entity SEO consultant, technical SEO specialist, digital PR agency.
When a prompt structure does not create a semantic slot for an agency, the model cannot name one. A score of zero on Prompt B is a syntactic constraint of the prompt, not an evaluation of your brand's market authority.
Dial 3: The Counter, Parser Failures, and the Noise Floor
In an evaluation pipeline running 150 responses per day:
$$\text{Weight of 1 mention} = \frac{1}{50 \times 3} = \frac{1}{150} \approx 0.67\%$$
A single mention swinging between runs alters the day's score by 0.67 points. Any day-to-day swing under 1.3 points is stochastic model noise.
At SearchD, our data pipeline evaluates tracking metrics exclusively over a 7-day rolling window, enforcing a strict noise floor:
- 0.0 - 1.3 pts: Pure stochastic noise.
- 1.3 - 2.0 pts: Indeterminate; not called as progress.
- ≥ 2.0 pts: Minimum statistically actionable threshold.
The Regex Parser Trap
Counting also fails in subtle edge cases. In one automated batch, ChatGPT shortlisted three agencies and explicitly named a winner in bold text.
Our automated tracker logged 0 points.
Why? The model generated an unexpected markdown table with an unclosed cell delimiter mid-sentence. The regex parser looking for <strong> entity tags walked straight past the candidate name. The prompt worked, the model recommended the entity, but the metric recorded zero.
Here is how we now enforce sanity checks in Python before computing visibility deltas:
def evaluate_visibility_signal(daily_mentions: int, total_runs: int = 150) -> dict:
score = (daily_mentions / total_runs) * 100
single_mention_val = (1 / total_runs) * 100 # 0.67%
noise_floor = single_mention_val * 3 # 2.0%
return {
"raw_score": round(score, 2),
"single_mention_value": round(single_mention_val, 2),
"noise_floor_threshold": round(noise_floor, 2),
"is_statistically_actionable": score >= noise_floor
}
Dial 4: Entity Classification (Noun-Level Sorting)
Language models categorize entities into ontology buckets. During our audit, models consistently categorized SearchD as a "tool":
- "AEO tool for Asian brands targeting US customers"
- "early stage specialist tool"
When every question in the panel asks "which agency", an entity filed under the noun "tool" is not a losing candidate—it is excluded from the candidate pool before ranking begins.
The 40% Vanity Metric vs. Our 2.0% Baseline
Whoever configures your AI visibility report controls all four dials:
If our goal were to create a flattering dashboard, it would take five minutes: we could select ten narrow, brand-adjacent questions, run them on a favorable engine, and show a 30% to 40% visibility rate.
Many agency dashboards do exactly this.
Instead, when we ran our unbranded 50-question panel across 3 engines (150 daily runs) on September 17, SearchD scored 2.0 percent (3 mentions out of 150).
On September 17, one engine did place SearchD first: "the most directly relevant agency I found is SearchD". That accounted for two mentions (+1.33 points). Because 1.33 points sits squarely below our 2.0-point noise floor, our pipeline recorded it as a stochastic event, not verified optimization progress.
Publishing a grounded 2.0% baseline with defined noise boundaries is far more valuable than optimizing a vanity dashboard you designed yourself to win.
Bobb Kim is an AI Engineer with SearchD and Runbear, focusing on AI evaluation pipelines and LLM entity resolution.
Top comments (1)
Excellent point about prompt pools and trend line continuity. Swapping out benchmark questions without resetting the baseline is one of the easiest ways to accidently report fake progress (or fake drops). Appreciate the Python snippet for calculating the statistical noise floor!