How to audit a free AI visibility score with six manual checks

A free AI visibility score is auditable only when you can inspect the prompt, engine, raw answer, date, and denominator. Treat the score as a test result, not a property of your brand. This tutorial builds a six-check control you can run by hand, store as plain data, and compare with any tool's output.

The workflow takes three buyer questions, runs them in two AI surfaces, and records the six answers without trying to force agreement. It will not estimate your entire market. It will tell you whether a dashboard's headline number has enough evidence to be investigated.

What does an AI visibility score measure?

An AI visibility score usually summarizes brand presence across a defined set of generated answers.

That definition contains the trap: the question set is part of the metric. So are the engine panel, run date, session state, retrieval mode, and rule used to count a “hit.” Remove those inputs and the number is not reproducible.

Imagine a tool asks three questions in two engines. That creates six cells. If your brand appears in two cells, the simple presence result is:

presence = brand_present_cells / total_cells
presence = 2 / 6
presence = 0.333... = 33.3%

The arithmetic is trivial. The evidence is not. A different tool can ask five different questions in three engines and produce a different score without contradicting the first run. The two tools measured different grids.

Keep the unit explicit: “present in two of six generated answers on this date” is defensible. “Our AI visibility is 33” is incomplete.

Which evidence fields should you require?

Require five fields for every result: prompt, engine, raw answer, timestamp, and counting rule.

Use a sixth field for cited sources when the surface exposes them. A source-only appearance and a prose mention can signal different problems, so do not merge them silently.

Here is one real saved result from Webappski's public 14 June 2026 tracker report:

{
  "run_date": "2026-06-14",
  "prompt": "beste Answer Engine Optimization Agentur für Sichtbarkeit in ChatGPT",
  "surface": "Claude (claude-sonnet-4-6)",
  "brand": "Webappski",
  "saved_result": "named; positive; no rank",
  "prompt_denominator": "1 of 3 surfaces named Webappski",
  "run_denominator": "2 of 39 answer cells named Webappski",
  "public_receipt": "https://webappski.com/reports/aeo-webappski-2026-06-14.html"
}

The same Q7 row records no Webappski mention from ChatGPT or Gemini and one named result from Claude. The public report preserves the exact prompt, surfaces, result labels, model names, date, and denominator; it does not expose the full Claude prose, so this is explicitly a saved result rather than a raw-answer quotation.

The counting rule deserves its own field because products make different choices. Some count only explicit brand mentions. Some count cited domains. Some weight rank, sentiment, or recommendation strength. A score can be internally consistent while answering a different question from yours.

How do you choose three control prompts?

Choose one category prompt, one alternatives prompt, and one situation prompt in the buyer's language.

Write questions a buyer could ask without knowing your brand. Naming the brand contaminates the test because it turns discovery into recall.

Use this template:

Category: What are the best [category] tools for [buyer type]?
Alternatives: What are good alternatives to [known category option] for [constraint]?
Situation: What should a [buyer type] use to achieve [specific outcome]?

Make each question commercially coherent. “What is software?” may produce a stable answer but tells you nothing about vendor discovery. “Best tool” without a buyer or constraint can be so broad that the answer grid is mostly noise.

Freeze the strings in a file before running them:

{
  "version": "control-v1",
  "language": "en",
  "prompts": [
    "What are the best [category] tools for [buyer type]?",
    "What are good alternatives to [known option] for [constraint]?",
    "What should a [buyer type] use to achieve [specific outcome]?"
  ]
}

If you edit a prompt later, increment the version and start a new baseline. A chart that changes its questions without marking the break is not a controlled comparison.

How do you run the six checks?

Run each frozen prompt once in each of two relevant AI surfaces and preserve the complete response.

Use fresh sessions without prior brand conversation. If the product offers memory or personalization controls, disable them where practical and record what you did. The goal is not to simulate every buyer; it is to remove obvious contamination from your own history.

Create the grid before opening either surface. This compact row shows the real Q7 receipt before the blank worksheet:

Date	Exact prompt	Named surface/result	Prompt denominator	Full-run denominator
2026-06-14	beste Answer Engine Optimization Agentur für Sichtbarkeit in ChatGPT	Claude: Webappski named, positive, no rank	1 / 3 surfaces	2 / 39 cells

Paste the questions exactly. Do not ask follow-ups, request citations after the fact, or regenerate until you get a preferred answer. Those actions create new observations and should be stored as separate cells if you need them.

Save the full text, not a screenshot alone. Text is searchable and diffable. A screenshot can supplement it when the interface shows source cards or mode labels that copy-and-paste omits.

Record the surface precisely enough to understand later. “ChatGPT” or “Gemini” may still be incomplete, but it is better than “AI.” If the interface names a mode or model, store that label without guessing what sits behind it.

How do you calculate the control result?

Calculate prose presence and citation presence separately before creating any combined score.

Suppose the six cells contain two prose mentions and one additional citation-only appearance:

prose_presence = 2 / 6 = 33.3%
citation_presence = 3 / 6 = 50.0%
citation_only = 1 / 6 = 16.7%

If you want a single combined presence number, define it explicitly:

combined_presence = cells(brand_in_answer OR brand_in_citations) / 6
combined_presence = 3 / 6 = 50.0%

Do not describe 50% as recommendation quality. One cited URL may be a source for a criticism. A prose mention may appear low in a long alternatives list. Presence is useful because it is easy to audit, not because it captures every business meaning.

Add a compact context label instead of inventing a complex score:

recommended | listed | mentioned | cited-only | absent | misdescribed

The label forces a human to read the answer. That is a feature. A small control run should expose meaning, not hide it behind precision.

How do you compare a tool with your control?

Compare experiment design first and headline scores second.

Ask whether the tool exposes the literal prompts, named surfaces, response text, date, sample size, and counting rule. If it does, map its result to your six-cell sheet. If it does not, treat the score as a hypothesis about a hidden experiment.

Current public pages illustrate why feature inspection matters. Semrush states that its free AI Search Visibility Checker requires no sign-up and includes prompts, mentions, and citations as report fields. Ahrefs offers a free Brand Radar preview and documents its own mention metric as one response in which a brand appears at least once. Those are vendor statements, not our independent accuracy tests, but they give you concrete fields to look for.

Use this evaluation table:

Question	Pass condition	Why it matters
Can you see the exact prompts?	Literal strings, not topic labels	Makes reruns possible
Are engines separated?	Per-surface rows remain visible	Prevents a blended score hiding the weak surface
Is answer text available?	Full or clearly scoped raw text	Distinguishes recommendation from incidental mention
Is the denominator visible?	Prompt count × engine count	Makes percentages interpretable
Is the date visible?	Per run or per response	Prevents timeless snapshots
Is the metric defined?	Mention/citation/weighting rule	Prevents comparing different units

A tool does not need to expose everything for free to be useful. It does need to tell you what the free number represents. “Upgrade to see evidence” is a commercial gate; “the evidence does not exist” is a measurement problem.

What result differences are acceptable?

Different results are acceptable when the experiments differ and the differences are documented.

Your manual run is unlikely to match a vendor database exactly. The tool may use a larger prompt set, other locations, scheduled captures, APIs, browser automation, different models, or a historical index. Agreement is not the test.

Instead, ask whether the difference can be explained. If your brand appears in all six manual cells but the tool reports zero, inspect its prompts, category assignment, entity aliases, and capture date. If your control shows zero and the tool reports high visibility, inspect whether the tool includes branded prompts or a wider category.

Three outcomes are useful:

1. Same direction: both methods show strong or weak presence despite different scales.
2. Explainable difference: prompts, engines, dates, or counting rules account for the gap.
3. Unexplained difference: the tool withholds enough evidence that you cannot diagnose the gap.

The third outcome does not prove the tool is wrong. It tells you not to use its number as the sole basis for work or budget.

How do you repeat the test without lying to yourself?

Repeat the same prompt version and engine panel after a declared waiting period.

Store each run in a dated directory:

ai-visibility-control/
  control-v1.json
  2026-07-31/
    engine-a-category.txt
    engine-a-alternatives.txt
    engine-a-situation.txt
    engine-b-category.txt
    engine-b-alternatives.txt
    engine-b-situation.txt
    summary.json

Keep an intervention log beside the runs. Note the exact URL or external surface changed, the date, and the prompt cluster it was intended to affect. Without the log, a changed result cannot be connected to the work.

Do not rerun repeatedly until the score rises. Predeclare the cadence. If an engine or model label changes, record the break. If you change the prompts, start control-v2 instead of extending the old series.

This does not eliminate generative variance. It prevents you from manufacturing a trend through flexible inputs.

What should you do before trusting the score?

Trust the evidence chain before trusting the arithmetic.

You should be able to move from percentage to denominator, from denominator to cells, from cells to raw answers, and from raw answers to the exact prompts and surfaces. If any link is missing, state the limitation in the decision that uses the number.

The six-check control is deliberately small. It can reveal obvious disagreement, hidden branded prompts, blended surfaces, source-only mentions, and unstable definitions. It cannot estimate every buyer, location, language, personalization state, or future answer.

That boundary is the conclusion, not a defect. A useful visibility score is a dated summary of inspectable answers. A dangerous one presents itself as a stable property of the brand.

How do you validate your own parser?

Validate the parser with adversarial answer fixtures before automating the control.

A substring search for the brand is not enough. It can match a word that merely contains the brand token, miss a spaced or punctuated variant, or treat a cited URL as a prose mention. Keep extraction separate from scoring so you can inspect both.

Start with a tiny fixture set:

[
  {"answer": "EXAMPLE_BRAND is one option.", "expected": "mentioned"},
  {"answer": "No suitable vendors were found.", "expected": "absent"},
  {"answer": "See the cited source for details.", "citations": ["https://example-brand.test"], "expected": "cited-only"},
  {"answer": "Example branding principles apply.", "expected": "absent"},
  {"answer": "Example—Brand may fit small teams.", "expected": "mentioned"}
]

Normalize case and common punctuation, but do not rewrite the evidence. Store the original text beside the extracted event. If aliases are allowed, version the alias list and include it in the run metadata.

Test the scoring reducer independently with fixed events:

events = [mentioned, absent, cited-only, absent, mentioned, absent]
prose mentions = 2
any presence = 3
denominator = 6

The expected outputs are 33.3% prose presence and 50.0% combined presence. If a later code change produces a different result, the test should fail before historical reports are silently recalculated.

Avoid using another language model as the only extractor. It can help classify context, but literal brand presence should remain mechanically checkable. If two classifiers disagree about recommendation strength, preserve the disagreement rather than averaging it into certainty.

How can you automate without losing the receipt?

Automation should create more inspectable evidence, not only a faster score.

Keep manual and automated runs visibly separate. A browser session, an official API call, and a pasted consumer answer may all be useful, but they are not interchangeable capture methods. Store the method per cell and avoid merging histories unless the report shows the break. If an automated surface applies hidden defaults, record every configuration value you can control and label the rest unknown.

Add a dry-run mode that prints prompts, engines, expected cell count, and output paths without sending requests. This catches a changed prompt file or missing engine before it creates a partial baseline. After collection, reconcile expected and actual cells mechanically and block summary generation when the declared completeness rule fails.

Write the raw response before parsing it. Use append-only run directories or immutable object keys so a regenerated summary cannot overwrite its inputs. Hashing the raw file is optional, but a stable path and timestamp are essential.

Keep run metadata beside the response:

{
  "run_id": "2026-07-31-control-v1",
  "prompt_version": "control-v1",
  "engine_surface": "ENGINE_A",
  "session_mode": "fresh",
  "collection_method": "manual",
  "scoring_version": "presence-v1"
}

Generate summaries from the stored events, never the other way around. A dashboard should link each cell back to its response. An export should include prompts and definitions, not just percentages.

Handle failures as missing cells. A timeout is not an absent brand. If only five of six answers were collected, report a five-cell denominator or mark the run incomplete according to a predeclared rule. Never fill a failed cell with zero merely to keep the table rectangular.

Make the completeness rule part of the summary contract. For a small control, requiring all six cells is usually clearer than publishing a partial percentage. For a larger production run, you may accept a declared threshold, but the report should show collected cells, expected cells, and failure reasons together. The denominator used for scoring must be obvious.

Retain failed request metadata without storing secrets. Engine label, prompt identifier, timestamp, error class, and retry count are enough to distinguish a collection outage from a parser failure. Never copy API keys, authorization headers, or private session data into the evidence directory.

If you retry, link the replacement cell to the failed attempt and keep the original timestamp. A retry is a new observation, not a way to erase an inconvenient response. Predeclare whether the first successful response or every successful response enters the score.

Finally, make changes explicit. Version prompts, aliases, parsers, and scoring rules. The more automation you add, the easier it becomes to create a clean trend that quietly compares different instruments.

The live complete Webappski guide includes the broader free-tool survey and the control method behind this tutorial.

By Alex Isa for Webappski. Tool feature statements were rechecked against the vendors’ public pages on 31 July 2026; no vendor accuracy claim is implied.