When you ask an LLM whether it recommends your product, you're running a test where you're both the subject and the observer, from inside the system you're measuring. Here's a clean-room protocol to test it properly.
Canonical URL: https://aeogeoai.net/blog-why-checking-your-own-ai-visibility-lies
You want to know whether ChatGPT recommends your product. So you open it and ask: "What's the best tool for X?" Your product shows up. You close the tab, reassured.
You just ran a broken experiment.
Not because the answer was wrong, but because of how you got it. You queried a system you're trying to measure, from inside that system, using an identity it already associates with the thing you're measuring. Every variable that should have been controlled was contaminated. If a colleague showed you that methodology in a code review, you'd reject it.
This is the Audience of One problem, and it catches almost everyone who tries to check their own AI visibility.
Why your own session is a contaminated test environment
LLM-backed search doesn't return one universal answer. Public documentation from the major providers indicates outputs can be shaped by signals about the request: location, conversation history, personalization settings, account state, and the model version in use. That's fine — until you try to use your own session as a measurement instrument.
When you check your own product, you introduce confounding variables you'd never accept in any other test:
- Observer identity: you're often signed in to an account that has searched for your own product before.
- Location bias: you're querying from the same network and region your business operates in.
- Session history: prior turns in the same conversation prime the model toward your brand.
- Selection effect: you phrase the query the way you think about your product, not the way a stranger would.
Each of these nudges the result toward you. The output feels objective — it's a fresh-looking answer on a clean-looking screen — but it's the single most flattering measurement you could possibly take. You are an audience of one, and the test was built to please you.
The tell: ask the model directly, "Would you recommend my product?" and it will often say yes enthusiastically. That's not evidence. A model can't reliably predict its own future retrievals, and after a conversation about your product it's biased toward agreeing with you. Willingness is not behavior.
The clean-room protocol
The fix is the same one you'd apply to any contaminated experiment: remove the confounds and reproduce. Here's the procedure.
1. Strip your identity
- Log out completely, or use a temporary/incognito session with personalization/memory disabled (private browsing alone doesn't remove account or location signals).
- Best case: run it on a device and network that has never searched for your product. A colleague's phone on cellular data is a surprisingly good clean room.
2. Query as a stranger would
- Ask the actual user's question — "best [category] tool for [use case]" — not your brand name.
- Do not mention your product until after the tool has answered. Naming it first re-contaminates the session.
- Phrase it the way a real user would, not the way your marketing does.
3. Read the source list, not the prose
- Use a tool that exposes its citations (Perplexity is good for this — it shows which domains it actually pulled).
- You're checking whether your domain appears in the retrieved sources, not whether the model says something nice. Retrieval is the ground truth; the prose is downstream of it.
4. Reproduce across sessions and engines
- Run it several times. Retrieval varies per session — one run proves almost nothing.
- Run it across ChatGPT, Claude, and Gemini. They retrieve differently; being cited by one is not being cited by all.
- Log what appeared each time. A stable pattern is signal; a single lucky hit is noise.
5. Interpret with tiers
Keep three buckets separate, because conflating them is how people fool themselves:
- Observed: what actually appeared in the retrieved sources across clean runs.
- Documented: what providers publicly say influences outputs.
- Inferred: your theory about why — held loosely, not stated as fact.
If your product shows up in the clean, logged-out, reproduced, third-party test — that's real. If it only appears when you check from your own chair, you've been measuring an audience of one.
Why this matters beyond vanity
If you're building anything that depends on being discoverable through AI answers — a dev tool, a SaaS, an API — your own perception of your visibility is the least reliable data point you have. The contaminated test doesn't just flatter you; it hides the gap between what you see and what a prospective user sees. And the prospective user is the only observer who matters.
We ran this on our own site. The signed-in check looked healthy. The clean-room test disagreed: for the query that mattered most we weren't retrieved at all, and where we did appear, the engine had surfaced the wrong page. None of that was visible from the comfortable version of the test. We only found it by running the uncomfortable one.
Build the clean room. Test from outside your own reflection. It's the only version of the result that isn't lying to you.
I research AI search visibility and citation behavior at AEOGeoAI. If you want to see the multi-engine version of this test run automatically, there's a free checker on the site — the manual protocol above costs nothing and teaches you more.
Top comments (0)