A 200 Response Does Not Prove Your LLM Used Web Search
A retrieval-on versus retrieval-off test sounds like one of the cleanest experiments available in AI visibility.
Take the same buyer question.
Run it once with web retrieval enabled.
Run it again with retrieval disabled.
If a brand appears only when retrieval is on, live search probably helped. If it survives when retrieval is off, the brand may already be familiar to the model.
That was the experiment I intended to build.
It failed before collection began.
That failure was more useful than another dashboard.
The pre-flight that stopped the study
I tested the same neutral prompt on two OpenAI-compatible API surfaces.
I tried more than one search-looking request format.
A tool-style search payload returned an explicit error on one surface. That was inconvenient, but safe. The system told me that the requested contract was unsupported.
The more dangerous result came from simpler parameters.
Fields such as enable_search or web_search were accepted. The APIs returned HTTP 200 and produced normal-looking answers.
There was no executed search query in the response.
There was no tool-call event.
There was no retrieval record.
There was no provider metadata establishing that a search had taken place.
The request asked for retrieval. The response did not prove retrieval occurred.
Had I launched the full collection, I could have saved those rows as retrieval-on observations. The final comparison might have looked methodical, balanced and completely wrong.
The mistake would not have produced an error rate.
Every request could have been successful.
Every answer could have been non-empty.
Every row could have passed extraction.
The experiment label itself would have been false.
Requested state and observed state are different fields
The failure exposed a distinction that AI visibility pipelines need to make explicit.
Requested retrieval state records what the collector asked the provider to do.
Observed retrieval state records what the collector can verify actually happened.
Those are not the same field.
A useful row should therefore be able to say:
Retrieval requested: yes
Retrieval observed: unproven
That is more honest than forcing the row into either the retrieval-on or retrieval-off group.
The same principle applies to other model features.
Requesting JSON does not prove that the returned object follows the schema.
Requesting a specific language does not prove that the answer stayed in that language.
Requesting citations does not prove that the displayed URLs support the claims beside them.
Requesting a tool does not prove that the tool executed.
Configuration is an intention. Evidence describes the event.
Why HTTP 200 is such a weak signal
HTTP 200 answers a narrow transport question: did the server successfully process the request well enough to return a response?
It does not establish that every optional field was understood.
Some APIs reject unsupported parameters. That is the safest behavior because the failure is visible.
Others may accept, ignore or partially interpret fields they do not implement. Compatibility layers are especially risky because they may reproduce the shape of another provider’s API without reproducing every behavior behind it.
A pipeline that checks only the status code can therefore report:
Request succeeded.
Answer received.
Retrieval on.
Only the first two statements are supported.
The third requires separate evidence.
What should count as retrieval evidence?
There is no universal field because provider implementations differ. But evidence can still be graded.
The strongest observable evidence may include:
An explicit search or browsing tool-call event.
The executed search query or queries.
Provider-returned retrieval metadata.
A structured list of fetched source records.
A documented retrieval mode accompanied by response fields specific to that mode.
A trace showing that a search tool started and completed.
Visible URLs alone are weaker evidence.
A model can emit a familiar domain without fetching it in the current session. It can also produce a malformed or invented URL. A citation-like string proves that text was rendered. It does not automatically prove that live retrieval occurred.
When a provider exposes no independent signal, retrieval should remain unproven.
That may make the dataset smaller. It also prevents a configuration assumption from becoming a published finding.
The on/off comparison still does not reveal causality
Suppose retrieval can be independently verified.
The experiment becomes much stronger, but it still does not create a clean map from source to recommendation.
AI answers vary.
The same model can recommend different brands across repeated runs with the same prompt and settings. A brand appearing in the retrieval-on answer and disappearing from one retrieval-off answer could reflect retrieval. It could also reflect ordinary sampling variation.
One run per arm is therefore insufficient.
I would use repeated paired runs under matched conditions:
The same prompt.
The same model and product surface.
The same date window.
The same language.
The same system instructions.
The same sampling settings where controllable.
The same account and regional conditions where relevant.
The only intended difference should be the verified retrieval state.
The four outcomes worth reporting
For each brand and question, the paired experiment creates four useful states.
Present in both modes
The brand persisted across the tested retrieval conditions.
This does not prove why the model knew the brand. It establishes only that the recommendation did not depend on observable retrieval in every tested run.
Present only with retrieval on
The recommendation is retrieval-sensitive under the tested conditions.
This is consistent with live retrieval contributing to the result. It does not prove that the visibly cited page caused the brand to be selected.
Present only with retrieval off
This should not be discarded as an impossible result.
It can arise from answer variance, changed answer composition, different context introduced by retrieved documents, or other mode interactions. It is evidence that a simple “retrieval adds brands” story is incomplete.
Absent in both modes
The brand did not enter the recommendation set in those tested cells.
It is not proof of universal invisibility. The conclusion remains attached to the prompt, surface, model, date and run contract.
Reporting these transitions is more informative than publishing one pooled visibility rate.
Retrieval sensitivity is not source provenance
The most tempting overstatement is:
The brand appeared only with retrieval on, therefore the cited website caused the recommendation.
The experiment does not establish that.
A retrieved answer may consult several pages and display only some of them.
A displayed citation may support a product detail without introducing the brand.
A comparison page may be cited because it supports the final shortlist, even if another source influenced which candidates entered consideration.
The system may retrieve a source and ignore it.
The answer may name a brand from model knowledge and use retrieval only to verify a date, location or price.
Retrieval access, source influence and citation rendering are different variables.
From outside the model, source causality is usually unresolved.
The defensible claim is narrower:
The recommendation changed when verified retrieval state changed under a matched and repeated test.
That is still useful. It tells a team whether live search access appears to matter for that buyer question. It does not tell them that purchasing one directory listing or publishing one article will cause future recommendations.
What the collector should store
A retrieval experiment needs more than an on/off column.
For each request I would retain:
The exact prompt and its buyer intent.
The requested retrieval state.
The observed retrieval state.
The evidence used to classify that state.
The provider, product surface and model identifier.
The request and response timestamps.
Sampling parameters where exposed.
Tool-call, query and source metadata.
The raw answer.
The rendered URLs or source records.
The brand recommendation label and review method.
The response hash.
The collector, extractor and scorer versions.
Invalid requests must remain invalid.
If the provider accepts the request but retrieval cannot be verified, the row should say unproven. It should not quietly enter the on arm.
The practical audit question
When a vendor presents a retrieval-on citation study, I would now ask one question before looking at the headline:
How did you verify that retrieval actually ran for every row labelled retrieval-on?
If the answer is “we set the parameter,” the study has described its request configuration, not its observed retrieval state.
If the answer is “the API returned 200,” it has verified transport success.
If the answer is “the response contained URLs,” it has observed rendered strings.
None is automatically equivalent to an executed search.
A good experiment may ultimately have fewer eligible rows, more unproven states and a less dramatic headline.
That is not a weakness.
It is the measurement system refusing to invent an experiment that did not leave evidence.
The retrieval-on versus retrieval-off design remains worth running. But its honest output is retrieval sensitivity under declared conditions, not a perfect division between model memory and the live web.
And before the comparison begins, the collector must prove that both arms actually exist.
Related measurement framework:
Top comments (0)