I make the observed surface a required field. A generated Gemini API response and a captured consumer-product answer should never become indistinguishable after ingestion.
The pipeline would have two stages: answer observation and cited-source verification. Each produces evidence with its own timestamp.
The observation preserves the experiment
Here is an illustrative record, not output from a provider:
{
"observation_id": "example-observation",
"surface": "gemini_api",
"provider": "example-provider",
"prompt_id": "example-category-question",
"prompt_version": "example-v1",
"prompt_text": "Illustrative research prompt",
"language": "en",
"region": null,
"account_context": "not_applicable_to_this_example",
"model_id": "example-model",
"observed_at": "2026-10-01T12:00:00Z",
"answer_text": "Illustrative answer",
"citations": [],
"status": "complete_without_citations"
}
For a consumer-surface observer, I record the available context and explicitly mark unavailable conditions. I don't populate a model ID based on a guess.
The Gemini Search grounding documentation is my reference for an official grounded-generation path. It should not be used as evidence that API output reproduces the consumer interface.
Citation extraction needs its own acceptance test
I preserve citation URL, visible label, position or supported relationship, and any provider reference ID. If the provider does not expose attachment to a passage, I don't invent that mapping.
My fixtures would include an answer with no citations, one citation, several citations, repeated domains, redirecting URLs, and a brand mentioned only in a cited source.
The last case checks whether answer-text mentions and source mentions remain separate. A flattened text field can silently turn source material into an apparent generated recommendation.
I compare normalized URLs while retaining originals. Tracking parameters may be removable, but path or query differences can identify distinct documents or versions.
I repeat a versioned prompt panel
A single prompt result is one observation. I create a small panel of relevant factual, category, comparison, and navigational prompts, then repeat part of it across scheduled windows.
I hold controllable conditions constant and record what cannot be controlled. Prompt wording, provider configuration, parser version, and observation time belong in the experiment history.
If a provider changes extraction behavior, I would distinguish that event from a change in the answer source. Saving permitted raw outputs and a small reviewed panel helps diagnose the difference.
Source support is downstream work
After answer collection, I would fetch authorized public citation URLs and store final URL, fetch time, content hash, and relevant passage evidence. A separate result would say whether the passage supports the specific claim under review.
I use states such as supported, not_supported_by_reviewed_passage, ambiguous, and source_unavailable. An unavailable page is not proof of a false citation, and topical relevance is not enough to set supported.
The source fetch timestamp must remain distinct from the answer timestamp. A later edit can change what a reviewer sees.
I keep incomplete outcomes visible
Timeout, provider error, partial answer, no answer, and complete answer without citations should have separate states. An empty citations array by itself cannot tell me which condition occurred.
My metrics would include answer fidelity on the reviewed sample, citation precision, record completeness, failure distribution, and natural variation across repeated prompts. I avoid calling all of these one success rate.
I also avoid confidential prompts and another user's chats or uploads. The collection scope should match the research question.
My first milestone would be a small prompt panel whose observations and citations can be manually traced. Automation becomes useful when that trace survives every export and transformation, rather than disappearing behind a visibility score.
Top comments (0)