How to Build a Citation Replay Harness for AI Search Audits

AI-search answers are volatile. Run the same prompt twice and you may get different wording, different citations, or a different recommendation order. That makes a normal screenshot useful as evidence, but weak as a measurement system.

A stronger approach is to build a citation replay harness: a small pipeline that stores every input and raw output needed to reproduce an audit, then re-runs the same benchmark against a versioned measurement contract.

This guide describes the data model, normalization rules, replay logic, and QA checks I would use.

What a replay harness should answer

A useful audit should let an analyst distinguish between three kinds of change:

1. The model changed. The same prompt and conditions produced a different answer.
2. The source changed. A cited page redirected, disappeared, or changed its canonical URL.
3. The measurement changed. A parser, matching rule, or scoring formula was updated.

If the system stores only a final visibility score, those causes become indistinguishable. Store the evidence before calculating the metric.

Treat every run as an immutable event

The safest mental model is event sourcing. Each prompt run becomes an immutable event with enough context to replay or reinterpret it later.

A minimal record could look like this:

type PromptRun = {
  runId: string
  benchmarkId: string
  benchmarkVersion: string
  promptId: string
  promptText: string

  provider: string
  model: string
  locale: string
  requestedAt: string
  completedAt: string

  rawAnswer: string
  rawCitations: RawCitation[]
  requestParameters: Record<string, unknown>

  parserVersion: string
  normalizationVersion: string
  scoringVersion: string
}

Do not overwrite this record after scoring. If parsing logic improves, create a new derived result that points back to the original run.

type DerivedObservation = {
  observationId: string
  runId: string
  createdAt: string

  parserVersion: string
  normalizationVersion: string
  brandMatchVersion: string

  mentions: BrandMention[]
  citations: NormalizedCitation[]
  scoreComponents: Record<string, number>
}

This separation is important. Raw evidence answers “what happened?” Derived observations answer “how did version N interpret it?”

Store raw citations before normalizing them

Citation data can arrive as inline links, footnotes, source cards, or provider-specific JSON. Preserve the original form first.

type RawCitation = {
  position: number
  displayText?: string
  href?: string
  providerPayload?: unknown
}

Then normalize into a separate structure:

type NormalizedCitation = {
  rawPosition: number
  inputUrl: string
  resolvedUrl?: string
  canonicalUrl?: string

  hostname: string
  registrableDomain: string
  normalizedPath: string

  resolutionStatus:
    | 'resolved'
    | 'redirect_loop'
    | 'timeout'
    | 'blocked'
    | 'invalid'

  normalizationVersion: string
}

Keeping both forms prevents a future normalization change from erasing the evidence needed to understand an old report.

Version URL normalization explicitly

URL normalization is where many visibility reports quietly drift. Two citations may refer to the same document while appearing different:

• HTTP versus HTTPS
• uppercase versus lowercase hostnames
• a trailing slash
• tracking parameters
• a redirecting short URL
• a mobile or AMP route
• a canonical tag pointing elsewhere

Write the normalization policy down and give it a version.

A conservative pipeline might:

1. Parse the URL without guessing missing components.
2. Lowercase the scheme and hostname.
3. Remove a default port.
4. Resolve redirects with a strict hop limit.
5. Remove only an approved list of tracking parameters.
6. Preserve query parameters that may change page meaning.
7. Read the canonical tag, but retain both resolved and declared-canonical URLs.
8. Normalize the path without collapsing meaningful case.
9. Record every transformation.

Do not remove every query string. A product filter, language code, or document version may live there.

Add a transformation ledger

A normalized URL is much easier to audit when each transformation is visible.

type UrlTransformation = {
  step: string
  before: string
  after: string
  reason: string
}

type UrlNormalizationResult = {
  input: string
  output: string
  version: string
  transformations: UrlTransformation[]
}

An analyst can now see whether two citations were merged because of a redirect, a tracking-parameter rule, or a canonical declaration.

Make the benchmark replayable

A benchmark is more than a list of prompts. It should include the conditions that make comparisons meaningful.

type BenchmarkDefinition = {
  benchmarkId: string
  version: string
  createdAt: string

  prompts: {
    promptId: string
    text: string
    intent: string
    weight: number
  }[]

  providers: {
    provider: string
    model: string
    requestParameters: Record<string, unknown>
  }[]

  locales: string[]
  samplingPolicy: {
    repetitions: number
    spacingMinutes: number
  }

  scoringContract: string
}

When prompts change, publish a new benchmark version. Never silently replace prompt text under an existing ID.

Replay in two different modes

A good harness supports two kinds of replay.

Interpretation replay

Reuse stored raw answers and citations, but apply a new parser, matcher, normalization rule, or scoring formula.

This answers:

How would the new measurement contract interpret the old evidence?

Acquisition replay

Run the original prompt set again against the selected providers and models.

This answers:

What does the current AI-search environment return under comparable conditions?

Keep the modes separate. Otherwise a parser update and a new model response can land in the same diff.

Classify differences instead of showing one delta

A single score change hides useful information. Produce a structured diff.

type ReplayDiff = {
  runIdA: string
  runIdB: string

  answerTextChanged: boolean
  brandMentionChanged: boolean
  citationSetChanged: boolean
  citationOrderChanged: boolean
  normalizationChanged: boolean
  parserChanged: boolean
  scoringChanged: boolean

  addedCitationUrls: string[]
  removedCitationUrls: string[]
  redirectedCitationUrls: {
    before: string
    after: string
  }[]
}

The report can still show an aggregate metric, but every aggregate should link back to the run-level diff and raw evidence.

Test the harness with fixtures

Network calls make tests slow and nondeterministic. Save representative provider responses as fixtures, including awkward cases:

• an answer with no citations
• the same citation repeated twice
• a redirect chain
• a relative canonical URL
• an invalid URL
• two brands with similar names
• a brand name used as a generic word
• a citation embedded in a source card
• a page whose canonical tag crosses domains
• a Unicode hostname or path

Then write golden tests for normalized citations and derived observations.

it('keeps semantic query parameters', () => {
  const result = normalizeUrl(
    'https://example.com/report?year=2026&utm_source=test'
  )

  expect(result.output).toBe(
    'https://example.com/report?year=2026'
  )
})

The test is valuable because it encodes the policy, not merely the code path.

Add operational QA gates

Before a run contributes to a report, validate:

• every prompt maps to one benchmark version
• every raw answer has an acquisition timestamp
• every citation points to its raw provider object
• every derived observation names parser and normalization versions
• failed URL resolutions remain visible
• sampling repetitions match the benchmark policy
• no score is calculated from a partially missing run without a flag
• aggregate values can be traced to individual prompt runs

A failed gate should create a visible status, not silently drop the record.

Keep the public report explainable

The public-facing result does not need to expose internal secrets or every raw payload. It should, however, explain:

• what was measured
• which platforms and locales were included
• when sampling occurred
• how prompts were selected
• how brand mentions and citations were matched
• which methodology version produced the report
• where limitations apply

The goal is not to pretend AI answers are deterministic. It is to make uncertainty inspectable.

Where this fits

This replay pattern is useful for any team measuring answer-engine optimization, generative-engine optimization, or brand visibility inside AI-generated answers.

I work on the same evidence-first problem at Corank, where the focus is making AI visibility reporting traceable to prompts, raw answers, citations, and methodology versions.

The key design principle is simple: preserve raw evidence, version every interpretation, and make every score replayable. Once that foundation exists, a visibility chart becomes something an analyst can investigate instead of merely trust.