DEV Community: Waleed Arshad

Version AI-Visibility Prompt Sets Without Breaking Before/After Comparisons

Waleed Arshad — Tue, 11 Aug 2026 23:10:13 +0000

A prompt set is part of an AI-visibility measurement instrument. If you edit that instrument between two collection runs, a change in the output may reflect the prompts rather than a change in the brand, sources, or answer system.

That sounds obvious, yet prompt libraries often live in a spreadsheet where wording, intent labels, and sampling rules are overwritten in place. The team then compares “before” and “after” percentages as if both runs used the same test.

The fix is not to freeze prompts forever. It is to version them like code and make comparability an explicit property of every report.

Treat the prompt set as an immutable release

Give each release a stable identifier and never mutate it after it has produced observations. A compact JSON representation can look like this:

{
  "prompt_set_id": "retail-ai-discovery-us",
  "version": "1.2.0",
  "parent_version": "1.1.0",
  "effective_at": "2026-08-11",
  "locale": "en-US",
  "sampling": {
    "engines": ["engine-a", "engine-b"],
    "runs_per_prompt": 3
  },
  "prompts": [
    {
      "prompt_id": "local-discovery-001",
      "intent": "local_discovery",
      "template": "Which providers offer {{service}} in {{city}}?",
      "variables": ["service", "city"],
      "status": "active"
    }
  ]
}

prompt_set_id identifies the long-lived measurement program. version identifies one immutable release. parent_version creates a traceable lineage. The sampling block belongs in the same release because a different engine list or run count can change what the aggregate means even when every prompt string is identical.

Keep prompt IDs stable across releases only when the underlying measurement intent is stable. A prompt ID is not a row number; it is the join key for matched comparisons.

Use a change policy that reflects analytical impact

Semantic versioning is useful if its rules are defined for measurement rather than package compatibility.

Patch: metadata corrections that do not alter the generated prompt, such as fixing a description or owner field. A typo correction should be treated cautiously; if it changes the submitted string, store the new string and hash even when the team believes the meaning is unchanged.

Minor: additive changes that preserve the existing prompt IDs and sampling policy. New prompts may expand coverage, but before/after reporting must separate the matched core from the expanded full set.

Major: any change that edits, removes, reweights, or reinterprets an existing prompt; changes variables in a way that alters intent; or modifies the engine, locale, geography, or sampling policy used for the headline comparison.

This policy prevents a new set of high-intent prompts from being blended into an old baseline and presented as brand growth.

Record a real change log

A useful change log needs more than a sentence saying “updated prompts.” Capture these fields for every release:

version and parent version;
release timestamp and owner;
added, modified, deprecated, and removed prompt IDs;
old and new normalized prompt hashes;
intent, locale, and variable changes;
sampling-policy changes;
reason for the change;
expected comparability impact;
reviewer and approval timestamp;
bridge-run requirement and completion state.

A normalized hash should be deterministic, but do not normalize away meaningful punctuation, negation, ordering, or variable placement. Preserve the exact submitted text separately. The hash detects accidental mutation; the raw text explains it.

A synthetic example: when a rewrite breaks the baseline

Assume version 1.0 contains this prompt:

Which providers offer laser engraving in Austin?

A proposed replacement says:

What is the best laser engraving company in Austin?

Both concern the same service and city, but they do not measure the same thing. The first is a discovery query. The second introduces an evaluative superlative and a narrower company label. Reusing the old prompt ID would hide an intent change.

The safer release creates a new prompt ID, deprecates the old one without deleting it, and increments the major version. During a bridge period, collect both prompts under the same engine, locale, and sampling conditions. Report three views:

the matched legacy core, using only unchanged prompt IDs;
the new prompt’s results as a separate series;
the full new set, clearly labeled as a new baseline.

The bridge does not make the prompts equivalent. It shows how the measurement behaves while both definitions are observable.

Validate releases before collection

A lightweight release pipeline can stop most comparability failures.

1. Validate structure

Require the schema fields, a valid version, unique prompt IDs, a declared parent, and an allowed intent value. Reject empty templates and undeclared variables.

2. Compile every prompt

Render all variable combinations used in the run. Fail on missing placeholders, unused variables, unexpected braces, or blank values. Store the compiled text that will actually be submitted.

3. Diff against the parent

Compare IDs, exact text, normalized hashes, intent labels, variables, weights, locale, engine set, and sampling counts. Produce a machine-readable diff and a short reviewer summary.

4. Enforce the version rule

If an existing prompt string or intent changed, a patch release should fail. If prompts were added, require reporting rules for the matched subset. If the engine list changed, require a new baseline or bridge plan.

5. Freeze an observation manifest

Before collection, write a manifest containing the prompt-set version, compiled prompt hashes, run timestamp, engine labels, locale, and sampling configuration. Each observation row should reference that manifest and the stable prompt ID.

6. Test the reporting join

Run the comparison code against a synthetic fixture with unchanged, added, modified, and removed prompts. Confirm that only unchanged IDs enter the matched before/after metric and that new prompts cannot silently enter the denominator.

Report comparability, not just movement

Every chart should state which prompt-set versions it compares and assign a comparability class:

direct: prompt IDs, compiled text, locale, engine set, and sampling policy match;
bridged: a controlled overlap exists, but at least one measurement component changed;
new baseline: the change is too material for a defensible direct comparison.

This label is more honest than adding another decimal place. It also gives reviewers a quick way to decide whether a movement claim is supported by the experiment design.

For teams that need an initial snapshot before building this pipeline, the free AI-visibility audit at Corank can provide a practical starting point; subsequent internal runs should still preserve their own prompt, source, and configuration records.

Limitations

Prompt versioning does not remove model variability, engine updates, personalization, location effects, temporal source changes, or incomplete citation surfaces. A bridge run can still be noisy. Stable prompts also do not prove that a content or SEO change caused an answer change.

Version control solves a narrower but important problem: it prevents the measurement instrument from changing invisibly. Once prompt releases, manifests, and matched-set rules are explicit, the team can separate “the answer changed” from “we changed the question” and make before/after reporting substantially more auditable.

Build a Prompt–Source Gap Triage Queue in TypeScript (Without a Vanity Score)

Waleed Arshad — Tue, 11 Aug 2026 15:04:46 +0000

A visibility dashboard can look precise while hiding the question that actually matters: what useful source is missing from this answer?

This tutorial builds a small, deterministic triage pipeline for AI-answer observations. It does not guess how a model works, scrape private interfaces, or collapse every signal into one “visibility score.” It accepts a frozen prompt set plus human-reviewed source roles and produces an editorial queue you can inspect.

The examples are synthetic. They do not represent measured rankings or customer results.

The unit of analysis

Keep one row per observed prompt run. A minimal input needs:

a stable prompt ID and version;
an intent class;
the answer system and timestamp;
whether the brand was mentioned;
whether the target domain was cited;
source roles that a reviewer observed;
a link to a permissioned evidence artifact.

Source roles describe the job a cited page performs:

definition — explains what an entity or category is;
evidence — supports a factual claim or measurement;
comparison — helps distinguish options;
implementation — helps the reader act;
identity — verifies canonical entity facts;
selection — supports a shortlist or recommendation;
context — adds background without carrying the central claim.

A role is not a quality judgment. Record support separately.

Define the schema

type Intent =
  | "discovery"
  | "diagnosis"
  | "comparison"
  | "implementation"
  | "selection"
  | "risk";

type SourceRole =
  | "definition"
  | "evidence"
  | "comparison"
  | "implementation"
  | "identity"
  | "selection"
  | "context";

type SupportStatus =
  | "supports"
  | "partially_supports"
  | "does_not_support"
  | "unclear";

interface SourceObservation {
  url: string;
  role: SourceRole;
  support: SupportStatus;
}

interface PromptObservation {
  promptId: string;
  promptVersion: string;
  promptText: string;
  intent: Intent;
  answerSystem: string;
  runTimestamp: string;
  brandMentioned: boolean;
  targetDomainCited: boolean;
  sources: SourceObservation[];
  evidenceArtifact: string;
  isSynthetic: boolean;
}

interface TriageItem {
  promptId: string;
  intent: Intent;
  missingRole: SourceRole;
  priority: number;
  reason: string;
}

Keep the answer system and timestamp visible. A repeated run is a sample, not a permanent ranking.

Map intent to expected source roles

The mapping is an editorial hypothesis. It should describe which roles would make an answer more useful, not which publisher the model must cite.

const expectedRoles: Record<Intent, SourceRole[]> = {
  discovery: ["definition", "identity"],
  diagnosis: ["evidence", "context"],
  comparison: ["comparison", "evidence"],
  implementation: ["implementation", "evidence"],
  selection: ["selection", "comparison", "identity"],
  risk: ["evidence", "context"],
};

Version this mapping. If you change it after seeing the results, your new output is not directly comparable with the old one.

Add transparent priority weights

A useful priority is not a claim about model probability. It is simply an ordered work queue. Keep the formula visible.

const intentWeight: Record<Intent, number> = {
  discovery: 2,
  diagnosis: 3,
  comparison: 4,
  implementation: 4,
  selection: 5,
  risk: 3,
};

const roleWeight: Record<SourceRole, number> = {
  definition: 2,
  evidence: 4,
  comparison: 4,
  implementation: 3,
  identity: 2,
  selection: 4,
  context: 1,
};

These values reflect editorial urgency in this example only. Do not present them as a validated industry benchmark.

Calculate missing roles

We count a role as present only when a reviewer found at least partial support. A citation that does not support its attached claim should not make the gap disappear.

function supportedRoles(row: PromptObservation): Set<SourceRole> {
  return new Set(
    row.sources
      .filter(
        (source) =>
          source.support === "supports" ||
          source.support === "partially_supports"
      )
      .map((source) => source.role)
  );
}

function triage(rows: PromptObservation[]): TriageItem[] {
  const queue: TriageItem[] = [];

  for (const row of rows) {
    if (!row.promptId || !row.promptVersion) {
      throw new Error("Every row needs a stable prompt ID and version");
    }

    if (!Number.isFinite(Date.parse(row.runTimestamp))) {
      throw new Error("Invalid timestamp for " + row.promptId);
    }

    const present = supportedRoles(row);

    for (const role of expectedRoles[row.intent]) {
      if (present.has(role)) continue;

      const citationGap = row.targetDomainCited ? 0 : 2;
      const mentionGap = row.brandMentioned ? 0 : 1;

      queue.push({
        promptId: row.promptId,
        intent: row.intent,
        missingRole: role,
        priority:
          intentWeight[row.intent] +
          roleWeight[role] +
          citationGap +
          mentionGap,
        reason: [
          "Expected " + role + " support for " + row.intent + " intent",
          row.brandMentioned ? "brand mentioned" : "brand not mentioned",
          row.targetDomainCited
            ? "target domain cited"
            : "target domain not cited",
        ].join("; "),
      });
    }
  }

  return queue.sort(
    (a, b) =>
      b.priority - a.priority ||
      a.promptId.localeCompare(b.promptId) ||
      a.missingRole.localeCompare(b.missingRole)
  );
}

The deterministic tie-breakers matter. Two people running the same input should receive the same ordering.

Test with synthetic observations

const observations: PromptObservation[] = [
  {
    promptId: "example-selection-01",
    promptVersion: "1.0",
    promptText:
      "What should a mid-market team evaluate in an AI visibility platform?",
    intent: "selection",
    answerSystem: "Example Answer System",
    runTimestamp: "2026-08-11T20:00:00Z",
    brandMentioned: true,
    targetDomainCited: false,
    sources: [
      {
        url: "https://example.org/category-overview",
        role: "identity",
        support: "supports",
      },
    ],
    evidenceArtifact: "https://example.org/evidence/example-selection-01",
    isSynthetic: true,
  },
  {
    promptId: "example-implementation-01",
    promptVersion: "1.0",
    promptText:
      "What evidence should an AI visibility audit preserve?",
    intent: "implementation",
    answerSystem: "Example Answer System",
    runTimestamp: "2026-08-11T20:05:00Z",
    brandMentioned: false,
    targetDomainCited: false,
    sources: [
      {
        url: "https://example.org/audit-method",
        role: "evidence",
        support: "partially_supports",
      },
    ],
    evidenceArtifact:
      "https://example.org/evidence/example-implementation-01",
    isSynthetic: true,
  },
];

console.table(triage(observations));

For the selection prompt, the queue will preserve the missing selection and comparison roles. The identity page confirms the entity; it does not independently justify a recommendation.

For the implementation prompt, evidence is present, but an actionable implementation source is still missing.

Convert a gap into an asset brief

A missing role is a research lead, not automatic permission to publish another article. Confirm the prompt genuinely calls for the role.

function assetBrief(item: TriageItem): string {
  const format: Record<SourceRole, string> = {
    definition: "concise definition page with scope and exclusions",
    evidence: "methodology or dataset with dates and limitations",
    comparison: "criteria-led comparison with explicit tradeoffs",
    implementation: "reproducible tutorial, template, or checklist",
    identity: "canonical entity profile with consistent facts",
    selection: "use-case matrix with qualification criteria",
    context: "background explainer tied to the answer claim",
  };

  return [
    "Prompt: " + item.promptId,
    "Role gap: " + item.missingRole,
    "Suggested format: " + format[item.missingRole],
    "Priority: " + item.priority,
    "Reason: " + item.reason,
  ].join("
");
}

The output is small enough for an editor to review. That is the point. Automation should make the judgment legible, not hide it.

Production guardrails

Before using this in a reporting workflow:

Freeze and version the prompt portfolio.
Use a real CSV parser instead of splitting arbitrary CSV on commas.
Validate timestamps, URLs, enum values, and duplicate run keys.
Keep synthetic rows physically or logically separate from observations.
Preserve answer-level evidence where terms and platform policy allow it.
Store the source claim and reviewer note, not only the role label.
Measure reviewer agreement for consequential analyses.
Re-run the same prompt set before interpreting change.
Keep mentions, citations, recommendations, support, and prominence separate.
Never describe a correlation as proof that one backlink caused an answer.

What this queue cannot tell you

This method cannot reveal hidden retrieval, training data, or causal attribution. It cannot guarantee that publishing a missing asset will change an answer. Interfaces, citations, personalization, geography, and model versions can change between runs.

It also cannot replace editorial judgment. A definition page may be the wrong asset for a selection question, and a vendor-authored comparison may not be independent enough to support a recommendation.

Use the queue to choose the next evidence question. Then publish the smallest truthful asset that answers it, distribute it where the intended audience already participates, and re-test with the frozen prompt set.

If you want a broader baseline before building the queue, Corank offers a free AI visibility audit across major answer-engine surfaces.

Build a Reproducible AI Visibility Evidence Table with JSON

Waleed Arshad — Mon, 10 Aug 2026 03:09:52 +0000

AI visibility audits often collapse into a single score. That makes the report easy to scan, but hard to reproduce.

A better technical workflow keeps the raw evidence first, calculates summaries second, and preserves enough context for another analyst to re-run the same test.

This post shows a compact JSON model for that workflow.

1. Store the test, not just the answer

For every prompt, record the exact conditions under which the answer was observed:

{
  "run_id": "2026-08-09-us-en-001",
  "observed_at": "2026-08-09T18:00:00Z",
  "engine": "example-ai-surface",
  "market": "US",
  "language": "en",
  "prompt": "Which tools help measure brand visibility in AI answers?",
  "brand": "Corank",
  "mentioned": true,
  "linked": false,
  "position": 3,
  "sources": [
    {
      "url": "https://example.com/resource",
      "role": "supporting-evidence"
    }
  ]
}

The important design choice is separating mentioned from linked. An entity can appear without a citation, and a cited source can appear without a direct brand mention. Treating those as different fields prevents a lot of misleading reporting.

2. Define source roles

Not every cited page is doing the same job. A small controlled vocabulary makes later analysis much more useful:

definition: explains what a category or method is
evidence: contains data, research, or a reproducible result
comparison: helps a user evaluate alternatives
implementation: shows how to do the work
entity: confirms who a company or product is
supporting-evidence: adds context without carrying the main claim

This lets you ask a better question than “Were we cited?” You can ask, “Which source role do competitors own, and which role is missing from our content?”

That gap is often the next linkable asset to build.

3. Normalize URLs before counting

The same source can appear with tracking parameters, fragments, mixed casing, or multiple trailing-slash variants. Normalize before aggregation:

function normalizeSourceUrl(input) {
  const url = new URL(input);
  url.hash = "";

  for (const key of [...url.searchParams.keys()]) {
    if (key.startsWith("utm_") || key === "ref") {
      url.searchParams.delete(key);
    }
  }

  url.hostname = url.hostname.toLowerCase();
  url.pathname = url.pathname.replace(/\/+$/, "") || "/";

  return url.toString();
}

Do not remove query parameters blindly. Some parameters identify a genuinely different public resource.

4. Calculate transparent metrics

Once raw observations are saved, summaries become simple and auditable:

const mentionRate =
  runs.filter(run => run.mentioned).length / runs.length;

const citationRate =
  runs.filter(run => run.linked).length / runs.length;

const sourceRoleCoverage = new Set(
  runs.flatMap(run => run.sources.map(source => source.role))
);

Keep the numerator and denominator beside every percentage. “42% visibility” is not meaningful unless the reader can see whether that means 42 of 100 prompts or 5 of 12.

5. Use a fixed prompt set

For trend analysis, freeze a prompt set for a period of time. Re-run it on a schedule, but avoid rewriting the prompts every time the result changes.

A useful prompt set usually mixes:

category discovery questions
comparison and alternative questions
“how do I” workflow questions
vendor or tool evaluation questions
branded confirmation questions

Version the set when you intentionally change it. Otherwise, a chart may be measuring prompt drift instead of visibility drift.

6. Turn the evidence gap into an asset

If the table shows that competitors are repeatedly cited for implementation guidance, build an implementation guide. If comparison pages dominate, create a fair, sourced comparison framework. If original data wins citations, publish the methodology and dataset.

The evidence table is not only a report. It is a content roadmap.

For a practical starting point, Corank publishes a free AI visibility audit and guides to answer engine optimization and generative engine optimization.

Final checklist

Before calling a run complete, verify that you preserved:

the exact prompt
engine or product surface
date, market, and language
mention and link status separately
normalized source URLs
source roles
raw counts behind percentages
the prompt-set version

That structure makes AI visibility work easier to audit, easier to compare, and much harder to turn into a vanity metric.

Audit Backlink Domain Concentration with Node.js (Without Fooling Yourself)

Waleed Arshad — Sat, 08 Aug 2026 04:43:01 +0000

A spreadsheet can contain hundreds of valid backlink URLs and still describe a fragile off-page footprint.

I ran into this while auditing 1,023 verified public placements. About 84.8% of them lived on four hosts. The URLs were real, but the graph was concentrated.

This tutorial builds a small Node.js audit that answers four practical questions:

How many unique placement URLs do we have?
How many unique referring hosts?
What share belongs to the largest one and four hosts?
Which hosts should be paused before the next publishing sprint?

The code uses only built-in Node APIs, so it is easy to run in CI or adapt to a Google Sheets export.

Input format

Export the tracker as a CSV with two columns:

name,url
Medium article,https://medium.com/@example/article
Company profile,https://directory.example.com/company
Technical guide,https://dev.to/example/guide

For a production tracker, use a proper CSV parser because titles can contain commas. The parsing below is intentionally small so the concentration logic is easy to see.

Normalize URLs before counting

Tracking parameters, fragments, casing, and www variants can make identical destinations look different.

function normalizeUrl(raw) {
  const url = new URL(raw.trim());
  url.hash = '';

  for (const key of [...url.searchParams.keys()]) {
    if (key.startsWith('utm_') || key === 'ref' || key === 'source') {
      url.searchParams.delete(key);
    }
  }

  url.hostname = url.hostname.toLowerCase().replace(/^www\./, '');
  url.pathname = url.pathname.replace(/\/+$/, '') || '/';

  return url.toString();
}

function referringHost(raw) {
  return new URL(normalizeUrl(raw)).hostname;
}

Do not remove every query parameter blindly. Some platforms use query parameters as part of the canonical identity of a public page. Strip only parameters you have classified as tracking noise.

Calculate concentration

function concentrationReport(rows) {
  const uniqueUrls = new Map();

  for (const row of rows) {
    try {
      const normalized = normalizeUrl(row.url);
      uniqueUrls.set(normalized, row);
    } catch {
      // Keep invalid or non-public URLs out of verified totals.
    }
  }

  const byHost = new Map();

  for (const normalized of uniqueUrls.keys()) {
    const host = referringHost(normalized);
    byHost.set(host, (byHost.get(host) || 0) + 1);
  }

  const hosts = [...byHost.entries()]
    .map(([host, placements]) => ({ host, placements }))
    .sort((a, b) => b.placements - a.placements);

  const total = uniqueUrls.size;
  const topOne = hosts[0]?.placements || 0;
  const topFour = hosts.slice(0, 4)
    .reduce((sum, row) => sum + row.placements, 0);

  return {
    totalPlacements: total,
    uniqueHosts: hosts.length,
    topOneShare: total ? topOne / total : 0,
    topFourShare: total ? topFour / total : 0,
    hosts
  };
}

Turn the result into a publishing control

A report is useful only if it changes the queue.

function allocationDecision(report, host) {
  const current = report.hosts.find(row => row.host === host);
  const share = current ? current.placements / report.totalPlacements : 0;

  if (share >= 0.20) return 'PAUSE';
  if (share >= 0.08) return 'REVIEW';
  return 'ELIGIBLE';
}

The thresholds are policy choices, not search-engine rules. Their purpose is to keep an automated publishing process from selecting the lowest-friction host forever.

One useful sprint rule is stricter: give the current top four hosts a temporary allocation of zero. That forces research into new editorial, technical, professional, and reviewed-directory surfaces.

Add source types

Hostname diversity alone is not enough. Ten unrelated directories still represent one kind of evidence.

Add a sourceType field such as:

editorial
technical
company_profile
reviewed_directory
social_post
community_answer
earned_media

Then calculate the same distribution by source type. Your goal is a portfolio of independent sources doing different jobs.

Verify links before counting

A row should enter the verified corpus only when:

The URL returns a public page in a signed-out session.
The page contains a clickable link to the intended site.
The placement is stable, relevant, and not duplicated.
Pending review and drafts remain in separate states.

For dynamic sites, a raw HTTP fetch may miss client-rendered content. Use a real browser check as a fallback, but do not treat an authenticated preview as proof that the page is public.

Why this matters for SEO and AEO

A raw backlink total is a production metric. Referring-domain and source-type diversity are closer to evidence metrics.

For answer-engine optimization, the external page also needs to contribute something worth citing: a definition, method, dataset, checklist, or defensible point of view. The linked destination should match the off-site context.

For example, a technical post about entity consistency should link to the relevant guide rather than defaulting to a homepage.

I am applying this measurement approach to the public SEO/AEO work documented by Corank.

The dashboard I would ship

At minimum, expose:

verified placement URLs
unique referring hosts
net-new hosts this sprint
top-one and top-four shares
source-type distribution
deep-link distribution
indexed placements
referral visits
non-branded impressions
AI answer mentions and cited-source appearances

That makes a failure mode visible before another few hundred URLs accumulate on the same host.

The point is not to replace judgment with a formula. It is to give the agent enough memory and constraints that judgment survives scale.

How to Encode a Publish-or-Suppress Policy for AI Visibility Scores

Waleed Arshad — Mon, 03 Aug 2026 18:09:25 +0000

AI-search visibility dashboards are very good at producing numbers. They are usually less explicit about the conditions under which a number should not be published.

That is a reliability problem.

If collection coverage collapses, citations cannot be traced, or classification cannot be replayed, a polished score can create more confidence than the evidence deserves. A safer system treats publication as a policy decision, not the automatic last step of a calculation.

This guide shows how to encode that decision as data, evaluate it consistently, and test the failure paths.

Start with five output states

A binary pass/fail flag is too coarse. A practical publication policy can return one of five states:

publish — the score and its components are decision-grade.
publish_with_warning — the score is usable, but a named reliability issue must travel with it.
components_only — underlying measurements may be shown, but the composite score is withheld.
hold_last_valid — the current run is unreliable, so the UI keeps the most recent valid score and labels its age.
suppress — no score should be shown until the evidence is repaired.

A temporary drop in one model's completion rate may justify a warning. Missing raw answers across half the prompt set may require suppression.

Define the evidence contract first

A publication policy cannot inspect evidence that the pipeline never stored. At minimum, every observation should expose:

type Observation = {
  observationId: string;
  promptId: string;
  engine: string;
  modelVersion?: string;
  scheduledAt: string;
  completedAt?: string;
  status: "completed" | "timeout" | "blocked" | "parse_error" | "excluded";
  eligible: boolean;
  rawAnswer?: string;
  citations: Array<{
    observedUrl: string;
    resolvedUrl?: string;
    canonicalUrl?: string;
    resolutionReason?: string;
  }>;
  classification?: {
    brandMentioned: boolean;
    recommended: boolean;
    confidence: number;
    methodologyVersion: string;
  };
};

Notice what is not erased:

failed observations remain rows;
the raw citation survives URL normalization;
methodology has a version;
eligibility is explicit;
collection status is separate from brand performance.

That separation prevents a timeout from quietly becoming "brand absent."

Express the policy as configuration

Thresholds should be reviewable without reading application code. YAML is convenient:

version: "2026-08-03"
window:
  max_age_hours: 36
  min_eligible_observations: 80

coverage:
  warn_below: 0.95
  suppress_below: 0.80

evidence:
  min_raw_answer_rate: 0.98
  min_citation_trace_rate: 0.95
  min_replay_success_rate: 0.97

classification:
  min_mean_confidence: 0.90
  max_methodology_versions_per_window: 1

actions:
  low_coverage: publish_with_warning
  critical_coverage: hold_last_valid
  missing_raw_evidence: suppress
  broken_citation_lineage: components_only
  mixed_methodology: suppress
  stale_window: hold_last_valid

These numbers are examples, not universal benchmarks. A high-stakes workflow may require stricter thresholds. An exploratory internal dashboard may tolerate more warnings.

The important part is that thresholds and consequences are defined before the team reviews a surprising score.

Evaluate policy separately from scoring

Keep publication logic outside the function that calculates visibility. That makes it harder for scoring changes to silently weaken quality gates.

type Metrics = {
  eligible: number;
  completed: number;
  rawAnswerRate: number;
  citationTraceRate: number;
  replaySuccessRate: number;
  meanConfidence: number;
  methodologyVersions: number;
  ageHours: number;
};

type Decision = {
  state:
    | "publish"
    | "publish_with_warning"
    | "components_only"
    | "hold_last_valid"
    | "suppress";
  reasons: string[];
};

function decidePublication(m: Metrics): Decision {
  const reasons: string[] = [];

  if (m.methodologyVersions > 1) {
    return {
      state: "suppress",
      reasons: ["Window mixes multiple methodology versions"],
    };
  }

  if (m.rawAnswerRate < 0.98) {
    return {
      state: "suppress",
      reasons: ["Too many completed observations lack raw evidence"],
    };
  }

  if (m.ageHours > 36) {
    return {
      state: "hold_last_valid",
      reasons: ["Current evidence window is stale"],
    };
  }

  const coverage = m.eligible === 0 ? 0 : m.completed / m.eligible;

  if (coverage < 0.80) {
    return {
      state: "hold_last_valid",
      reasons: ["Collection coverage is below the critical threshold"],
    };
  }

  if (m.citationTraceRate < 0.95) {
    return {
      state: "components_only",
      reasons: ["Citation lineage is incomplete"],
    };
  }

  if (m.replaySuccessRate < 0.97) {
    reasons.push("Some classifications cannot be reproduced");
  }

  if (m.meanConfidence < 0.90) {
    reasons.push("Classification confidence is below target");
  }

  if (coverage < 0.95) {
    reasons.push("Collection coverage is below target");
  }

  return reasons.length
    ? { state: "publish_with_warning", reasons }
    : { state: "publish", reasons: [] };
}

In production, validate the configuration against a schema and include the policy version in every decision record.

Make the decision auditable

Store the result as its own object:

{
  "scoreWindowId": "weekly-brand-2026-31",
  "policyVersion": "2026-08-03",
  "evaluatedAt": "2026-08-03T17:00:00Z",
  "state": "components_only",
  "reasons": ["Citation lineage is incomplete"],
  "metrics": {
    "coverage": 0.96,
    "rawAnswerRate": 1,
    "citationTraceRate": 0.82,
    "replaySuccessRate": 0.99
  },
  "lastValidScoreWindowId": "weekly-brand-2026-30"
}

Now a reviewer can answer which policy produced the state, which threshold failed, what evidence window was evaluated, and which prior score remains valid.

Do not overwrite the original decision after a repair. Append a new evaluation and preserve the history.

Test counterfactuals, not only happy paths

A score can be numerically correct while the publication policy is broken. Add tests that deliberately damage the evidence:

describe("publication policy", () => {
  it("suppresses when raw answers are missing", () => {
    const decision = decidePublication({
      eligible: 100,
      completed: 100,
      rawAnswerRate: 0.70,
      citationTraceRate: 1,
      replaySuccessRate: 1,
      meanConfidence: 0.98,
      methodologyVersions: 1,
      ageHours: 2,
    });

    expect(decision.state).toBe("suppress");
  });

  it("does not convert timeouts into brand absence", () => {
    const eligible = 100;
    const completed = 75;
    const mentionCount = 30;

    expect(mentionCount / completed).toBe(0.4);
    expect(mentionCount / eligible).not.toBe(0.4);
  });

  it("holds the last valid score when evidence is stale", () => {
    const decision = decidePublication({
      eligible: 100,
      completed: 99,
      rawAnswerRate: 1,
      citationTraceRate: 1,
      replaySuccessRate: 1,
      meanConfidence: 0.98,
      methodologyVersions: 1,
      ageHours: 48,
    });

    expect(decision.state).toBe("hold_last_valid");
  });
});

Other valuable counterfactual tests include:

remove a detected brand mention and confirm the mention metric changes;
mark a completed answer as failed and confirm it leaves performance scoring;
replace a canonical URL while preserving the observed URL;
mix two methodology versions and confirm publication stops;
expire the evidence window and confirm the UI labels the last valid score.

Design the UI around reliability state

Do not hide policy output in a tooltip. The interface should show:

the current publication state;
every reason code;
eligible, completed, failed, and excluded counts;
the policy and methodology versions;
evidence freshness;
a link from the aggregate score to raw observations;
the timestamp and age of any held score.

"Score: 64" and "Last valid score: 64, collected nine days ago; current run suppressed because 43% of raw answers are missing" are not equivalent messages.

Avoid three common mistakes

Failing open

If the policy service errors, the dashboard should not quietly publish. Choose a safe fallback such as holding the last valid score or suppressing the current one.

Hiding failures in the denominator

Keep scheduled, eligible, completed, failed, and excluded counts separate. A collection failure is an operational fact, not negative brand performance.

Repairing history with a fresh model response

A live replay may be useful, but it cannot replace what was actually observed. Store the original answer and the replay as separate events.

The operating principle

A visibility score earns trust when the system can explain both why it published the number and what failure would have stopped publication.

I expanded the measurement side of this idea into a practical falsification protocol:
https://www.linkedin.com/pulse/how-falsify-ai-visibility-score-before-you-trust-waleed-arshad-jqkdc/

Corank applies the same evidence-first principle to AI-search visibility and optimization:
https://corank.ai

The implementation can be small. The key is making the stop rule explicit, versioned, testable, and visible to the people who rely on the score.

How to Audit AI Crawler Access with robots.txt and Server Logs

Waleed Arshad — Mon, 03 Aug 2026 16:51:12 +0000

AI visibility work often starts with prompts, citations, and answer tracking. But there is a more basic question to settle first:

Can the systems you care about reliably fetch the pages you expect them to use?

A site can publish excellent research and still create avoidable discovery problems through a restrictive robots.txt rule, a CDN challenge, inconsistent redirects, or pages that return different status codes to different user agents.

This guide shows a practical, repeatable audit for AI-crawler access using two evidence sources:

the site's declared policy in robots.txt, and
observed requests in server or CDN logs.

The goal is not to guarantee inclusion in any AI answer. It is to make access decisions visible, testable, and easier to maintain.

1. Start with an explicit crawler policy

A short robots.txt file is easier to review than a long collection of overlapping rules.

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

If your organization intentionally applies different rules to specific crawlers, document the business reason next to the configuration in version control. Robots.txt itself does not support inline operational context well, so keep a companion policy file.

# ai-crawler-policy.yml
owner: growth-platform
review_interval_days: 90
default_policy: allow
exceptions:
  - user_agent: ExampleBot
    path: /private-research/
    action: disallow
    reason: unpublished source material
    approved_by: security
    review_on: 2026-10-01

The important part is not the exact schema. It is having an owner, a review date, and a reason for every exception.

2. Parse robots.txt as data

Manual inspection is useful, but automated checks catch accidental changes during deployment.

The following TypeScript example reads a robots.txt file and extracts simple allow/disallow rules. Production parsers should account for the full standard and the matching behavior of the crawlers you monitor, but even a small check can surface unexpected policy changes.

type Rule = {
  userAgent: string;
  directive: "allow" | "disallow";
  path: string;
};

export function parseRobots(input: string): Rule[] {
  const rules: Rule[] = [];
  let currentAgents: string[] = [];

  for (const rawLine of input.split(/\r?\n/)) {
    const line = rawLine.replace(/#.*/, "").trim();
    if (!line) continue;

    const separator = line.indexOf(":");
    if (separator === -1) continue;

    const key = line.slice(0, separator).trim().toLowerCase();
    const value = line.slice(separator + 1).trim();

    if (key === "user-agent") {
      currentAgents = [...currentAgents, value.toLowerCase()];
      continue;
    }

    if (key === "allow" || key === "disallow") {
      for (const userAgent of currentAgents) {
        rules.push({
          userAgent,
          directive: key,
          path: value,
        });
      }
    }
  }

  return rules;
}

A useful CI assertion is that important public paths remain allowed for the default agent.

import { readFile } from "node:fs/promises";
import { parseRobots } from "./parse-robots";

const robots = await readFile("public/robots.txt", "utf8");
const rules = parseRobots(robots);

const blockedPublicPaths = rules.filter(
  (rule) =>
    rule.userAgent === "*" &&
    rule.directive === "disallow" &&
    ["/blog", "/research", "/docs"].some((path) =>
      rule.path.startsWith(path),
    ),
);

if (blockedPublicPaths.length > 0) {
  throw new Error(
    `Public content blocked in robots.txt: ${JSON.stringify(blockedPublicPaths)}`,
  );
}

This does not replace a standards-compliant parser. It creates a focused regression test for the paths your team has declared important.

3. Test the real HTTP response

Robots.txt is only one layer. A crawler can be allowed by policy and still receive a 403, 429, redirect loop, or JavaScript-only shell.

Run a small set of HTTP checks from outside your production network:

curl -I https://example.com/robots.txt
curl -I https://example.com/sitemap.xml
curl -I https://example.com/research/
curl -L -o /dev/null -s -w "%{http_code} %{url_effective}\n" \
  https://example.com/research/

For each important URL, record:

initial status code
final status code after redirects
final canonical URL
response content type
cache or challenge headers
response time
whether the page contains meaningful server-rendered text

Do not rely on a single homepage test. Sample templates separately: documentation, articles, comparison pages, pricing pages, and structured research assets can behave differently at the edge.

4. Query logs for observed crawler traffic

Server and CDN logs show what actually happened. Start with a bounded time window and retain only the fields needed for diagnosis.

Example SQL for a generic request-log table:

SELECT
  DATE_TRUNC('day', request_time) AS day,
  user_agent,
  status_code,
  COUNT(*) AS requests,
  COUNT(DISTINCT path) AS unique_paths
FROM edge_requests
WHERE request_time >= CURRENT_TIMESTAMP - INTERVAL '30 days'
  AND (
    LOWER(user_agent) LIKE '%bot%'
    OR LOWER(user_agent) LIKE '%crawler%'
    OR LOWER(user_agent) LIKE '%spider%'
  )
GROUP BY 1, 2, 3
ORDER BY day DESC, requests DESC;

User-agent text is a discovery signal, not proof of identity. If identity matters, validate requests using the publisher's documented verification method, such as IP or reverse-DNS procedures where available. Do not create a universal list from memory and assume it stays current.

Then inspect errors by path:

SELECT
  path,
  status_code,
  COUNT(*) AS requests,
  MAX(request_time) AS last_seen
FROM edge_requests
WHERE request_time >= CURRENT_TIMESTAMP - INTERVAL '30 days'
  AND status_code >= 400
  AND LOWER(user_agent) LIKE '%bot%'
GROUP BY 1, 2
ORDER BY requests DESC
LIMIT 100;

This turns a vague concern into a queue of concrete fixes.

5. Separate access, discovery, and citation

These are different layers and should have different metrics.

Layer	Question	Example evidence
Access	Can the page be fetched?	200 response, no challenge
Discovery	Was the page requested?	verified log event
Interpretation	Was usable content returned?	rendered text and metadata
Citation	Did an answer reference the page?	observed answer citation

A successful fetch does not prove citation. A citation does not prove that every page is accessible. Keeping the layers separate prevents teams from drawing causal conclusions that the data cannot support.

6. Build a small weekly audit

A useful weekly job can be simple:

fetch robots.txt and store a hash
compare it with the previous version
test a representative URL set
aggregate verified crawler requests
flag new 4xx/5xx patterns
attach owners and deadlines
review access changes alongside AI-visibility changes

Example audit record:

{
  "runAt": "2026-08-03T07:00:00Z",
  "robotsHash": "sha256:replace-with-real-hash",
  "testedUrls": 24,
  "successfulUrls": 23,
  "blockedUrls": 1,
  "newErrorPatterns": [
    {
      "pathPattern": "/research/*",
      "status": 403,
      "owner": "edge-platform"
    }
  ]
}

Use real values from your own audit. The placeholder hash above is intentionally labeled so it cannot be mistaken for production evidence.

7. Review changes with an evidence checklist

Before closing an access issue, verify:

the robots.txt rule is intentional
the live response matches the declared policy
the canonical URL is stable
the page returns meaningful content without a browser-only dependency
bot-management rules are not creating unintended challenges
log retention is long enough to compare periods
verified and unverified user agents are reported separately
access metrics are not presented as citation metrics

Where this fits in an AI-visibility program

Crawler access is infrastructure hygiene. The next layer is measuring how a brand appears across AI answers, how fresh those observations are, and which cited sources recur.

Corank is built around AI-visibility measurement. Its workflow can complement an access audit by helping teams observe the answer layer after the underlying pages are available and monitorable.

The practical sequence is:

make access policy explicit,
verify real HTTP behavior,
observe crawler requests,
measure answer visibility and citations,
review changes with dated evidence.

That sequence keeps technical fixes, content work, and visibility reporting connected without treating any one signal as proof of the others.

Final takeaway

An AI-crawler audit should be boring in the best way: versioned rules, repeatable tests, bounded log queries, clear owners, and explicit uncertainty.

When those basics are in place, teams can spend less time guessing whether a page was reachable and more time improving the content and evidence that people—and answer engines—may choose to use.``

How to Encode a Freshness Policy for AI Visibility Data

Waleed Arshad — Mon, 03 Aug 2026 16:42:08 +0000

AI visibility teams often start with a freshness rule buried inside a query, a notebook, or a dashboard formula. That works until the provider mix changes, a new reporting use case appears, or somebody needs to explain why yesterday's score moved.

A better pattern is to treat freshness as a versioned policy. Store the assumptions in configuration, validate them at startup, and publish the policy version beside every reported score.

This article shows one implementation pattern using YAML and TypeScript. All thresholds are illustrative, not universal benchmarks. Tune them to your provider volatility, collection cadence, and decision risk.

What the policy must express

A useful policy should answer five questions:

How quickly does an observation lose weight?
At what age is it excluded entirely?
How much recent collection coverage is required?
Which operational state should downstream systems receive?
Which policy version produced the score?

Do not collapse all five into one "quality score." Freshness weighting and collection coverage represent different failure modes.

An illustrative YAML policy

policy:
  id: ai-visibility-freshness-v1
  effective_from: 2026-08-03T00:00:00Z

defaults:
  decay:
    shape: exponential
    half_life_hours: 168
    maximum_age_hours: 720
  coverage:
    window_hours: 168
    minimum_ratio: 0.90
  states:
    current:
      maximum_median_age_hours: 48
    aging:
      maximum_median_age_hours: 168
    stale:
      maximum_median_age_hours: 720

overrides:
  - match:
      provider: example-fast-changing-provider
    decay:
      half_life_hours: 72
      maximum_age_hours: 336

  - match:
      query_class: breaking-news
    coverage:
      window_hours: 24
      minimum_ratio: 0.95

The provider and query names above are placeholders. The value of this structure is not the numbers; it is the explicit separation between decay, exclusion, coverage, and state.

Define a narrow TypeScript model

type DecayShape = "linear" | "exponential" | "tiered";

type FreshnessPolicy = {
  id: string;
  effectiveFrom: string;
  decay: {
    shape: DecayShape;
    halfLifeHours?: number;
    maximumAgeHours: number;
  };
  coverage: {
    windowHours: number;
    minimumRatio: number;
  };
  states: {
    currentMaxMedianAgeHours: number;
    agingMaxMedianAgeHours: number;
    staleMaxMedianAgeHours: number;
  };
};

type Observation = {
  observedAt: Date;
  visibility: 0 | 1;
};

type FreshnessState = "current" | "aging" | "stale" | "suppressed";

Keep the model deliberately small. Provider-specific and query-class overrides can be resolved before this object reaches the scoring function.

Calculate the observation weight

For an exponential policy, the half-life formula is straightforward:

function exponentialWeight(
  ageHours: number,
  halfLifeHours: number,
  maximumAgeHours: number,
): number {
  if (ageHours < 0) {
    throw new Error("observedAt cannot be in the future");
  }

  if (ageHours > maximumAgeHours) {
    return 0;
  }

  return Math.pow(0.5, ageHours / halfLifeHours);
}

With a seven-day half-life, a seven-day-old observation receives 0.5 weight and a fourteen-day-old observation receives 0.25 weight. That is arithmetic from the configured formula, not a claim about how quickly any specific AI system changes.

A linear policy can be useful when stakeholders want a simple deadline:

function linearWeight(ageHours: number, maximumAgeHours: number): number {
  if (ageHours < 0) {
    throw new Error("observedAt cannot be in the future");
  }

  return Math.max(0, 1 - ageHours / maximumAgeHours);
}

Produce a weighted visibility rate

function weightedVisibility(
  observations: Observation[],
  now: Date,
  policy: FreshnessPolicy,
): number | null {
  let weightedMentions = 0;
  let totalWeight = 0;

  for (const observation of observations) {
    const ageHours =
      (now.getTime() - observation.observedAt.getTime()) / 3_600_000;

    const weight =
      policy.decay.shape === "exponential"
        ? exponentialWeight(
            ageHours,
            policy.decay.halfLifeHours!,
            policy.decay.maximumAgeHours,
          )
        : linearWeight(ageHours, policy.decay.maximumAgeHours);

    weightedMentions += observation.visibility * weight;
    totalWeight += weight;
  }

  return totalWeight === 0 ? null : weightedMentions / totalWeight;
}

Returning null when total weight is zero is important. A missing current measurement must not silently become a zero visibility result.

Keep recent coverage separate

Coverage asks whether the system collected the runs it expected. Freshness asks how much an existing observation should influence the present score.

Track both:

type ScoreEnvelope = {
  policyId: string;
  computedAt: string;
  weightedVisibility: number | null;
  recentCoverageRatio: number;
  medianAgeHours: number | null;
  oldestIncludedAgeHours: number | null;
  state: FreshnessState;
};

This envelope gives dashboards and APIs enough context to avoid false precision.

Derive an operational state

function classifyState(
  coverageRatio: number,
  medianAgeHours: number | null,
  policy: FreshnessPolicy,
): FreshnessState {
  if (
    medianAgeHours === null ||
    coverageRatio < policy.coverage.minimumRatio
  ) {
    return "suppressed";
  }

  if (medianAgeHours <= policy.states.currentMaxMedianAgeHours) {
    return "current";
  }

  if (medianAgeHours <= policy.states.agingMaxMedianAgeHours) {
    return "aging";
  }

  if (medianAgeHours <= policy.states.staleMaxMedianAgeHours) {
    return "stale";
  }

  return "suppressed";
}

The state should travel with the score. Do not rely on a dashboard color that disappears when data is exported.

Validate policy changes

Configuration is only safer when invalid combinations are rejected. Add startup checks for:

half-life greater than zero;
maximum age greater than the half-life;
coverage ratio between zero and one;
state age thresholds in increasing order;
an effective date and immutable policy identifier;
overrides that match a known provider or query class.

Also store the policy ID in every materialized score. When a policy changes, historical reports can then be reproduced under the original rules.

Test failure scenarios, not only happy paths

A minimal test matrix should include:

every observation is fresh;
all observations are beyond maximum age;
collection coverage falls below the minimum;
one provider continues while another stops;
a future timestamp arrives because of clock skew;
a backfill is delivered late but keeps its real observation time;
an override shortens the half-life for one query class;
a policy version changes halfway through a reporting period.

The expected behavior should be visible degradation: lower weights, an aging or stale state, or suppression. The system should never manufacture a confident current score from insufficient recent evidence.

Rollout checklist

Version the policy in source control or a controlled configuration store.
Validate it before scoring starts.
Record observedAt and computedAt separately.
Preserve raw observations so historical scores can be replayed.
Publish coverage and age statistics beside the rate.
Suppress the score when guardrails fail.
Review overrides when providers or query sets change.
Document the policy where analysts and downstream developers can find it.

Corank is building measurement workflows for how brands appear across AI search and answer systems. For the conceptual background behind decay shapes and decision horizons, see the companion guide: How to Model Freshness Decay in AI Visibility Monitoring.

The governing principle is simple: an old observation may remain useful, but it should never impersonate a current one. A versioned freshness policy makes that distinction explicit, testable, and reproducible.

How to Set Collection SLOs for AI Search Visibility Monitoring

Waleed Arshad — Mon, 03 Aug 2026 15:53:32 +0000

AI-search visibility reports are only as trustworthy as the collection system behind them.

If a dashboard shows a zero, the reader needs to know whether the brand was genuinely absent from a completed answer or whether the run timed out, the response stream ended early, authentication failed, or citations could not be resolved. Treating all of those outcomes as the same number makes a polished chart less useful, not more.

A collection service-level objective (SLO) gives teams a practical way to define, measure, and improve that reliability.

This guide describes an implementation pattern. The thresholds are illustrative starting points, not universal benchmarks.

Start with an explicit run event

Every scheduled observation should create a durable run event before any provider call begins. A useful minimum schema includes:

run_id
prompt_set_version
prompt_id
provider and model
requested_at, started_at, and completed_at
status
failure_class
eligibility_state
attempt_count
answer_preserved
citations_resolved
raw_artifact_uri
collector_version

The important design choice is that an attempted run never disappears. A retry may recover the observation, but it should not erase the evidence that the first attempt failed.

A companion failure taxonomy should distinguish network and timeout errors, authentication and rate limits, incomplete streams, parser failures, citation-resolution failures, and explicit exclusions. I published a deeper implementation guide here: How to build a failure taxonomy for AI visibility data collection.

Define reliability indicators before setting targets

An SLO is only useful when the underlying service-level indicators (SLIs) are measurable from the event data.

1. Eligible-run completion rate

This answers: of the runs the methodology said should execute, how many ended with a completed, preserved answer?

eligible completion rate =
completed eligible runs with preserved answers
/
all eligible scheduled runs

Keep excluded and limited runs out of the denominator only when the eligibility rule is explicit and versioned. Silently dropping them makes the rate look healthier than the collection system really is.

2. Evidence capture rate

A completed status is not enough. The answer artifact should be stored so an analyst can reproduce the classification later.

evidence capture rate =
completed runs with preserved answer artifacts
/
all completed runs

This indicator catches pipelines that calculate a metric successfully but lose the underlying evidence.

3. Citation-resolution rate

For answers that expose citations or outbound sources, measure how often the collector stores both the displayed reference and a resolved destination.

Track this separately from answer completion. A run can produce a perfectly readable answer while the citation resolver fails.

4. Freshness attainment

Define a maximum acceptable age for each reporting tier.

For example, a daily executive report may require that all included observations were collected within the previous 24 hours. A slower research benchmark may use a wider window. The correct limit depends on the reporting promise; the key is to make it visible.

5. Recovery without evidence loss

Retries are normal. Hidden retries are dangerous.

Measure the percentage of recovered runs that retain the original attempt history, including failure class, timestamp, and collector version. This protects the difference between a healthy first-pass system and one that succeeds only after repeated attempts.

Turn the indicators into an SLO

A starting policy might look like this:

Reliability dimension	Illustrative target	Measurement window
Eligible-run completion	98% or better	Rolling 7 days
Evidence capture	99.5% or better	Rolling 7 days
Citation resolution	97% or better	Rolling 7 days
Freshness attainment	99% or better	Per reporting cycle
Recovered runs retaining attempt history	100%	Rolling 30 days

These values are examples. A team should choose targets based on the decisions the report supports, provider constraints, collection frequency, and the cost of a wrong conclusion.

The most important rule is semantic: a failed or indeterminate run must not be converted into a brand-absence observation simply to satisfy a completion target.

Use an error budget

Suppose a benchmark schedules 1,000 eligible runs in a week and the completion SLO is 98%.

That leaves an error budget of 20 runs.

The budget turns reliability into an operating decision:

If five runs fail independently across unrelated prompts, investigate but continue normal collection.
If 15 runs fail for one provider within an hour, treat the pattern as a likely incident.
If the budget is exhausted, pause methodology changes and prioritize collector reliability.
If a retry recovers a run, preserve both attempts and report first-pass and final completion separately.

A single aggregate percentage should not hide correlated failures. Break down the budget by provider, model, prompt group, geography, and collector version so one healthy segment cannot mask another that is unusable.

Build alerts around decision risk

Not every failure deserves a page.

A practical alert policy can use three levels:

Review

Trigger a review when a small number of isolated runs fail but the reporting window remains within budget. Queue samples for replay and inspect the preserved artifacts.

Incident

Open an incident when failures are correlated by provider, model, collector release, or citation resolver, or when a reporting tier risks missing its freshness promise.

Measurement hold

Prevent the affected slice from contributing to visibility scores when the system cannot distinguish genuine absence from collection uncertainty. Mark the slice as limited or under review instead of publishing a misleading zero.

Keep reliability separate from the visibility score

Collection health and brand visibility answer different questions.

Visibility asks whether and how a brand appears in eligible AI answers.
Reliability asks whether the observation pipeline produced enough evidence to trust that conclusion.

Do not reward the visibility score because the collector completed more runs, and do not punish the brand because the collector failed. Show the two dimensions side by side.

For every chart or export, include:

methodology version
prompt-set version
collection window
eligible, limited, excluded, and review counts
first-pass and final completion rates
unresolved citation count
known incident annotations

That context makes changes interpretable when providers, prompts, or collector code evolve.

A compact implementation checklist

Create the run event before the external request.
Version prompts, eligibility rules, parsers, and collectors.
Preserve completed answers before calculating brand presence.
Store every failed attempt, including recovered attempts.
Separate answer completion from citation resolution.
Define SLIs with explicit denominators.
Set targets by reporting risk, not by aesthetics.
Break error budgets down by important segments.
Hold indeterminate slices out of visibility scores.
Publish reliability metadata beside the measurement.

Reliable AI-visibility monitoring is not about pretending every provider behaves the same way. It is about making uncertainty explicit enough that a reader can tell the difference between a market signal and a collection problem.

Corank focuses on measurable AI-search visibility. The reliability framework above is an educational implementation pattern for teams designing or evaluating this kind of monitoring.

How to Design a Data Dictionary for AI Visibility Analytics

Waleed Arshad — Mon, 03 Aug 2026 15:18:55 +0000

AI-visibility programs often start with dashboards. That is understandable: charts make a new discipline feel concrete.

But the dashboard is not the foundation. The foundation is a shared data dictionary that tells every analyst, crawler, pipeline, and stakeholder what one observation means.

Without that dictionary, two people can review the same answer and disagree about whether a brand was merely mentioned, actively recommended, or cited as evidence. The chart may still render. The number may still have two decimal places. It just will not be trustworthy.

This guide describes a practical way to design the data model behind an AI-visibility workflow. It is intended for teams auditing brand mentions, recommendations, citations, and source destinations across AI answer surfaces.

Educational note: this is an implementation framework, not a claim about any particular AI platform's ranking algorithm.

1. Start by declaring the observation grain

Before choosing columns, write one sentence that defines a row.

A useful default is:

One row represents one captured answer for one versioned prompt, on one named AI surface, in one market and locale, at one capture time.

That sentence prevents a common mistake: mixing prompt-level facts, answer-level facts, brand-level annotations, and citation-level facts in a single wide table.

A captured answer may mention several brands and include several citations. Those are one-to-many relationships. Model them as related tables or nested records instead of repeatedly overwriting fields.

A simple logical model has four layers:

Prompt — the question, its intent, and its version.
Observation — where and when the answer was captured.
Entity assessment — how a brand or product appears in that answer.
Citation assessment — which URLs were surfaced and what role they played.

2. Give every mutable thing a version

Prompt text changes. Entity aliases change. URL canonicalization rules change. Analyst rubrics improve.

If a changing object has only an ID, a historical result can silently acquire a new meaning. Pair stable IDs with explicit versions:

prompt_id + prompt_version
entity_id + entity_dictionary_version
methodology_version
canonicalization_rule_version

Never edit an old prompt definition in place after collecting observations. Create a new version and preserve the bridge between versions. This keeps trend lines auditable.

3. Separate collection facts from analyst judgments

Collection facts should describe what happened without interpretation:

observation_id
prompt_id
prompt_version
engine_surface
market
locale
captured_at
capture_status
raw_answer
evidence_bundle_uri

Analyst judgments belong in a second layer:

entity_id
mention_state
recommendation_strength
answer_position
claim_alignment
confidence_label
analyst_id
reviewed_at

That separation matters operationally. If the rubric changes, the team can re-annotate preserved raw evidence without pretending the source observation changed.

4. Use enums that can survive review

Free-text labels drift quickly. One analyst writes "strong mention," another writes "top pick," and a third writes "recommended." A dashboard later treats them as different categories.

Prefer a small controlled vocabulary. For example:

Mention state

absent — the entity is not present.
named — the entity is mentioned without evaluation.
described — the answer provides meaningful descriptive context.
compared — the entity is evaluated against alternatives.
recommended — the answer explicitly suggests choosing or considering it.

Recommendation strength

none
conditional — suitable only for a stated situation or constraint.
positive — clearly favorable, but not the leading choice.
primary — presented as a leading or first choice.

Confidence label

high — the evidence directly matches the definition.
medium — the label is reasonable but requires interpretation.
low — evidence is incomplete or ambiguous.

Confidence should describe the annotation, not the analyst's enthusiasm for the brand.

5. Model citations as first-class records

A URL is not merely a string attached to an answer. It passes through redirects, tracking parameters, fragments, and canonical tags. Preserve both the observed value and the normalized result.

Useful fields include:

raw_cited_url
resolved_url
canonical_url
registrable_domain
source_role
citation_position
http_status_at_capture
last_verified_at

Keep the raw URL immutable. Canonicalization can then be re-run as rules improve.

A conservative normalization policy might:

lowercase the hostname;
remove a default port;
strip a fragment;
remove only known tracking parameters;
follow redirects with a bounded hop count;
honor a canonical tag only when it is safe and relevant;
retain both the resolved and canonical values.

Do not remove every query parameter. Some parameters identify genuinely different resources.

6. Store the evidence needed to reproduce a judgment

A row without evidence is difficult to audit. An evidence bundle can include:

captured answer text;
citation list in displayed order;
screenshot or render;
capture timestamp and timezone;
prompt text and version;
engine surface and visible mode;
collection notes and error state.

The bundle should be addressable with a stable URI or object key. Access controls and retention policies still apply, but the record needs to tell a reviewer where the evidence lives.

A compact observation can store a generated observation ID, the prompt ID and version, the named surface, market, locale, UTC capture time, capture status, methodology version, and evidence-bundle URI. The specific storage system is less important than immutable identity and traceable provenance.

7. Define invariants in plain language and code

An invariant is a rule that must remain true. Good data dictionaries include both a human explanation and a machine-checkable constraint.

Examples:

captured_at must include a timezone.
raw_answer may be empty only when capture_status is not complete.
recommendation_strength cannot be primary when mention_state is absent.
every citation record must reference an existing observation.
one observation cannot contain duplicate citation positions.
a normalized URL never replaces its raw source value.

Put these checks close to ingestion. A dashboard should not be the first place anyone discovers an impossible combination.

8. Build a small adjudication loop

Before scaling collection, ask two analysts to label the same small sample independently.

Do not begin by chasing an impressive agreement score. Begin by finding ambiguous definitions.

For every disagreement:

reveal the evidence;
identify which phrase in the rubric allowed two interpretations;
refine the definition or add a boundary example;
record the decision in a change log;
re-label the affected sample.

Boundary examples are especially valuable. A dictionary becomes useful when it explains not only what a label is, but also the nearest thing that does not qualify.

9. Keep methodology changes visible

A methodology change should create a release note. At minimum, record:

version number;
effective date;
changed fields or definitions;
reason for the change;
expected effect on historical comparability;
migration or re-annotation plan.

If an old and new method cannot be compared directly, show a break in the trend rather than blending them. Honest discontinuity is better than false precision.

10. Design the reporting layer last

Once the schema and definitions are stable, metrics become easier to defend.

For example, "recommendation rate" still needs a denominator. Is it:

all scheduled prompts;
only successful captures;
only answers where the category was understood;
or only observations eligible under a particular rubric version?

Write that denominator into the metric definition. Include exclusions, required fields, and version scope. Then the same calculation can be implemented in SQL, a notebook, or a dashboard without changing meaning.

A practical review checklist

Before shipping an AI-visibility data dictionary, confirm that:

every table has a declared grain;
stable IDs and mutable versions are separate;
raw evidence is preserved;
collection facts and judgments are not conflated;
enum values have positive and negative examples;
URLs retain raw, resolved, and canonical forms;
invariants are machine-checkable;
timestamps include timezones;
analyst and methodology provenance are recorded;
metric denominators and exclusions are explicit;
changes have release notes;
historical observations remain reproducible.

Closing thought

AI-visibility reporting is young enough that teams are still inventing vocabulary while building the measurement systems. That makes definitions a product feature, not administrative cleanup.

At Corank, we think about visibility as an evidence and measurement problem: preserve the answer, define the observation, make judgments reviewable, and let reporting inherit those disciplined choices.

For a companion reference of common terms, see the AI visibility reporting glossary. Use it as a starting point, then adapt definitions to your own methodology and document every change.

How to Audit Where AI Answers Actually Send Users

Waleed Arshad — Mon, 03 Aug 2026 12:55:52 +0000

An AI answer can mention a brand, cite a source, and still send the user somewhere unhelpful.

The displayed citation may resolve through a redirect. The final page may be an old article, a marketplace listing, a review site, or a URL that no longer works. Some interfaces show a publisher name without exposing the full destination until a click. Others include a brand name but no outbound link at all.

That is why AI visibility needs a destination-access audit in addition to mention and citation counts.

Define the unit of analysis

Use the prompt run as the parent record.

One answer can contain several citations and links, but it is still one visibility opportunity. Store outbound destinations as child records instead of treating every link as a separate answer-level win.

A parent run might contain:

{
  "run_id": "run_2026_08_03_001",
  "prompt_id": "comparison_smb_analytics_01",
  "benchmark_version": "v3",
  "provider": "example_provider",
  "interface": "web_search",
  "collected_at": "2026-08-03T12:00:00Z",
  "response_text": "...",
  "entity_id": "brand_123",
  "mention_role": "shortlisted"
}

Each link record should reference that run.

Preserve four URL states

Do not normalize a citation into one URL and discard the rest. Keep at least four states:

Displayed destination: the label or URL visible in the answer interface.
Raw href: the exact link attached to the visible element.
Resolved destination: the final URL after redirects.
Canonical destination: the page's declared or analyst-approved canonical URL.

These fields answer different questions.

The displayed destination shows what the user sees. The raw href preserves the interface output. The resolved destination reveals where the click actually lands. The canonical destination supports deduplication across tracking parameters and equivalent URLs.

A link record can look like this:

{
  "link_id": "link_0042",
  "run_id": "run_2026_08_03_001",
  "display_label": "Example publisher",
  "raw_href": "https://redirect.example/xYz",
  "resolved_url": "https://publisher.example/reviews/product?utm_source=ai",
  "canonical_url": "https://publisher.example/reviews/product",
  "http_status": 200,
  "captured_at": "2026-08-03T12:00:03Z"
}

Never overwrite the raw value with the normalized one.

Classify destination ownership

The final URL should receive an ownership class. A conservative taxonomy is:

Owned product: a product, service, pricing, or signup page controlled by the entity.
Owned editorial: an article, guide, documentation page, or research page controlled by the entity.
Owned profile: an official social, marketplace, or directory profile controlled by the entity.
Independent editorial: a publisher, analyst, or journalist page not controlled by the entity.
Independent review: a review or comparison platform.
Marketplace: an app store, extension store, reseller, or integration marketplace.
Aggregator: a directory or database that summarizes multiple entities.
Unknown: ownership cannot be established from the evidence.
Invalid: the destination fails, loops, or produces a blocked response.

Do not infer ownership only from the displayed publisher name. Resolve and inspect the destination domain.

If a corporate group owns several domains, keep both the registrable domain and the normalized owner ID. That prevents a subsidiary or documentation subdomain from being misclassified as independent.

Record interface accessibility

A link that exists in source data may not be equally accessible to a user.

For every captured destination, record:

clickable in the captured interface
visible without an extra interaction
exposed only after expanding citations
displayed as a full URL, publisher label, favicon, or numbered citation
opens in the same tab or a new context
requires sign-in
blocked by a warning or interstitial
available on mobile, desktop, or both

This is not only a technical detail. A brand can have strong citation coverage and weak referral opportunity if users cannot easily reach the owned domain.

Follow redirects carefully

Resolve redirects with a bounded, auditable process.

Recommended fields include:

redirect hop number
source URL
status code
destination URL
timestamp
method used
error class

Stop on loops, invalid schemes, excessive hops, or destinations outside the allowed protocol set. Do not execute downloaded files or active content just to resolve a destination. A safe resolver should usually limit itself to HTTP and HTTPS requests and apply normal network-security protections.

Keep temporary tracking parameters in the raw chain. Remove them only in the canonicalization layer.

Normalize without over-merging

URL normalization is useful, but aggressive rules can hide meaningful differences.

Safe candidates often include:

lowercasing the host
removing a default port
stripping a fragment when it does not change server content
removing explicitly approved tracking parameters
normalizing a trailing slash according to the site's canonical behavior

Risky candidates include:

deleting every query parameter
treating HTTP and HTTPS as equivalent without checking
collapsing product and editorial subdomains
assuming every locale path is duplicate content
trusting a canonical tag that points to an unrelated page

Store the normalization-rule version. When a rule changes, reprocess a sample and compare the old and new destination groups before rewriting historical reports.

Check claim-to-destination fit

A successful link is not automatically a useful link.

Compare the answer's claim with the final destination:

Does the page support the statement associated with the citation?
Is the content about the named entity or a different product?
Is the page current enough for the claim?
Does the destination contain the feature, price, statistic, or policy referenced in the answer?
Does the link land on the specific evidence page or only on a homepage?

Use a small review taxonomy:

Supports claim
Partially supports claim
Related but non-supporting
Contradicts claim
Unavailable
Not reviewed

Store the evidence sentence and reviewer state. Do not let a classifier silently promote “related” to “supports.”

Calculate destination metrics

Once the records are preserved, useful rates include:

Owned-link rate

Eligible prompt runs with at least one clickable owned destination divided by eligible prompt runs.

Owned-product-link rate

Eligible prompt runs with a clickable owned product or service page divided by eligible prompt runs.

Citation resolution success

Citation links that resolve to a valid final destination divided by citation links tested.

Destination mismatch rate

Resolved destinations whose owner or page type differs materially from the displayed label divided by resolved destinations.

Claim-support rate

Reviewed citation destinations classified as supporting the associated claim divided by reviewed destinations.

Third-party dependency rate

Eligible prompt runs where every clickable destination is independent or marketplace-owned divided by eligible prompt runs with at least one link.

Keep the denominator beside every metric. Do not combine answer-level and link-level records in one rate.

Build a repeatable audit workflow

A practical sequence is:

Freeze the prompt set and eligibility policy.
Capture the full answer and interface context.
Extract visible citations and outbound links.
Store the displayed label and raw href.
Resolve redirects with bounded safety rules.
Capture status, final URL, and redirect chain.
Determine registrable domain and normalized owner.
Apply versioned canonicalization rules.
Classify destination type and interface accessibility.
Review claim-to-destination fit for important links.
Calculate answer-level and link-level metrics separately.
Preserve corrections and rule versions in an audit trail.

Repeat a subset of prompts to learn whether destination behavior is stable. An answer may continue mentioning the same brand while switching its citations or outbound destination.

Example failure patterns

Several patterns deserve separate alerts:

Mention without access: the brand is recommended but no clickable destination is available.
Owned source without attribution: an owned page is cited while the brand is not named.
Third-party detour: the brand is recommended, but the only link points to a review site or marketplace.
Broken evidence: the citation fails or redirects to an unrelated page.
Homepage fallback: a specific claim links only to the company homepage.
Canonical collision: two materially different pages collapse into one normalized URL.
Visibility-only link: the destination exists in extracted data but is not visible or clickable in the captured interface.

These patterns lead to different fixes. A third-party detour is not the same problem as a broken citation.

Keep destination access separate from recommendations

Recommendation is a semantic role. Destination access is an interface and URL outcome.

A brand can be recommended with no link. It can receive an owned link without being recommended. It can be cited as evidence while a competitor is selected. Preserve each state independently.

A useful report can therefore show:

named coverage rate
recommendation rate
owned-source citation rate
owned-link rate
owned-product-link rate
destination mismatch rate
claim-support rate

The combination is more diagnostic than a single visibility score.

I’m building Corank around the broader problem of understanding how brands appear in search and AI-generated answers. Destination access is a critical part of that measurement because an appearance and a usable path to the company are not the same thing.

Track what the interface shows, where the click goes, who owns the destination, and whether the page supports the claim. That turns a citation count into evidence you can act on.

How to Measure Brand Share of Voice in AI Answers Without Double Counting

Waleed Arshad — Mon, 03 Aug 2026 12:16:37 +0000

Share of voice sounds simple: count how often a brand appears and divide by a total.

In AI-generated answers, that shortcut breaks quickly. One answer can mention the same brand several times, cite three pages from one domain, recommend two products from the same company, or fail to return at all. A benchmark can also mix providers, intents, locales, and repeated runs.

If those units are not defined before collection, a polished percentage can hide duplicate counting and a changing denominator.

This guide describes a practical measurement model for brand share of voice in AI answers.

1. Start with the observation unit

The safest base unit is an observation, not a prompt.

An observation is one documented run of:

one exact prompt;
one provider or interface;
one locale and account condition;
one timestamp;
one repeat or retry identifier.

The same prompt on two providers creates two observations. The same prompt repeated three times on one provider creates three observations. A retry remains linked to the failed attempt and should not silently become an unrelated new run.

A compact observation key can combine the prompt version, provider, locale, scheduled window, and repeat number. That makes the denominator auditable.

2. Resolve entities before counting them

AI answers rarely use one canonical brand string. They may use a company name, product name, abbreviation, former name, domain, or punctuation variant.

Create an entity registry before calculating share of voice. Each record should include:

a stable entity ID;
the canonical name;
aliases;
associated domains;
product names;
explicit exclusions.

Entity resolution must also define what does not count. A generic word that happens to match a brand name should not be classified automatically. Ambiguous matches should be reviewed or marked unresolved.

Store the raw matched string alongside the canonical entity ID so the classification can be reproduced.

3. Separate mentions, citations, and recommendations

These are different events.

A mention means the entity appears in the answer.

A citation means a source associated with the entity is linked or referenced as evidence.

A recommendation means the answer presents the entity as an option for the user’s stated task.

One observation can contain all three, but they should not be collapsed into one flag called “visibility.”

For each entity and observation, store three booleans: mentioned, cited, and recommended. Also keep the evidence span or cited URL that supports each classification.

This separation prevents a cited source from being misreported as a product recommendation.

4. Deduplicate within an answer

For basic entity share of voice, count an entity at most once per observation.

If an answer says “Acme” four times, that is still one mentioned observation for Acme. If it links to three Acme pages, that is one cited observation for Acme in the entity-level metric.

Repeated references can be useful as a separate frequency or source-depth metric, but they should not inflate the binary share-of-voice numerator.

The aggregation rule is effectively:

entity_mentioned_in_observation =
  maximum value of all mention events for that entity and observation

The same pattern applies to citation and recommendation flags.

5. Define the eligible denominator

Not every scheduled run belongs in every denominator.

A run may fail because of provider access, timeout, empty response, parsing error, policy block, unsupported locale, or a collection-system failure.

A failed observation is not an absent brand mention.

For a basic mention-share metric:

mention share =
  observations where the entity was mentioned
  divided by successful eligible observations

“Successful eligible” must be explicit. An observation may be successful for answer-level mention analysis but ineligible for citation analysis if the interface does not expose citations.

Keep collection status, mention eligibility, citation eligibility, recommendation eligibility, and failure reason separately. Never turn collection failures into zeros just to keep the denominator convenient.

6. Choose between presence share and competitive share

Two useful metrics answer different questions.

Presence share

entity presence share =
  eligible observations mentioning the entity
  divided by all eligible observations

This asks: “In what percentage of relevant answers did the brand appear?”

Competitive mention share

competitive mention share =
  entity observation-mentions
  divided by total observation-mentions across the comparison set

This asks: “Of all brand appearances in the defined competitive set, what portion belonged to this entity?”

An answer that mentions three competitors contributes one observation-mention to each. The competitive numerator and denominator therefore operate on entity-observation pairs, not raw word frequency.

Report which definition is being used. Both may be valid, but they are not interchangeable.

7. Do not let large answers dominate

Suppose one answer lists ten vendors and another recommends one vendor. Raw competitive share treats the ten-vendor answer as ten entity-observation pairs.

That may be appropriate for coverage, but not for recommendation strength.

A second view can allocate one unit of weight per observation:

entity weight in observation =
  1 divided by the number of comparison-set entities in that observation

If four brands appear, each receives 0.25 for that observation. If one brand appears, it receives 1.0.

This normalized view prevents broad list answers from dominating the result. Keep it separate from binary presence share instead of blending the two without explanation.

8. Handle repeated runs deliberately

Generative outputs vary. Repeats reveal whether visibility is persistent or intermittent.

Two common aggregation choices are:

Run-level share: every successful repeat remains one observation.
Prompt-cell share: repeats are collapsed into a prompt-provider-locale cell before calculating the portfolio metric.

Run-level share reflects observed frequency. Prompt-cell share prevents heavily repeated prompts from receiving accidental extra weight.

For a prompt cell with three repeats, an entity may have persistence of two out of three. The cell can then contribute its persistence value to a higher-level metric.

Do not mix repeated and non-repeated prompts in one naive denominator. Either weight prompt cells equally or disclose why some cells receive more observations.

9. Weight prompts only for a documented reason

Equal weighting is the cleanest default. Weighted benchmarks can be useful when intents differ materially in business importance, but weights create governance work.

If weights are used:

assign them before looking at the current result;
record the source and owner of each weight;
keep them stable for the comparison period;
version every change;
show both weighted and unweighted results.

A transparent weighted presence metric is:

weighted presence =
  sum of prompt-cell weight multiplied by entity persistence
  divided by the sum of eligible prompt-cell weights

Weights should represent a decision model, not a way to improve a disappointing score.

10. Keep provider and intent slices visible

A global share can conceal opposing movements.

For example, one provider may rise while another falls. Branded prompts may improve while non-branded commercial prompts decline. The total may look unchanged.

Always retain sliceable fields for provider and interface, visible model or experience version, intent class, funnel stage, branded versus unbranded, locale, prompt-set version, and collection window.

Do not average providers with different sample sizes without showing the component results.

11. Canonicalize citations without erasing evidence

Citation metrics need two URL forms: the raw URL returned by the interface and the canonical URL used for aggregation.

Redirects, tracking parameters, fragments, mobile paths, and protocol variants can make one source appear to be several.

Store the raw citation URL, resolved URL, canonical URL, registrable domain, and canonicalization-rule version.

For domain share, count one domain at most once per observation. For page-level analysis, preserve individual canonical pages.

Never overwrite the raw URL. It is part of the evidence.

12. A worked example

Assume a small benchmark produces 12 scheduled observations.

one times out;
one returns a successful answer but exposes no citation surface;
Acme is mentioned in five successful observations;
Acme is cited in three citation-eligible observations;
the comparison set produces 14 total entity-observation mentions.

The denominators differ:

mention-eligible observations = 11
citation-eligible observations = 10

Acme’s presence share is five divided by 11, or 45.5%.

Its citation presence is three divided by 10, or 30%.

If Acme owns five of 14 competitive entity-observation mentions, its competitive mention share is 35.7%.

These are three valid numbers answering three different questions. Reporting one as “AI share of voice” without its definition would be misleading.

The example is illustrative; real programs need uncertainty, repeat behavior, and slice sizes shown alongside percentages.

13. Minimum event schema

A practical long-form event table can contain:

observation_id
prompt_id
prompt_version
provider
interface_version
locale
scheduled_window
repeat_number
collection_status
failure_reason
entity_id
raw_entity_string
mentioned
cited
recommended
evidence_span
raw_citation_url
canonical_citation_url
classification_rule_version
review_status

A separate observation table should exist even when no entity event is found. Otherwise, observations with zero matches disappear and the denominator becomes impossible to reconstruct.

14. Quality checks before publishing the metric

Run at least these checks:

every event points to a valid observation;
failed observations are not classified as absent mentions;
one entity has at most one binary event per observation per event type;
aliases map to one canonical entity;
ambiguous aliases are reviewable;
raw and canonical citation URLs are both preserved;
prompt weights sum as expected;
repeated cells follow the declared aggregation policy;
provider and locale sample sizes are visible;
prompt-set or classification changes create a new version.

Also sample raw answers manually. A metric can pass database checks while the underlying classification rule is wrong.

15. Report the metric as a contract

Every chart should state:

the numerator;
the denominator;
the eligibility rule;
the deduplication rule;
the prompt and provider scope;
the weighting policy;
the time window;
the relevant versions;
the number of failures and unresolved classifications.

That contract makes trend changes debuggable. It also allows another analyst to reproduce the result from the observation ledger.

Final principle

AI-answer share of voice is not a count of brand words. It is an aggregation over documented observations, resolved entities, eligible denominators, and versioned rules.

At Corank, we are building around this audit-first view of AI visibility: the evidence and measurement contract should be inspectable, not hidden behind a single unexplained score.

Disclosure: Corank is our product. This article is an educational measurement framework, not a claim that one formula fits every provider, market, or decision.

How to Build an Entity Consistency Ledger for AI Search

Waleed Arshad — Mon, 03 Aug 2026 11:26:51 +0000

AI-search measurement can fail before a prompt is ever sampled.

If a company appears online under several names, domains, categories, addresses, product labels, or descriptions, an answer engine may treat those records as different entities. A monitoring system can make the same mistake. The result is a mix of false negatives, missed citations, duplicate counts, and explanations nobody can reproduce.

An entity consistency ledger is a practical control for this problem. It is not a giant knowledge graph. It is a versioned record of the public facts that identify a company, product, or person, plus the evidence showing where each fact was observed.

This guide describes a schema, collection workflow, conflict policy, and QA process for building one.

What the ledger should answer

For every monitored entity, an analyst should be able to answer:

What is the canonical name?
Which aliases are legitimate?
Which domain and URLs are authoritative?
Which products belong to the entity?
Which categories are claimed publicly?
Which locations and contact details are current?
Where was each value observed?
When was it last verified?
Is a difference an acceptable variant or a real conflict?
Who owns the correction?

If the system cannot answer those questions, entity matching becomes guesswork.

Preserve observations before normalizing them

The most important design rule is to keep raw observations separate from canonical values.

Suppose four profiles contain these names:

Corank
Corank AI
CoreRank.ai
Remap

A destructive cleanup that rewrites every value to "Corank" hides useful history. Some values may be valid aliases, one may be an old brand, and one may be an error.

Store both layers:

Observed value — the exact text found on the source.
Canonical interpretation — the normalized value chosen under an explicit rule.

That gives you an audit trail and makes future corrections possible.

A minimal data model

A compact implementation can use four tables or collections.

1. Entities

Field	Purpose
entity_id	Stable internal identifier
entity_type	Organization, product, person, location
canonical_name	Approved display name
canonical_domain	Primary domain
status	Active, legacy, acquired, retired
owner	Person or team responsible for accuracy
valid_from	Start of this entity version
valid_to	End of this entity version, if closed

2. Aliases

Field	Purpose
entity_id	Parent entity
alias	Observed or approved alternate name
alias_type	Brand, legal, product, abbreviation, legacy, misspelling
match_policy	Exact, case-insensitive, token, manual-only
valid_from	When the alias became valid
valid_to	When it stopped being valid
evidence_url	Source supporting the alias

3. Claims

Field	Purpose
claim_id	Stable claim identifier
entity_id	Entity described
field	Name, category, address, phone, pricing, capability, etc.
canonical_value	Approved interpretation
confidence	Confirmed, probable, disputed, unknown
effective_date	Date the claim became true
expiry_date	Optional review or sunset date
owner	Team responsible for the claim

4. Observations

Field	Purpose
observation_id	Stable record
claim_id	Claim being evaluated
source_url_raw	Exact retrieved URL
source_url_canonical	Normalized URL
observed_value	Exact public value
retrieved_at	Collection timestamp
source_type	Website, directory, social, review, repository, press
classification	Match, acceptable variant, stale, conflicting, unknown
evidence	Snapshot, response excerpt, or hash
reviewer	Human or rule that classified it

A relational database is helpful but not required. A spreadsheet can work for a small inventory if identifiers, evidence URLs, and dates are treated as real fields rather than comments.

Example record

{
  "entity_id": "org_corank",
  "canonical_name": "Corank",
  "canonical_domain": "corank.ai",
  "alias": {
    "value": "Remap",
    "type": "legacy_brand",
    "valid_to": "2026-07-01"
  },
  "observation": {
    "source_url_raw": "https://example.com/company/remap",
    "source_url_canonical": "https://example.com/company/remap",
    "observed_value": "Remap",
    "retrieved_at": "2026-08-03T10:00:00Z",
    "classification": "stale"
  },
  "owner": "brand-operations"
}

The record does not pretend the old name was never used. It explains why the value is now stale.

Step 1: inventory identity-bearing claims

Start with facts that help people and systems decide whether two records describe the same entity:

canonical name
legal name
former names
domain
product names
logo
primary category
short description
headquarters and service locations
phone and public email
founders or key people
social profiles
repository and documentation URLs

Do not begin with every marketing statement. Identity-bearing facts have the highest matching value and are usually easier to verify.

Step 2: declare canonical values and sources

A canonical value needs an owner and a source of truth.

For example:

product capabilities may be owned by product documentation
address and phone may be owned by operations
legal name may be owned by corporate records
category and positioning may be owned by marketing
repository names may be owned by engineering

The source of truth is not necessarily the highest-authority third-party page. It is the record authorized to define the value. Independent sources then corroborate it.

Step 3: collect public observations

Collect from legitimate surfaces where entity facts are likely to appear:

the website and structured data
documentation and changelogs
social profiles
business and software directories
review platforms
public repositories
app marketplaces
press coverage
association listings
partner pages
conference and podcast pages

Respect access controls and platform rules. The purpose is to audit public facts, not to bypass sign-in or scrape private data.

Record the exact retrieved URL, not only the domain. Preserve redirects and tracking parameters before canonicalization so transformations remain inspectable.

Step 4: normalize conservatively

Normalization should remove technical noise without erasing semantic differences.

Usually safe:

lowercasing domains
removing default ports
normalizing protocol where redirects prove equivalence
sorting or removing known tracking parameters
resolving canonical URLs
standardizing whitespace
case-insensitive comparison for approved fields

Potentially unsafe:

collapsing two product names because they share tokens
treating a former brand as a current alias forever
merging locations with similar street names
assuming two social handles belong to the same company
replacing a legal name with a marketing name
stripping locale or path segments that change meaning

Every nontrivial transformation should be named and testable.

Step 5: classify differences

Not every mismatch is an error. Use explicit classes:

Match

The observed value is equivalent to the canonical value under an approved rule.

Acceptable variant

The value is legitimate in context, such as an abbreviation or localized form.

Stale

The value was once valid but should now be updated.

Conflicting

The source asserts a different current fact.

Unknown

The system lacks enough evidence to decide.

This classification prevents teams from wasting time "fixing" valid variants while missing true contradictions.

Step 6: prioritize by identity risk

Fixing every mismatch at once is rarely practical. Score conflicts by:

Reach — how visible is the source?
Relevance — does it influence important prompts or audiences?
Identity weight — name, domain, and product ownership matter more than punctuation.
Recency — is the record still active?
Contradiction severity — could the value cause a wrong recommendation or lost attribution?
Correction control — can the team edit it directly, request an update, or only monitor it?

A stale phone number on a prominent profile can be more urgent than a minor description difference across ten low-use pages.

Step 7: connect the ledger to AI-search measurement

An entity ledger should influence both collection and reporting.

During detection

Use approved aliases and validity windows when identifying mentions. A legacy brand should not count as a current positive mention unless the analysis explicitly asks for historical visibility.

During citation normalization

Attach cited URLs to the entity and claim they support. Preserve raw URLs alongside canonical records.

During scoring

Report uncertain matches separately. Do not silently convert low-confidence entity matches into visibility wins or losses.

During incident review

When a mention disappears, check whether the brand actually lost visibility or whether an alias, URL, or category rule changed.

During remediation

Create tasks against the conflicting source, canonical owner, and verification test. A recommendation without an owner or recheck condition is not operational.

QA rules worth automating

A useful first ruleset can flag:

one active entity with multiple canonical domains
one domain assigned to multiple active entities
aliases with overlapping validity windows and conflicting meanings
public profiles using a legacy name after its validity window
different current phone or address values
product pages that attribute a product to the wrong parent
canonical URLs that redirect to a different entity
observations without timestamps
claims without owners
normalization transformations without a rule identifier

These checks should create review queues, not automatically overwrite evidence.

Measure ledger quality

Track operational metrics rather than a vague "consistency score":

percentage of identity claims with owners
percentage with authoritative evidence
unresolved conflicts by severity
stale observations past their review date
sources with failed retrieval
uncertain entity matches in AI-answer samples
time from conflict discovery to verified correction
regression rate after correction

A single percentage can hide the exact conflict that causes attribution failure.

Common failure modes

Copying one profile everywhere

Consistency is not the same as duplication. Each platform should have content appropriate to its audience while core identity facts remain aligned.

Treating all backlinks as identity evidence

A link can exist without supporting any claim. Evaluate the surrounding page, not only the href.

Overwriting legacy history

Former names and domains are useful when bounded by dates. Deleting them can make historical citations impossible to interpret.

Trusting structured data alone

Schema helps machines parse information, but visible page content and public profiles still need to agree.

Using normalization to hide conflicts

If two values differ in meaning, a fuzzy-match threshold should not force them together.

Scoring failed collection as inconsistency

A timeout is not a conflicting fact. Collection health must be evaluated before entity quality.

A practical rollout

Week 1: define the entity, claim, alias, and observation schemas.

Week 2: inventory high-impact identity claims and assign owners.

Week 3: collect the website, documentation, social, directory, and repository surfaces.

Week 4: classify conflicts and correct the highest-risk records.

Then version the ledger, rerun the checks, and monitor new observations continuously.

The system becomes valuable when another analyst can reproduce why two records were merged, why a value was marked stale, and how a correction was verified.

Closing principle

Entity consistency is not cosmetic brand hygiene. It is part of the measurement layer.

If the identity model is wrong, citation attribution, competitor comparison, and visibility scoring can all be wrong in ways that look statistically convincing.

Disclosure: I work on Corank, where we are building auditable AI-search measurement and hands-on remediation. You can see the approach at https://corank.ai.

The goal is not to make every page say the same thing. It is to make every important claim traceable to a clear entity, a responsible owner, and verifiable public evidence.