How to Encode a Publish-or-Suppress Policy for AI Visibility Scores

AI-search visibility dashboards are very good at producing numbers. They are usually less explicit about the conditions under which a number should not be published.

That is a reliability problem.

If collection coverage collapses, citations cannot be traced, or classification cannot be replayed, a polished score can create more confidence than the evidence deserves. A safer system treats publication as a policy decision, not the automatic last step of a calculation.

This guide shows how to encode that decision as data, evaluate it consistently, and test the failure paths.

Start with five output states

A binary pass/fail flag is too coarse. A practical publication policy can return one of five states:

1. publish — the score and its components are decision-grade.
2. publish_with_warning — the score is usable, but a named reliability issue must travel with it.
3. components_only — underlying measurements may be shown, but the composite score is withheld.
4. hold_last_valid — the current run is unreliable, so the UI keeps the most recent valid score and labels its age.
5. suppress — no score should be shown until the evidence is repaired.

A temporary drop in one model's completion rate may justify a warning. Missing raw answers across half the prompt set may require suppression.

Define the evidence contract first

A publication policy cannot inspect evidence that the pipeline never stored. At minimum, every observation should expose:

type Observation = {
  observationId: string;
  promptId: string;
  engine: string;
  modelVersion?: string;
  scheduledAt: string;
  completedAt?: string;
  status: "completed" | "timeout" | "blocked" | "parse_error" | "excluded";
  eligible: boolean;
  rawAnswer?: string;
  citations: Array<{
    observedUrl: string;
    resolvedUrl?: string;
    canonicalUrl?: string;
    resolutionReason?: string;
  }>;
  classification?: {
    brandMentioned: boolean;
    recommended: boolean;
    confidence: number;
    methodologyVersion: string;
  };
};

Notice what is not erased:

• failed observations remain rows;
• the raw citation survives URL normalization;
• methodology has a version;
• eligibility is explicit;
• collection status is separate from brand performance.

That separation prevents a timeout from quietly becoming "brand absent."

Express the policy as configuration

Thresholds should be reviewable without reading application code. YAML is convenient:

version: "2026-08-03"
window:
  max_age_hours: 36
  min_eligible_observations: 80

coverage:
  warn_below: 0.95
  suppress_below: 0.80

evidence:
  min_raw_answer_rate: 0.98
  min_citation_trace_rate: 0.95
  min_replay_success_rate: 0.97

classification:
  min_mean_confidence: 0.90
  max_methodology_versions_per_window: 1

actions:
  low_coverage: publish_with_warning
  critical_coverage: hold_last_valid
  missing_raw_evidence: suppress
  broken_citation_lineage: components_only
  mixed_methodology: suppress
  stale_window: hold_last_valid

These numbers are examples, not universal benchmarks. A high-stakes workflow may require stricter thresholds. An exploratory internal dashboard may tolerate more warnings.

The important part is that thresholds and consequences are defined before the team reviews a surprising score.

Evaluate policy separately from scoring

Keep publication logic outside the function that calculates visibility. That makes it harder for scoring changes to silently weaken quality gates.

type Metrics = {
  eligible: number;
  completed: number;
  rawAnswerRate: number;
  citationTraceRate: number;
  replaySuccessRate: number;
  meanConfidence: number;
  methodologyVersions: number;
  ageHours: number;
};

type Decision = {
  state:
    | "publish"
    | "publish_with_warning"
    | "components_only"
    | "hold_last_valid"
    | "suppress";
  reasons: string[];
};

function decidePublication(m: Metrics): Decision {
  const reasons: string[] = [];

  if (m.methodologyVersions > 1) {
    return {
      state: "suppress",
      reasons: ["Window mixes multiple methodology versions"],
    };
  }

  if (m.rawAnswerRate < 0.98) {
    return {
      state: "suppress",
      reasons: ["Too many completed observations lack raw evidence"],
    };
  }

  if (m.ageHours > 36) {
    return {
      state: "hold_last_valid",
      reasons: ["Current evidence window is stale"],
    };
  }

  const coverage = m.eligible === 0 ? 0 : m.completed / m.eligible;

  if (coverage < 0.80) {
    return {
      state: "hold_last_valid",
      reasons: ["Collection coverage is below the critical threshold"],
    };
  }

  if (m.citationTraceRate < 0.95) {
    return {
      state: "components_only",
      reasons: ["Citation lineage is incomplete"],
    };
  }

  if (m.replaySuccessRate < 0.97) {
    reasons.push("Some classifications cannot be reproduced");
  }

  if (m.meanConfidence < 0.90) {
    reasons.push("Classification confidence is below target");
  }

  if (coverage < 0.95) {
    reasons.push("Collection coverage is below target");
  }

  return reasons.length
    ? { state: "publish_with_warning", reasons }
    : { state: "publish", reasons: [] };
}

In production, validate the configuration against a schema and include the policy version in every decision record.

Make the decision auditable

Store the result as its own object:

{
  "scoreWindowId": "weekly-brand-2026-31",
  "policyVersion": "2026-08-03",
  "evaluatedAt": "2026-08-03T17:00:00Z",
  "state": "components_only",
  "reasons": ["Citation lineage is incomplete"],
  "metrics": {
    "coverage": 0.96,
    "rawAnswerRate": 1,
    "citationTraceRate": 0.82,
    "replaySuccessRate": 0.99
  },
  "lastValidScoreWindowId": "weekly-brand-2026-30"
}

Now a reviewer can answer which policy produced the state, which threshold failed, what evidence window was evaluated, and which prior score remains valid.

Do not overwrite the original decision after a repair. Append a new evaluation and preserve the history.

Test counterfactuals, not only happy paths

A score can be numerically correct while the publication policy is broken. Add tests that deliberately damage the evidence:

describe("publication policy", () => {
  it("suppresses when raw answers are missing", () => {
    const decision = decidePublication({
      eligible: 100,
      completed: 100,
      rawAnswerRate: 0.70,
      citationTraceRate: 1,
      replaySuccessRate: 1,
      meanConfidence: 0.98,
      methodologyVersions: 1,
      ageHours: 2,
    });

    expect(decision.state).toBe("suppress");
  });

  it("does not convert timeouts into brand absence", () => {
    const eligible = 100;
    const completed = 75;
    const mentionCount = 30;

    expect(mentionCount / completed).toBe(0.4);
    expect(mentionCount / eligible).not.toBe(0.4);
  });

  it("holds the last valid score when evidence is stale", () => {
    const decision = decidePublication({
      eligible: 100,
      completed: 99,
      rawAnswerRate: 1,
      citationTraceRate: 1,
      replaySuccessRate: 1,
      meanConfidence: 0.98,
      methodologyVersions: 1,
      ageHours: 48,
    });

    expect(decision.state).toBe("hold_last_valid");
  });
});

Other valuable counterfactual tests include:

• remove a detected brand mention and confirm the mention metric changes;
• mark a completed answer as failed and confirm it leaves performance scoring;
• replace a canonical URL while preserving the observed URL;
• mix two methodology versions and confirm publication stops;
• expire the evidence window and confirm the UI labels the last valid score.

Design the UI around reliability state

Do not hide policy output in a tooltip. The interface should show:

• the current publication state;
• every reason code;
• eligible, completed, failed, and excluded counts;
• the policy and methodology versions;
• evidence freshness;
• a link from the aggregate score to raw observations;
• the timestamp and age of any held score.

"Score: 64" and "Last valid score: 64, collected nine days ago; current run suppressed because 43% of raw answers are missing" are not equivalent messages.

Avoid three common mistakes

Failing open

If the policy service errors, the dashboard should not quietly publish. Choose a safe fallback such as holding the last valid score or suppressing the current one.

Hiding failures in the denominator

Keep scheduled, eligible, completed, failed, and excluded counts separate. A collection failure is an operational fact, not negative brand performance.

Repairing history with a fresh model response

A live replay may be useful, but it cannot replace what was actually observed. Store the original answer and the replay as separate events.

The operating principle

A visibility score earns trust when the system can explain both why it published the number and what failure would have stopped publication.

I expanded the measurement side of this idea into a practical falsification protocol:
https://www.linkedin.com/pulse/how-falsify-ai-visibility-score-before-you-trust-waleed-arshad-jqkdc/

Corank applies the same evidence-first principle to AI-search visibility and optimization:
https://corank.ai

The implementation can be small. The key is making the stop rule explicit, versioned, testable, and visible to the people who rely on the score.