Build a Prompt–Source Gap Triage Queue in TypeScript (Without a Vanity Score)

#ai #typescript #seo #webdev

A visibility dashboard can look precise while hiding the question that actually matters: what useful source is missing from this answer?

This tutorial builds a small, deterministic triage pipeline for AI-answer observations. It does not guess how a model works, scrape private interfaces, or collapse every signal into one “visibility score.” It accepts a frozen prompt set plus human-reviewed source roles and produces an editorial queue you can inspect.

The examples are synthetic. They do not represent measured rankings or customer results.

The unit of analysis

Keep one row per observed prompt run. A minimal input needs:

a stable prompt ID and version;
an intent class;
the answer system and timestamp;
whether the brand was mentioned;
whether the target domain was cited;
source roles that a reviewer observed;
a link to a permissioned evidence artifact.

Source roles describe the job a cited page performs:

definition — explains what an entity or category is;
evidence — supports a factual claim or measurement;
comparison — helps distinguish options;
implementation — helps the reader act;
identity — verifies canonical entity facts;
selection — supports a shortlist or recommendation;
context — adds background without carrying the central claim.

A role is not a quality judgment. Record support separately.

Define the schema

type Intent =
  | "discovery"
  | "diagnosis"
  | "comparison"
  | "implementation"
  | "selection"
  | "risk";

type SourceRole =
  | "definition"
  | "evidence"
  | "comparison"
  | "implementation"
  | "identity"
  | "selection"
  | "context";

type SupportStatus =
  | "supports"
  | "partially_supports"
  | "does_not_support"
  | "unclear";

interface SourceObservation {
  url: string;
  role: SourceRole;
  support: SupportStatus;
}

interface PromptObservation {
  promptId: string;
  promptVersion: string;
  promptText: string;
  intent: Intent;
  answerSystem: string;
  runTimestamp: string;
  brandMentioned: boolean;
  targetDomainCited: boolean;
  sources: SourceObservation[];
  evidenceArtifact: string;
  isSynthetic: boolean;
}

interface TriageItem {
  promptId: string;
  intent: Intent;
  missingRole: SourceRole;
  priority: number;
  reason: string;
}

Keep the answer system and timestamp visible. A repeated run is a sample, not a permanent ranking.

Map intent to expected source roles

The mapping is an editorial hypothesis. It should describe which roles would make an answer more useful, not which publisher the model must cite.

const expectedRoles: Record<Intent, SourceRole[]> = {
  discovery: ["definition", "identity"],
  diagnosis: ["evidence", "context"],
  comparison: ["comparison", "evidence"],
  implementation: ["implementation", "evidence"],
  selection: ["selection", "comparison", "identity"],
  risk: ["evidence", "context"],
};

Version this mapping. If you change it after seeing the results, your new output is not directly comparable with the old one.

Add transparent priority weights

A useful priority is not a claim about model probability. It is simply an ordered work queue. Keep the formula visible.

const intentWeight: Record<Intent, number> = {
  discovery: 2,
  diagnosis: 3,
  comparison: 4,
  implementation: 4,
  selection: 5,
  risk: 3,
};

const roleWeight: Record<SourceRole, number> = {
  definition: 2,
  evidence: 4,
  comparison: 4,
  implementation: 3,
  identity: 2,
  selection: 4,
  context: 1,
};

These values reflect editorial urgency in this example only. Do not present them as a validated industry benchmark.

Calculate missing roles

We count a role as present only when a reviewer found at least partial support. A citation that does not support its attached claim should not make the gap disappear.

function supportedRoles(row: PromptObservation): Set<SourceRole> {
  return new Set(
    row.sources
      .filter(
        (source) =>
          source.support === "supports" ||
          source.support === "partially_supports"
      )
      .map((source) => source.role)
  );
}

function triage(rows: PromptObservation[]): TriageItem[] {
  const queue: TriageItem[] = [];

  for (const row of rows) {
    if (!row.promptId || !row.promptVersion) {
      throw new Error("Every row needs a stable prompt ID and version");
    }

    if (!Number.isFinite(Date.parse(row.runTimestamp))) {
      throw new Error("Invalid timestamp for " + row.promptId);
    }

    const present = supportedRoles(row);

    for (const role of expectedRoles[row.intent]) {
      if (present.has(role)) continue;

      const citationGap = row.targetDomainCited ? 0 : 2;
      const mentionGap = row.brandMentioned ? 0 : 1;

      queue.push({
        promptId: row.promptId,
        intent: row.intent,
        missingRole: role,
        priority:
          intentWeight[row.intent] +
          roleWeight[role] +
          citationGap +
          mentionGap,
        reason: [
          "Expected " + role + " support for " + row.intent + " intent",
          row.brandMentioned ? "brand mentioned" : "brand not mentioned",
          row.targetDomainCited
            ? "target domain cited"
            : "target domain not cited",
        ].join("; "),
      });
    }
  }

  return queue.sort(
    (a, b) =>
      b.priority - a.priority ||
      a.promptId.localeCompare(b.promptId) ||
      a.missingRole.localeCompare(b.missingRole)
  );
}

The deterministic tie-breakers matter. Two people running the same input should receive the same ordering.

Test with synthetic observations

const observations: PromptObservation[] = [
  {
    promptId: "example-selection-01",
    promptVersion: "1.0",
    promptText:
      "What should a mid-market team evaluate in an AI visibility platform?",
    intent: "selection",
    answerSystem: "Example Answer System",
    runTimestamp: "2026-08-11T20:00:00Z",
    brandMentioned: true,
    targetDomainCited: false,
    sources: [
      {
        url: "https://example.org/category-overview",
        role: "identity",
        support: "supports",
      },
    ],
    evidenceArtifact: "https://example.org/evidence/example-selection-01",
    isSynthetic: true,
  },
  {
    promptId: "example-implementation-01",
    promptVersion: "1.0",
    promptText:
      "What evidence should an AI visibility audit preserve?",
    intent: "implementation",
    answerSystem: "Example Answer System",
    runTimestamp: "2026-08-11T20:05:00Z",
    brandMentioned: false,
    targetDomainCited: false,
    sources: [
      {
        url: "https://example.org/audit-method",
        role: "evidence",
        support: "partially_supports",
      },
    ],
    evidenceArtifact:
      "https://example.org/evidence/example-implementation-01",
    isSynthetic: true,
  },
];

console.table(triage(observations));

For the selection prompt, the queue will preserve the missing selection and comparison roles. The identity page confirms the entity; it does not independently justify a recommendation.

For the implementation prompt, evidence is present, but an actionable implementation source is still missing.

Convert a gap into an asset brief

A missing role is a research lead, not automatic permission to publish another article. Confirm the prompt genuinely calls for the role.

function assetBrief(item: TriageItem): string {
  const format: Record<SourceRole, string> = {
    definition: "concise definition page with scope and exclusions",
    evidence: "methodology or dataset with dates and limitations",
    comparison: "criteria-led comparison with explicit tradeoffs",
    implementation: "reproducible tutorial, template, or checklist",
    identity: "canonical entity profile with consistent facts",
    selection: "use-case matrix with qualification criteria",
    context: "background explainer tied to the answer claim",
  };

  return [
    "Prompt: " + item.promptId,
    "Role gap: " + item.missingRole,
    "Suggested format: " + format[item.missingRole],
    "Priority: " + item.priority,
    "Reason: " + item.reason,
  ].join("
");
}

The output is small enough for an editor to review. That is the point. Automation should make the judgment legible, not hide it.

Production guardrails

Before using this in a reporting workflow:

Freeze and version the prompt portfolio.
Use a real CSV parser instead of splitting arbitrary CSV on commas.
Validate timestamps, URLs, enum values, and duplicate run keys.
Keep synthetic rows physically or logically separate from observations.
Preserve answer-level evidence where terms and platform policy allow it.
Store the source claim and reviewer note, not only the role label.
Measure reviewer agreement for consequential analyses.
Re-run the same prompt set before interpreting change.
Keep mentions, citations, recommendations, support, and prominence separate.
Never describe a correlation as proof that one backlink caused an answer.

What this queue cannot tell you

This method cannot reveal hidden retrieval, training data, or causal attribution. It cannot guarantee that publishing a missing asset will change an answer. Interfaces, citations, personalization, geography, and model versions can change between runs.

It also cannot replace editorial judgment. A definition page may be the wrong asset for a selection question, and a vendor-authored comparison may not be independent enough to support a recommendation.

Use the queue to choose the next evidence question. Then publish the smallest truthful asset that answers it, distribute it where the intended audience already participates, and re-test with the frozen prompt set.

If you want a broader baseline before building the queue, Corank offers a free AI visibility audit across major answer-engine surfaces.