Don't Count Failed AI Visibility Checks as Brand Misses

Cover: AI-generated editorial illustration.

An AI-answer tracker schedules eight checks. Five finish, two fail, and one is still pending. Two of the completed answers mention the configured brand. What percentage should the dashboard display?

This is a synthetic example. It exposes a reporting bug that is easy to introduce: dividing mentions by every row, including rows that never produced a usable answer.

Two out of eight is 25%. But that number treats three missing results as if they were completed answers that omitted the brand. Two out of five is 40% of the completed, classified answers. That still needs a coverage label: five of eight checks completed.

Separate collection state from answer classification

A request failing tells you something about the collection process. It does not tell you whether the brand would have appeared. Keep that state separate from the result of inspecting a successful answer.

There is another unknown to preserve. A request may complete while classification is unavailable or inconclusive. That answer should not silently become a negative result either.

For this example, completed means an answer was collected successfully. The mentioned field is true, false, or unclassified. Mentioned means only that the configured brand was identified in that answer; it does not mean recommended or cited.

Aggregate only the records that support the metric

The function below expects validated input with one final record per logical check. Resolve retries and duplicate records before calling it. Its recorded count is the number of rows supplied, not an independently verified schedule size.

export function summarize(rows) {
  const completed = rows.filter(r => r.status === 'completed');
  const classified = completed.filter(r =>
    typeof r.mentioned === 'boolean'
  );
  const mentions = classified.filter(r => r.mentioned).length;
  return {
    recorded: rows.length,
    completed: completed.length,
    classified: classified.length,
    mentions,
    misses: classified.length - mentions,
    unclassified: completed.length - classified.length,
    failed: rows.filter(r => r.status === 'failed').length,
    pending: rows.filter(r => r.status === 'pending').length,
    mentionRate: classified.length
      ? mentions / classified.length
      : null
  };
}

The boolean guard is intentional. A missing field, null, or the string false cannot supply a trustworthy negative classification. With no classified answers, the rate is unavailable rather than zero.

Show the denominator beside the percentage

For the eight-record example, the summary is: 8 recorded, 5 completed, 5 classified, 2 mentions, 3 misses, 2 failed, 1 pending, and a mention rate of 0.4.

A useful label is “2 of 5 classified answers mentioned the brand.” Put “5 of 8 checks completed” beside it. If some completed answers are unclassified, show that count too.

For a separately configured schedule, compare the collected records with the expected logical check IDs. A record missing from the database is not automatically a pending check, and this function does not discover missing scheduled jobs.

Test the cases that can make a chart lie

The included function was run against the eight-record example and three boundary cases: an empty collection returns an unavailable rate; a failed row carrying mentioned: true does not increase mentions; and a completed row carrying mentioned: null remains unclassified with an unavailable rate.

Also verify retry handling in the system that produces these records. Counting three attempts at the same question as three separate scheduled observations changes the denominator before the reporting function ever runs.

Keep interpretation separate from arithmetic

A correct percentage does not describe every person's AI experience. It describes the answers in the selected collection. Keep the question set, provider, dates and known request conditions available so a reader can judge what was measured.

If a later run has lower completion or a different question set, display that context before presenting the change as a visibility trend. Fixing an aggregation bug improves the report; it does not change the underlying answers.

AI disclosure: This article was drafted with an AI assistant. The examples use synthetic data and were checked by running the included JavaScript. They are not production measurements or customer results.