Test an AI Visibility Parser Against Missing Answers and Citation Links

Dmytro Nasyrov — Sat, 07 Nov 2026 08:55:06 +0000

A visibility report can label a brand absent because an answer collector returned nothing. The parser may have turned a missing string into an empty string, then treated the empty string as a valid answer. Once that record reaches a percentage, a collection defect starts looking like an observed result.

The executable example here rejects that substitution. It accepts a small, explicit envelope around an already saved answer. It keeps a textual mention separate from a source link and excludes failed collections from the observed-answer count. The final fixture passes 25 local tests. A deliberately broken version fails three of those tests, including the empty-observation rate check.

AI disclosure: An AI agent prepared this tutorial, code and synthetic fixtures and ran the local checks. It has not received human manuscript review. No live AI provider, customer system or production dataset was tested. The company affiliation is TopClick; the tutorial makes no claim about that company's production implementation.

Start with the decisions a record can support

The example uses a fictional product, Northstar Ledger, and the reserved domain northstar.example. Its observations are invented inputs to a local program. A citation means that a retained source URL has an explicitly approved hostname. A mention means that an approved literal alias appears in the saved answer text. Neither result establishes a recommendation, a visit or agreement with a claim.

Saved input	Mentioned	Cited	Eligible as an observed answer
Valid answer names Northstar Ledger, no sources	true	false	Yes
Valid answer links to an approved host without naming the product	false	true	Yes
Valid answer contains neither signal	false	false	Yes
Collection failed and includes an error code	unknown	unknown	No
Collected record lacks answer text or an explicit sources array	No observation created	No observation created	No

There are two layers in that table.

The first is input validity. The second is classification of a valid input. Returning false for a missing field crosses those layers without evidence. The fixture therefore throws on malformed collected records and gives collection failures their own status.

Mention and citation remain independent booleans.

A single combined label can hide a useful case: an answer may cite a documentation page while referring to the product only as an alternative. Requiring a brand name before accepting a source would discard that citation. Conversely, a paragraph containing a product name does not establish that the answer cited its website.

Define the adapter boundary before writing the parser

The parser receives a JavaScript object after a collection adapter has saved its evidence. It does not open a browser, extract HTML or call a model. The adapter must decide whether it successfully collected a complete answer and its source list. If extraction is incomplete, it must emit a failed attempt rather than guessing that the missing source list is empty.

Every attempt carries an ID, query ID, engine identifier, market and language. It also carries a capture timestamp and an artifact ID pointing to the retained evidence. These fields make the example replayable within a stated context. They do not independently prove that the context is truthful; a separate capture process must maintain that connection.

A collected envelope adds nonempty answerText and an explicit citations array. The array can be empty. Each citation has a source ID and a raw URL. A failed envelope instead supplies an errorCode and omits both answer fields. This deliberately strict distinction prevents a retry path from carrying an old answer alongside a new failure.

For this demonstration, timestamps use the canonical UTC form returned by Date.toISOString(), including milliseconds. That is an input contract chosen for a small fixture. Other valid timestamp representations would need conversion in the adapter. Rejecting an alternate representation here does not mean that representation is generally invalid.

The artifact ID is required as a string, but this offline program cannot establish that its artifact exists. Production validation should resolve that ID within the evidence store and check access and retention. A record can pass structural validation while its underlying capture is missing. Keep that storage check visible as a separate acceptance gate.

Save the parser as parser.mjs

The implementation has no package dependencies. The executed environment was Node.js v26.11.1. It uses the built-in test runner, strict assertions and the WHATWG URL implementation. The Node.js URL documentation describes the constructor and its handling of absolute and relative inputs. This example supplies no base because its adapter contract requires absolute citation URLs.

import { createHash } from 'node:crypto';

const fail = message => { throw new TypeError(message); };
const text = (value, name) => {
  if (typeof value !== 'string' || !value.trim()) fail(name);
  return value;
};
const object = value => value !== null && typeof value === 'object' && !Array.isArray(value);
const escape = value => value.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');

export const policy = Object.freeze({
  version: 'ledger-demo-v1',
  aliases: ['Northstar Ledger'],
  hosts: ['northstar.example'],
});

export function parse(record) {
  if (!object(record)) fail('record must be an object');
  for (const key of ['id', 'queryId', 'engine', 'market', 'language', 'artifactId']) {
    text(record[key], key);
  }
  text(record.observedAt, 'observedAt');
  const date = new Date(record.observedAt);
  if (!Number.isFinite(date.valueOf()) || date.toISOString() !== record.observedAt) {
    fail('observedAt must be canonical UTC ISO');
  }
  const evidence = Object.fromEntries(
    ['id', 'queryId', 'engine', 'market', 'language', 'artifactId', 'observedAt']
      .map(key => [key, record[key]])
  );
  const base = { ...evidence, parserVersion: 'answer-record-v1', policyVersion: policy.version };
  if (record.outcome === 'failed') {
    text(record.errorCode, 'errorCode');
    if ('answerText' in record || 'citations' in record) fail('failed record contains answer fields');
    return { ...base, status: 'collection_failed', mentioned: null, cited: null,
      errorCode: record.errorCode };
  }
  if (record.outcome !== 'collected') fail('unknown outcome');
  text(record.answerText, 'answerText');
  if (!Array.isArray(record.citations)) fail('citations must be explicit array');
  const ids = new Set();
  const citations = record.citations.map(citation => {
    if (!object(citation)) fail('citation must be an object');
    text(citation.id, 'citation.id');
    if (ids.has(citation.id)) fail('duplicate citation id');
    ids.add(citation.id);
    const rawHref = text(citation.href, 'citation.href');
    if (!/^https?:\/\//i.test(rawHref) || /[\s\\]/u.test(rawHref)) fail('citation URL format');
    const url = new URL(rawHref);
    if (!['http:', 'https:'].includes(url.protocol) || url.username || url.password) {
      fail('citation URL policy');
    }
    return { id: citation.id, recordId: record.id, rawHref, parsedHref: url.href,
      hostname: url.hostname, owned: policy.hosts.includes(url.hostname) };
  });
  const normalized = record.answerText.normalize('NFKC');
  const mentioned = policy.aliases.some(alias => new RegExp(
    `(^|[^\\p{L}\\p{N}_])${escape(alias.normalize('NFKC'))}(?=$|[^\\p{L}\\p{N}_])`, 'iu'
  ).test(normalized));
  return { ...base, status: 'observed', mentioned,
    cited: citations.some(citation => citation.owned),
    answerText: record.answerText,
    answerSha256: createHash('sha256').update(record.answerText, 'utf8').digest('hex'),
    citations };
}

export function summarize(records) {
  const ids = new Set();
  const result = { observed: 0, failed: 0, rejected: [], mentions: 0, citations: 0 };
  for (const record of records) {
    const id = object(record) && typeof record.id === 'string' && record.id.trim() ? record.id : null;
    if (id !== null) {
      if (ids.has(id)) throw new TypeError('duplicate record id');
      ids.add(id);
    }
    let observation;
    try { observation = parse(record); }
    catch (error) {
      result.rejected.push({ id, reason: error.message });
      continue;
    }
    if (observation.status === 'collection_failed') { result.failed++; continue; }
    result.observed++;
    result.mentions += Number(observation.mentioned);
    result.citations += Number(observation.cited);
  }
  return { ...result, citationRate: result.observed ? result.citations / result.observed : null };
}

The policy object is deliberately small. An approved alias and an exact hostname are enough to exercise the branches without pretending to solve entity resolution. The policy has a version separate from the parser version. Changing an alias or approved hostname can change the result even when the implementation stays identical. Preserve both versions when comparing replays.

The fixture's Object.freeze() protects the top-level object only. It does not deeply freeze the nested arrays. Treat the policy as a constant during this example. A deployed application should load an immutable, validated policy snapshot for a run, or defensively copy its configuration. Allowing another module to modify those arrays midway through a batch would undermine replay consistency.

Input validation rejects blank strings but preserves the originals.

Trimming every field would make a malformed identifier appear repaired without a record of that change. A production adapter may normalize identifiers under an explicit rule; that rule should run before this contract and retain its input. The parser itself should remain predictable when handed an unexpected representation.

Retain the link before deciding what it means

Each accepted citation retains its original rawHref, parsed representation and source record ID. The raw string remains available even when URL serialization lowercases a hostname or removes a default port. That distinction matters during debugging: an engineer can inspect the captured link and the representation used for comparison without reconstructing either from a report. The fixture permits HTTP and HTTPS and rejects credentials in the authority. It also rejects whitespace, backslashes and relative references under its deliberately narrow input contract. These are parser acceptance rules. They do not establish that a URL is safe to fetch. The program performs no requests and makes no claim about the destination's availability or content.

Hostname ownership uses exact membership in policy.hosts. A substring check could accept northstar.example.evil.example, which belongs to a different hostname. Automatic suffix matching could also accept a subdomain that the policy never reviewed. This implementation leaves subdomains unapproved until they are listed explicitly. A false negative caused by a narrow allowlist is visible and can be corrected with evidence. A URL passing this check supports only the statement that its hostname matches this policy. It does not prove corporate ownership, editorial authorship or control of the cited page. Establishing an approved-host list requires separate evidence. An organization may publish on a shared platform, use a regional domain or outsource documentation. Those cases need a documented mapping rather than a permissive string comparison.

The parser preserves query parameters and fragments.

It does not follow redirects, inspect canonical tags or group source variants. Those are different operations with their own evidence requirements. Perform any later grouping against retained raw links, with a versioned rule and a reversible mapping. Rewriting a source during classification would make a later disagreement harder to inspect.

If these links are eventually displayed in an application, render them with an appropriate safe-link component and contextual escaping. Validation here is not a template sanitizer. Passing the fixture must never become permission to insert answer text or URLs directly into HTML. The record parser and the presentation layer have different responsibilities.

Treat a literal mention as a bounded signal

The matching rule normalizes a copy of the answer with NFKC, then performs a case-insensitive literal alias match with Unicode letter and number boundaries. The original answer remains unchanged for retention and hashing. In the test, uppercase text and a fullwidth spelling match the declared alias, while a longer token containing that alias does not.

That rule has useful limits.

It does not understand sentiment or resolve company ambiguity. A sentence saying that Northstar Ledger is unsuitable still contains a mention. A common phrase that happens to equal an alias may also match. The parser returns a lexical observation; a recommendation classifier would need a different contract and separately tested evidence.

Aliases should be reviewed as data. A short abbreviation can create many more accidental matches than a full product name. Adding translations, alternate punctuation or historical names changes the observation population. Keep those changes in a new policy version and preserve the answer text so that the old and new rules can be replayed against the same captures.

This fixture has one multiword English alias. Its boundary expression is not a general segmentation solution for every language or writing system. Before adopting it elsewhere, add examples from the intended scripts and inspect the expected outcomes with someone who understands those names. Passing the supplied English cases says nothing about languages absent from the fixture.

Save the tests beside it as parser.test.mjs

The tests exercise observable decisions rather than calling internal helpers. They check the state table, missing fields, unsafe or unresolved links and hostname lookalikes. They also check preserved evidence and the aggregate result. Keeping assertions at this boundary lets the implementation change while the acceptance contract remains readable.

The Node.js documentation describes one supported test form as "A synchronous function that is considered failing if it throws an exception, and is considered passing otherwise." This is a statement from the Node.js test runner documentation. The assertions below throw when the actual value differs from the stated contract. An exception inside an expected rejection check is handled by the assertion itself.

import test from 'node:test';
import assert from 'node:assert/strict';
import { createHash } from 'node:crypto';
import { parse, summarize } from './parser.mjs';

const saved = overrides => ({
  id: 'a1', queryId: 'q1', engine: 'offline-fixture', market: 'US', language: 'en',
  artifactId: 'synthetic-answer-a1', observedAt: '2026-11-06T12:00:00.000Z',
  outcome: 'collected', answerText: 'Northstar Ledger supports exports.', citations: [],
  ...overrides,
});
const failed = id => {
  const record = saved({ id, outcome: 'failed', errorCode: 'TIMEOUT' });
  delete record.answerText; delete record.citations;
  return record;
};
const link = href => [{ id: 's1', href }];

test('mention without citation', () => {
  const result = parse(saved());
  assert.equal(result.mentioned, true); assert.equal(result.cited, false);
});
test('citation without mention', () => {
  const result = parse(saved({ answerText: 'This source describes exports.',
    citations: link('https://northstar.example/docs') }));
  assert.equal(result.mentioned, false); assert.equal(result.cited, true);
});
test('no mention and no citation is observed', () => {
  const result = parse(saved({ answerText: 'Compare two alternatives.' }));
  assert.equal(result.status, 'observed'); assert.equal(result.mentioned, false);
  assert.equal(result.cited, false);
});
test('failed collection has unknown booleans', () => {
  const result = parse(failed('f1'));
  assert.equal(result.status, 'collection_failed');
  assert.equal(result.mentioned, null); assert.equal(result.cited, null);
});
for (const [name, change] of [
  ['missing answer', r => { delete r.answerText; }],
  ['empty answer', r => { r.answerText = ' '; }],
  ['missing citations', r => { delete r.citations; }],
  ['null citations', r => { r.citations = null; }],
  ['missing artifact', r => { delete r.artifactId; }],
  ['bad timestamp', r => { r.observedAt = '2026-02-30T12:00:00.000Z'; }],
  ['unknown outcome', r => { r.outcome = 'partial'; }],
]) test(`reject ${name}`, () => {
  const record = saved(); change(record); assert.throws(() => parse(record), TypeError);
});
test('reject failed record with answer fields', () => {
  assert.throws(() => parse({ ...failed('f1'), answerText: 'stale answer' }), TypeError);
});
for (const href of ['javascript:alert(1)', '/docs', 'https://user:pass@northstar.example/',
  'https://northstar.example/a b', 'https:\\northstar.example']) {
  test(`reject unsafe or unresolved link ${href}`, () => {
    assert.throws(() => parse(saved({ citations: link(href) })), TypeError);
  });
}
test('lookalike and unapproved subdomain are not owned', () => {
  for (const href of ['https://northstar.example.evil.example/',
    'https://docs.northstar.example/', 'https://other.example/northstar.example']) {
    assert.equal(parse(saved({ citations: link(href) })).cited, false);
  }
});
test('retain original URL and source record', () => {
  const href = 'https://NORTHSTAR.example:443/docs?x=2&x=1#export';
  const citation = parse(saved({ citations: link(href) })).citations[0];
  assert.equal(citation.rawHref, href); assert.equal(citation.recordId, 'a1');
  assert.equal(citation.parsedHref, 'https://northstar.example/docs?x=2&x=1#export');
});
test('reject duplicate citation ids', () => {
  const citations = [...link('https://northstar.example/a'), ...link('https://other.example/b')];
  assert.throws(() => parse(saved({ citations })), /duplicate citation id/);
});
test('literal alias respects boundaries and NFKC', () => {
  assert.equal(parse(saved({ answerText: 'XNORTHSTAR LEDGERX' })).mentioned, false);
  assert.equal(parse(saved({ answerText: '(NORTHSTAR LEDGER)' })).mentioned, true);
  assert.equal(parse(saved({ answerText: 'Ｎｏｒｔｈｓｔａｒ Ledger' })).mentioned, true);
});
test('hash covers exact answer text', () => {
  const record = saved(); const result = parse(record);
  assert.equal(result.answerSha256, createHash('sha256').update(record.answerText).digest('hex'));
  assert.notEqual(parse(saved({ answerText: record.answerText + ' ' })).answerSha256, result.answerSha256);
});
test('batch excludes failures and rejects', () => {
  const result = summarize([saved({ citations: link('https://northstar.example/') }),
    saved({ id: 'a2', answerText: 'Another provider.' }), failed('f1'),
    saved({ id: 'bad', citations: null })]);
  assert.equal(result.observed, 2); assert.equal(result.failed, 1);
  assert.equal(result.rejected.length, 1); assert.equal(result.citationRate, 0.5);
});
test('empty observed set has undefined rate', () => {
  assert.equal(summarize([failed('f1')]).citationRate, null);
});
test('duplicate attempt aborts aggregation', () => {
  assert.throws(() => summarize([saved(), saved()]), /duplicate record id/);
});

Run the files in the same directory with node --test parser.test.mjs. There is no install step and no external API key. The fictional hostname is parsed as text; it is never resolved. The strict assertion documentation explains the assertion interface used by the fixture. An unsupported local runtime should be investigated before interpreting a failure as a parser defect.

The main batch contains four attempts: one observed answer with an approved citation, one observed answer with neither signal, one failed collection and one malformed collected record. Its expected observed count is two. Its failed count is one, and its rejection list contains one entry. One cited observation divided by two observed answers produces 0.5 for this synthetic input.

A rejected record is retained in a separate list with its ID when available and a reason.

This is a quarantine result, not an observed absence. In a real pipeline, retain the full original input in an access-controlled error store as well. The short diagnostic returned here is enough for a unit test but insufficient for investigating a damaged capture days later.

What the local execution established

The completed implementation passed all 25 supplied tests with no skipped tests. To check whether the assertions catch the intended category mistake, a second local copy deliberately introduced two defects. It changed failure booleans from unknown to false and incremented the observed count before checking for a failed collection. The test file itself remained unchanged.

Local variant	Passed	Failed	Meaning
Final parser	25	0	Supplied acceptance examples pass
Deliberately broken copy	22	3	Failure-state and denominator regressions are detected

The three failures cover unknown booleans on collection failure, the mixed-batch citation rate and the rate when no observed answer exists. This is a small mutation check. It shows that those assertions can distinguish these two implementations. It does not establish coverage of every possible defect, prove production reliability or validate the behavior of an AI provider.

An all-green run by itself can be misleading when tests merely repeat the implementation's assumptions. The broken copy makes one assumption observable: a failed attempt must not become an observed negative. Add other mutations when they address a concrete acceptance concern, such as accepting a hostname lookalike. Keep those experiments bounded and preserve their expected failures separately from the release test run.

Execution time is not a performance result.

The fixture is tiny, and wall time includes test-runner overhead. No throughput, latency or memory claim follows from these checks. Benchmarking a production ingestion path would require representative record sizes, expected concurrency and a named operating environment, none of which this demonstration supplies.

Use the hash for replay integrity

The parser computes a SHA-256 digest over the exact UTF-8 answer text. Adding a trailing space changes that digest in the test. Normalized matching text does not replace the retained answer or its checksum. The Node.js crypto documentation describes the hash interface used for this operation.

A checksum can help detect that the text supplied to a replay differs from the text previously recorded. It cannot authenticate an answer. Anyone able to replace both the answer and its checksum can produce a consistent pair. If capture authenticity matters, preserve the original response or browser artifact under a storage policy with stronger provenance controls. The answer digest also leaves important context outside its scope.

It does not cover citation arrays, capture metadata or the policy snapshot. A production evidence manifest may hash those objects too, with a declared serialization method. Avoid calling this field a complete record signature. Its name deliberately identifies the narrower object it covers.

For debugging, keep a run manifest that names the parser version and policy version alongside artifact IDs and exact inputs. Replaying a capture through a new parser should produce a new derived result, preserving the earlier one. Overwriting the first classification would make it difficult to distinguish changed code from a changed answer.

Keep retry identity and aggregate eligibility explicit

The batch function rejects duplicate nonblank attempt IDs before producing a result.

Silently accepting a duplicate could double-count an answer after a queue redelivery. Silently dropping it could also hide two conflicting envelopes under the same ID. Aborting gives the caller a visible integrity problem to resolve before reporting a percentage.

A retry should receive its own attempt identity and preserve its relationship to the original query. This fixture does not choose which attempt represents a query in a reporting period. Counting every retry would change the question being measured. Define that selection policy upstream and retain the discarded attempts for operational diagnosis rather than treating them as observed answers. When there are no observed answers, the function returns null for citationRate. Zero would imply a defined denominator with no citations. Here the denominator is absent. An interface can display an unavailable state and the separate collection-failure count. Converting null to zero during chart preparation would reintroduce the same category error after the parser had avoided it.

The returned rejected list also deserves a reporting threshold. A batch can have a mathematically defined rate and still be unfit for comparison because too many inputs were malformed. This sample leaves that acceptance threshold to the caller. A useful deployment gate might block a release on any unexpected schema rejection while the adapter contract is being stabilized.

Decide what remains outside this envelope

The parser rejects an entire collected record if any citation violates its contract. That choice favors a visible rejection over a partial answer classification. It avoids silently dropping a damaged source that might have been the only relevant citation. If a production pipeline instead accepts partial source extraction, give that condition its own status and keep it outside metrics that require a complete source list.

Unknown fields are allowed, and field lengths are not bounded in this demonstration.

That makes it easier to add explanatory metadata to local fixtures, but it leaves ingestion controls unfinished. A network-facing endpoint should enforce payload size and field limits before parsing. It may also need a closed schema to catch misspelled keys rather than accepting them as unused extensions.

The timestamp check validates representation and calendar consistency. It does not reject future captures or establish that an answer belongs to a reporting window. Likewise, the required market and language fields are nonblank strings without a controlled vocabulary. Add those domain checks at the point where the application knows the authorized collection scope, and test rejection reasons independently.

The batch catches parser exceptions and records their messages as rejections. That keeps this small example easy to run. A service should distinguish expected validation errors from unexpected implementation failures so that a programming bug cannot quietly become a large quarantine count. A named validation-error class and an explicit rethrow path would make that separation clearer.

Finally, this object is an internal data contract rather than a proof that every possible JSON shape is safe or meaningful. The supplied cases cover the named acceptance decisions. Before accepting untrusted requests, test the actual JSON decoder, request limits and error-handling path used by the application. Keep those integration checks alongside the parser fixture instead of claiming that a local function test has exercised the full ingestion service.

Connect the fixture to a real collector carefully

Start by exporting retained records from a small known collection, with any private data removed from the test copy. Map the provider-specific fields into the documented envelope and record the adapter version. Compare the exported citations with the saved artifact before declaring an empty array authoritative. A successfully loaded answer panel does not establish that every source control was extracted.

Keep the adapter's capture failure separate from the parser's rejection. A timeout is a collection outcome. An object claiming successful collection while omitting answerText violates the envelope. Those problems have different owners and repair paths. Combining them into one generic error count makes it harder to decide whether to fix browser collection or the transformation code.

Add regression inputs when a real response shape exposes a new failure. Preserve the smallest example that demonstrates the defect and its original provenance in the appropriate protected store. The public test fixture can use a redacted or synthetic equivalent, clearly labeled. Do not present a synthetic reproduction as the original capture or as a customer result.

Before enabling reporting, replay the same retained batch through the current and proposed parser versions. Review differences at record level before reviewing the aggregate percentage. An unchanged rate can conceal offsetting classification errors. The useful handover is a list of changed attempt IDs, their old and new states and the evidence needed to explain each change.

Release the parser when its acceptance examples, adapter mapping and evidence-store checks agree. Keep the unchanged inputs available for the next replay. The supplied files give that review a small executable starting point: an absent answer cannot quietly become a negative visibility observation, and a source link remains attached to the record from which it came.

About the author

Dmytro Nasyrov. Photo supplied by the author.

Written by Dmytro Nasyrov PhD, software architect with 24 years of production experience. Dmytro is the CEO of TopClick.

DEV Community: TopClick