DEV Community: snagheelee

Compare saved citation excerpts before changing an AI search report

snagheelee — Mon, 09 Nov 2026 08:47:55 +0000

A citation can point to the same URL on two dates even though the paragraph behind it has changed. Save the passage you actually used and compare it with the next copy. That helps an editor distinguish a change in the source from a change in the AI answer.

This post was prepared with AI assistance for TheLogic. The example is fictional. The script runs on Python 3.10+ using the standard library.

Save two versions of one passage

Copy the heading and relevant sentences into UTF-8 files. Keep the final URL and capture time in an observation record. Use one logical sentence per line. Check that both captures came from the same product tier, language, and URL.

before.txt

Reporting exports
Workspace owners can export completion reports.
Exports are available for 7 calendar days after cancellation.

after.txt

Reporting exports
Workspace owners can export completion reports.
Exports are available for 7 business days after cancellation.

Compare the saved text

The script normalizes Unicode and runs of whitespace inside each line. It keeps case, punctuation, digits, and wording intact because those can change a policy’s meaning.

import argparse
import unicodedata
from difflib import unified_diff
from pathlib import Path


def normalized_lines(text):
    return [
        ' '.join(unicodedata.normalize('NFC', line).split())
        for line in text.splitlines()
        if line.strip()
    ]


def compare(before, after):
    return list(unified_diff(
        normalized_lines(before),
        normalized_lines(after),
        fromfile='earlier capture',
        tofile='later capture',
        lineterm='',
    ))


if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument('before', type=Path)
    parser.add_argument('after', type=Path)
    args = parser.parse_args()
    diff = compare(
        args.before.read_text(encoding='utf-8'),
        args.after.read_text(encoding='utf-8'),
    )
    print('\n'.join(diff) if diff else 'No differences in the saved passage.')

Save the script as compare_citations.py and run python compare_citations.py before.txt after.txt.

--- earlier capture
+++ later capture
@@ -1,3 +1,3 @@
 Reporting exports
 Workspace owners can export completion reports.
-Exports are available for 7 calendar days after cancellation.
+Exports are available for 7 business days after cancellation.

Decide what the difference supports

Result	Editorial action
Wording changed	Check the current source and effective date before revising the report.
AI answer changed while the passage stayed the same	Check the other cited pages and log the new answer separately.
Only whitespace changed	Keep both capture dates and leave the claim unchanged.
A source is unavailable	Mark the new observation as unverified and keep the earlier capture.

A text diff shows what changed between two saved excerpts. Check the effective date separately to establish which wording applies today.

FAQ

Does this fetch pages or ChatGPT answers? It compares files you already saved. Copy the relevant passage from the original citation and keep the source address in your observation log.

Can I compare different URLs? Yes. Label them as separate sources. A wording difference alone does not establish that either page was updated.

Where are the Python details? See the Python documentation for unified diffs. TheLogic’s public GEO methodology shows how the team separates brand mentions and source records.

Count AI answer mentions and official citations separately with Python

snagheelee — Mon, 09 Nov 2026 08:34:14 +0000

A brand can appear in an AI answer while the citations point entirely to other sites. A URL can also appear among the sources even when the answer does not name the brand. When auditing AI search, store both signals and keep collection errors out of the denominator.

This article was prepared by TheLogic with AI assistance. The data below is synthetic, and the script works locally with Python 3.10+ and no third-party packages.

Define the input

Use one JSON object per line. Here, answer is the visible answer text, with citation URLs extracted separately into citations. An error row represents a collection failure. A completed answer with no links uses an empty array. Use a separate file per question, platform, market, and observation period if those settings differ.

{"status":"ok","answer":"Harbor Ledger supports branch exports.","citations":["https://directory.example/harbor-ledger"]}
{"status":"ok","answer":"The platform supports branch exports.","citations":["https://docs.harborledger.example/export"]}
{"status":"ok","answer":"Harbor Ledger supports CSV exports.","citations":["https://harborledger.example/docs/csv"]}
{"status":"ok","answer":"Harbor Ledger appears in this directory.","citations":["https://harborledger.example.attacker.example/"]}
{"status":"error"}

Run a small audit

The hostname check compares DNS labels, so a URL that merely starts with the official hostname does not pass. The name check uses one exact name with case-insensitive word boundaries. A production workflow can maintain reviewed aliases and flag ambiguous names for human review.

import json
import re
import sys
from urllib.parse import urlsplit


def official_url(url, domain):
    try:
        parsed = urlsplit(url)
        host = (parsed.hostname or '').rstrip('.').lower()
    except ValueError:
        return False
    return parsed.scheme in {'http', 'https'} and (
        host == domain or host.endswith('.' + domain)
    )


def summarize(records, brand, domain):
    pattern = re.compile(r'(?<!\w)' + re.escape(brand) + r'(?!\w)', re.IGNORECASE)
    totals = dict(completed=0, errors=0, mentioned=0, cited=0, both=0)
    for row in records:
        if row['status'] != 'ok':
            totals['errors'] += 1
            continue
        mentioned = bool(pattern.search(row['answer']))
        cited = any(official_url(url, domain) for url in row['citations'])
        totals['completed'] += 1
        totals['mentioned'] += mentioned
        totals['cited'] += cited
        totals['both'] += mentioned and cited
    return totals


if __name__ == '__main__':
    with open(sys.argv[1], encoding='utf-8') as source:
        rows = (json.loads(line) for line in source if line.strip())
        print(summarize(rows, 'Harbor Ledger', 'harborledger.example'))

Save the fixture as observations.jsonl and the script as audit.py, then run python audit.py observations.jsonl.

Observation	Count	Denominator
Completed answers	4	5 attempted
Brand mentioned	3	4 completed
Official URL cited	2	4 completed
Both signals	1	4 completed
Collection errors	1	5 attempted

Keep the answer, its citations, and the observation conditions together. A total without those inputs is difficult to audit.

Interpret the output

The fourth answer mentions the brand, but its source is not on the official domain. The second answer cites an official subdomain without naming the brand. A matching domain also does not show that the source supports a specific claim: open the page and compare the relevant passage manually.

FAQ

Will this collect ChatGPT answers? It summarizes records you already have. ChatGPT’s web search can show citations and a Sources panel. Save the source destinations from the interface if available. OpenAI’s guide to ChatGPT Search

Should redirects count as official citations? Record the final destination separately after checking the link. This local example does not resolve redirects, so it only classifies the URL in the record.

Can I use a brand’s total mentions across unrelated questions? Keep the question set, platform, language, market, and time window in the report. Compare runs with equivalent settings before drawing a trend.

TheLogic uses the distinction between mentions and citations in its Korean-language GEO workflow. See its public GEO methodology.