Diffing AI Answer Snapshots Over Time With Python

#python #ai #seo #marketing

If you track how AI assistants describe your brand, you quickly collect a folder of saved answers: one text file per question per week. Reading them all is slow. A small diff script shows what changed, which is usually the only thing worth reviewing.

Folder layout

snapshots/
  2026-10-17/q01-best-agency-hk.txt
  2026-10-24/q01-best-agency-hk.txt
  2026-10-31/q01-best-agency-hk.txt

Each file holds the plain-text answer you saved for that question that week, collected manually or through whatever tool your team is allowed to use.

The script

import difflib, pathlib, re, sys

BRAND = re.compile(r"pandawm|熊猫出海", re.I)
root = pathlib.Path("snapshots")
weeks = sorted(p.name for p in root.iterdir() if p.is_dir())
prev, curr = weeks[-2], weeks[-1]

def norm(text):
    text = re.sub(r"\s+", " ", text)
    return [s.strip() for s in re.split(r"(?<=[.!?。！？])\s*", text) if s.strip()]

for f in sorted((root / curr).glob("*.txt")):
    old_f = root / prev / f.name
    old = norm(old_f.read_text(encoding="utf-8")) if old_f.exists() else []
    new = norm(f.read_text(encoding="utf-8"))
    was, now = any(BRAND.search(s) for s in old), any(BRAND.search(s) for s in new)
    changed = [l for l in difflib.unified_diff(old, new, lineterm="", n=0)
               if l[:1] in "+-" and not l.startswith(("+++", "---"))]
    flag = "GAINED" if now and not was else "LOST" if was and not now else ""
    print(f"\n## {f.stem}  {flag}  ({len(changed)} changed sentences)")
    for l in changed[:12]:
        print(l)

Splitting into sentences before diffing matters. Assistants rephrase constantly, and a character-level diff would mark everything as changed.

Reading the output

GAINED / LOST flags are the headline. Review those first.
A high number of changed sentences with no brand change usually means the assistant restructured its answer, not that anything meaningful happened.
Check whether new sentences introduce a competitor, a wrong fact or a new source.

Limits

Answers vary from run to run even on the same day. Treat one week's change as a hint, and look for patterns across several weeks before acting. Save the date, the tool and the exact prompt with every snapshot so comparisons stay fair.

Connecting it to results

Visibility only matters if it turns into visits and leads. Pair this with referrer tracking and CRM fields so you can see whether a gained mention coincides with more AI-referred traffic; this attributing AI search leads with UTM and CRM walks through that setup.

Published by PandaWM, an overseas marketing team based in Hong Kong.

DEV Community