You've measured share of voice and found a prompt cluster where a competitor keeps getting recommended and cited, and you don't. The obvious next question is what's different about their page?
Most people answer that by eyeballing two browser tabs. This post does it with a script: take the URLs AI engines cited for a prompt, fetch them the way a non-JavaScript crawler would, extract the features that tend to matter for extraction and citation, and print a diff against your page.
It won't explain everything. Brand reputation and third-party mentions matter too, and they don't show up in page HTML. But it reliably catches the boring structural gaps, which are the fastest ones to fix.
Step 1: Get the cited URLs
Start from stored citations (see the earlier post on parsing them) for the prompts where you lose:
import sqlite3, json
from collections import Counter
conn = sqlite3.connect("visibility.db")
rows = conn.execute("""
SELECT r.sources FROM run r
JOIN prompt p ON p.id = r.prompt_id
WHERE p.topic = ? AND r.run_date >= date('now','-28 days')
""", ("analytics-pricing",)).fetchall()
counts = Counter(u for (s,) in rows for u in json.loads(s))
competitor_urls = [u for u, n in counts.most_common(10) if "acmeanalytics.com" not in u]
Pick the top three to five cited URLs. Pages cited across many runs and engines are the signal. A URL cited once is noise.
Step 2: Fetch like a simple crawler
Fetch raw HTML with no JavaScript rendering. If a page's content needs JS to appear, that's itself a finding:
import requests
UA = "Mozilla/5.0 (compatible; page-diff/0.1; +https://example.com/bot)"
def fetch(url: str) -> str:
r = requests.get(url, headers={"User-Agent": UA}, timeout=20)
r.raise_for_status()
return r.text
Use an honest user agent and keep the volume tiny. You're reading a handful of public pages, not crawling a site.
Step 3: Extract features that matter for AI extraction
Retrieval systems split pages into passages, embed them and pull the best-matching chunks into the model's context. So the features worth measuring are the ones that affect whether a clean, self-contained answer passage exists and whether the facts are machine-readable.
import re, json
from bs4 import BeautifulSoup
def features(html: str) -> dict:
soup = BeautifulSoup(html, "html.parser")
ld_types = []
for tag in soup.find_all("script", type="application/ld+json"):
try:
data = json.loads(tag.string or "")
except json.JSONDecodeError:
ld_types.append("INVALID")
continue
items = data if isinstance(data, list) else data.get("@graph", [data])
for it in items:
t = it.get("@type")
ld_types.extend(t if isinstance(t, list) else [t])
for t in soup(["script", "style", "noscript", "nav", "footer", "header"]):
t.decompose()
main = soup.find("main") or soup.find("article") or soup.body or soup
text = main.get_text(" ", strip=True)
words = len(text.split())
headings = [h.get_text(" ", strip=True) for h in main.find_all(["h2", "h3"])]
question_heads = sum(1 for h in headings if h.endswith("?") or
re.match(r"(?i)^(how|what|why|when|which|is|can|does|should)\b", h))
# words before the first paragraph under each H2: proxy for "answer first"
first_para_lens = []
for h in main.find_all("h2"):
p = h.find_next("p")
if p:
first_para_lens.append(len(p.get_text(" ", strip=True).split()))
return {
"words": words,
"h2_h3": len(headings),
"question_headings": question_heads,
"median_first_para_words": sorted(first_para_lens)[len(first_para_lens)//2] if first_para_lens else None,
"tables": len(main.find_all("table")),
"lists": len(main.find_all(["ul", "ol"])),
"numbers": len(re.findall(r"\b\d[\d,.]*\s?(%|\$|ms|x|users|customers)?", text)),
"jsonld_types": sorted(set(filter(None, ld_types))),
"has_author": bool(soup.select_one('[rel=author], .author, [itemprop=author]')) or "Person" in ld_types,
"has_date": bool(soup.find("time")) or bool(re.search(r'"date(Published|Modified)"', html)),
"outbound_links": len([a for a in main.find_all("a", href=True) if a["href"].startswith("http")]),
}
Each feature is a proxy, and some are crude (numbers will count list markers in some layouts). That's acceptable here. We're comparing pages against each other, not scoring them in absolute terms.
Step 4: Diff
import pandas as pd
ours = "https://acmeanalytics.com/pricing"
table = {url: features(fetch(url)) for url in [ours, *competitor_urls[:4]]}
df = pd.DataFrame(table).T
df.index = ["OURS"] + [f"C{i}" for i in range(1, len(df))]
print(df.to_string())
words h2_h3 question_headings median_first_para_words tables lists numbers jsonld_types has_author has_date
OURS 410 4 0 62 0 2 11 [Organization] False False
C1 1840 14 9 28 2 6 74 [FAQPage, Organization, Product] True True
C2 2210 17 6 31 1 8 91 [Article, BreadcrumbList] True True
C3 1390 11 7 24 3 4 58 [Product, SoftwareApplication] False True
(Illustrative output.) A diff like this tends to make the problem obvious:
- Depth: the cited pages are 3–5× longer, and that isn't padding. They have 11–17 sections.
- Question headings: competitors title sections the way people ask. The other page has none.
- Answer first: the competitors open each section with a short (~25–30 word) direct answer. The other page opens with ~60 words of preamble.
- Specifics: many more concrete numbers, plus comparison tables.
-
Machine-readable facts:
Product/FAQPageschema, an author and a date.
Step 5: Check what's cited, not just what exists
Features tell you what a page has. The cited passages tell you which part the engine used. If you stored cited_text with your citations (here, in a citation table of run_id, url, cited_text rows produced by the normaliser from the citations post), line them up against the competitor page:
for url in competitor_urls[:3]:
print("\n", url)
for (s,) in conn.execute(
"SELECT cited_text FROM citation WHERE url = ? AND cited_text IS NOT NULL LIMIT 5", (url,)):
print(" -", s[:160])
You'll usually find the engine quoting the same two or three passages: a pricing sentence with a concrete number, a one-line definition, a row of a comparison table. Those are the passages to match or beat, with your own facts.
What the diff can't see
Be honest about what's out of scope:
- Off-site reputation. Reviews, forum threads, publications and lists where competitors are mentioned. In model-knowledge-heavy engines, these often matter more than on-page features.
- JavaScript-rendered content. If your page renders client-side, the raw fetch shows less than a browser does. That's informative (many AI fetchers don't render JS), but compare against a rendered version before drawing conclusions.
- Freshness and authority signals that the engines don't disclose.
So treat the output as a list of hypotheses. Ship the fixes, then check whether citation rate for that prompt cluster moves over the following weeks.
Automating the loop
Running this by hand works for one prompt cluster. Across dozens of clusters and several competitors, you want it running continuously.
Vista AI's Competitor Analysis handles the tracking side: it lists which competitor pages get cited most, where competitors beat you per engine, and surfaces content gaps and quick wins. For the page-level side, Vista's Page Audit scores your page and competitor pages side by side on AI visibility, understanding and citation potential, which is a more complete version of the feature diff above.
Whatever you use, the loop is the same: find the prompts you lose → read what gets cited instead → close the specific gap → re-measure.
Top comments (0)