When Perplexity, ChatGPT search or Google's AI features answer a question, they don't read your page like a person. A retrieval pipeline splits it into passages, ranks those passages against the query, and hands a few of them to the model. The model writes its answer from those passages.
So the unit that wins or loses citations is the passage, not the page. A page can be excellent overall and still lose if its best answer is split across three chunks, or starts with "As mentioned above, it…".
Nobody outside these companies knows their exact chunking. But the general mechanics are well understood from open-source RAG systems, and you can simulate them well enough to find weak passages. That's what this post does.
How a typical retriever sees your page
A common pipeline:
- Fetch HTML, strip navigation and boilerplate.
- Split by structure (headings), then by size (a few hundred tokens), with some overlap.
- Embed each chunk, and often score it with keyword search (BM25) as well.
- Rerank the top candidates against the query.
- Pass the top few chunks to the model with their source URLs.
Steps 2 and 4 are where page structure matters. A chunk that contains a heading phrased like the question, followed immediately by a direct answer, is easy to match and easy to quote.
Simulate the chunking
This splits a page at H2/H3 boundaries and then caps chunk size, which is roughly what many heading-aware splitters do:
import re, requests
from bs4 import BeautifulSoup
MAX_WORDS = 220
def chunks(url: str) -> list[dict]:
soup = BeautifulSoup(requests.get(url, timeout=20).text, "html.parser")
for t in soup(["script", "style", "nav", "footer", "header", "aside", "noscript"]):
t.decompose()
main = soup.find("main") or soup.find("article") or soup.body
out, heading, buf = [], "(intro)", []
def flush():
words = " ".join(buf).split()
for i in range(0, len(words), MAX_WORDS):
out.append({"heading": heading, "text": " ".join(words[i:i + MAX_WORDS])})
buf.clear()
for el in main.find_all(["h1", "h2", "h3", "p", "li", "td"]):
if el.name in ("h1", "h2", "h3"):
flush()
heading = el.get_text(" ", strip=True)
else:
buf.append(el.get_text(" ", strip=True))
flush()
return out
Print the chunks for one of your key pages and read them in isolation, as if each one was the only thing the model got. That exercise alone tends to be eye-opening.
Lint the chunks
Now automate the reading. Each check below flags a passage that's likely to lose to a cleaner one:
DANGLING = re.compile(r"^(this|that|these|those|it|they|as (mentioned|noted|discussed)|see above|the above)\b", re.I)
VAGUE_HEADINGS = {"overview", "introduction", "details", "more", "getting started", "our approach", "why us"}
def lint(c: dict) -> list[str]:
issues = []
words = c["text"].split()
first = " ".join(words[:40])
if DANGLING.match(c["text"]):
issues.append("starts with a back-reference; the chunk doesn't stand alone")
if c["heading"].lower().strip(" :") in VAGUE_HEADINGS:
issues.append(f"vague heading '{c['heading']}' carries no query signal")
if len(words) < 40:
issues.append("very short; may not hold a complete answer")
if not re.search(r"\d", c["text"]):
issues.append("no concrete numbers")
if c["heading"].endswith("?") and len(first.split(".")[0].split()) > 35:
issues.append("question heading, but the first sentence is long; answer may be buried")
return issues
for c in chunks("https://example.com/pricing"):
if problems := lint(c):
print(f"\n## {c['heading']}\n {c['text'][:120]}…")
for p in problems:
print(" -", p)
These are heuristics. "No numbers" isn't a defect on every chunk. Use the output as a reading list, not a score.
The fixes, as patterns
Self-contained answers. Repeat the subject instead of using a pronoun. The first sentence should make sense if it's the only thing quoted.
- ## Pricing
- It starts at $49, which includes everything above.
+ ## How much does Acme Analytics cost?
+ Acme Analytics costs $49 per month for up to 5 seats on the Starter plan,
+ and $199 per month for up to 25 seats on Growth. Both include unlimited events.
Answer first, then context. Lead each section with one or two sentences that directly answer its heading, then expand. A retriever ranks the chunk on its full content, but the model quotes what's easiest to lift.
Headings as queries. "How does Acme compare to Mixpanel?" matches a comparison query. "The Acme difference" doesn't.
Tables for comparisons. A comparison table chunks cleanly, and models quote it accurately. A comparison buried in four paragraphs of prose does neither.
Keep sections a sensible size. A 1,500-word section under one H2 gets split mid-thought. Break it with H3s at natural boundaries.
Pin down facts with JSON-LD
Prose is ambiguous. Structured data states facts explicitly: who made this, what it costs, when it was updated. Search engines use it heavily, and pages with clean, consistent facts are less likely to be misquoted.
<script type="application/ld+json">
{
"@context": "https://schema.org",
"@type": "SoftwareApplication",
"name": "Acme Analytics",
"applicationCategory": "BusinessApplication",
"operatingSystem": "Web",
"offers": [
{ "@type": "Offer", "name": "Starter", "price": "49", "priceCurrency": "USD" },
{ "@type": "Offer", "name": "Growth", "price": "199", "priceCurrency": "USD" }
],
"publisher": { "@type": "Organization", "name": "Acme Inc.", "url": "https://acmeanalytics.com" }
}
</script>
Rules that matter more than which types you pick:
- Match the visible page. If JSON-LD says $49 and the page says $59, you've created a contradiction for the model to resolve, and it may resolve it wrong.
- Validate it. Run it through the Schema.org validator or Google's Rich Results Test. Invalid JSON is silently ignored.
- Render it server-side. JSON-LD injected by client-side JavaScript has the same visibility problem as any other client-rendered content.
A quick check that it's in the raw HTML and parses:
curl -s https://example.com/pricing \
| grep -oP '(?s)<script type="application/ld\+json">\K.*?(?=</script>)' \
| python -m json.tool > /dev/null && echo "JSON-LD OK" || echo "JSON-LD missing or invalid"
(That grep handles the common case of one block on one line. For multi-line or multiple blocks, use the BeautifulSoup extractor from the competitor-diff post.)
What about llms.txt?
llms.txt is a proposed convention: a Markdown file at your site root that summarises the site and links to LLM-friendly versions of key pages. It's cheap to add and can help tools that read it, such as some coding assistants and agents working with documentation. Google has said it doesn't use it for Search, and none of the major AI search engines have confirmed relying on it for ranking or citation. Add it if it's easy, but don't treat it as a substitute for the passage-level work above.
Measuring whether it worked
Structural rewrites are hypotheses. Before you rewrite, record the page's baseline, ship the change, and watch citation rate for the prompts that page targets over the next few weeks. Rewrite one page cluster at a time so you can attribute the change.
For the baseline and the after-check, Vista AI's Page Audit scores a page on AI Visibility, AI Understanding and AI Citation (each 0–100), including content extraction (how easily passages can be lifted for RAG), schema and citation potential, with a prioritised fix list and competitor comparison. Running it before and after a rewrite like this is a quick way to see whether the changes did what you intended.
The short version: write every section as if it's the only passage the model will see, because often it is.
Top comments (1)
We need to produce a comment: short, one or two sentences, casual, start with lowercase, specific reaction/question about the video. No marketing, no URLs. Should be about "Writing Pages That Survive RAG Chunking". So comment could ask about how to structure content to avoid chunking issues, or mention a specific point. Ensure no double hyphens, no em dash, no curly quotes. Use straight ASCII. No hashtags. No quotes around the comment. Output only the comment text. Let's craft: "i