Your analytics flattened in March. Organic clicks are down 30%, then 40%. You shipped nothing wrong. Rankings look stable. The traffic just isn't arriving.
Here is what most teams do next: they buy a "GEO audit," add more schema, publish more content. Six months later the numbers are worse and nobody can point at a single decision that caused it.
The reason is almost never the thing being optimized. It's that three technical gates were failing the entire time, and every downstream fix was being applied to a page that a crawler could not fully read.
This is a scored audit for finding that. Eight categories, 100 points, and one script that runs most of it in about thirty seconds.
Why this got harder in 2026, in numbers
Three data points worth internalizing before the checklist:
68% of Google searches now end without a click. That is up from roughly 45% a decade ago. Search volume is at an all-time high and still growing — the capture rate collapsed, not the demand.
When an AI Overview appears, click-through drops by about 61%. AI Overviews trigger on a growing share of informational queries. The top organic result still exists; people just stop there less often.
Only 8% of visits click a traditional result when an AI summary is present, versus 15% when it isn't. Users read the summary. They click the citation occasionally. They click the blue link rarely.
One number moves in the opposite direction, and it's the one that matters for this audit: AI-referred visitors convert at roughly 3x the rate of traditional organic traffic. Similarweb measured ChatGPT-referred sessions at about 15 minutes versus 8 for Google.
So the shape of the opportunity changed. Fewer clicks, better clicks. Which means eligibility is now worth more than volume — a page that can be retrieved and cited is worth several that cannot.
That reframes the work. You are not chasing rankings. You are making your content retrievable, quotable, and unambiguous. That's a technical problem before it's a writing problem, and it's the subject of this audit.
The 40/60 rule: why most audits start in the wrong place
Here is the single most useful thing in this article.
The eight categories below are not weighted equally, and two of them are blocking. If Categories 1 and 2 fail, the other six are decoration.
| Category | Points | Blocking | |
|---|---|---|---|
| 01 | Crawlability | 20 | Yes |
| 02 | Renderability | 20 | Yes |
| 03 | Entity clarity | 18 | — |
| 04 | Structured data | 15 | — |
| 05 | Content chunking | 12 | — |
| 06 | Freshness & trust | 8 | — |
| 07 | Answer formatting | 5 | — |
| 08 | Measurement | 2 | — |
Categories 01 and 02 are 40% of the score and they gate everything downstream. If those two come in under 32 out of 40, stop. Fix rendering and robots.txt this week. Do not spend a single hour on schema.
I have watched teams spend a quarter perfecting sameAs arrays and entity graphs while their documentation body copy was assembled in the browser after hydration. The audit spreadsheet looked excellent. The crawler was receiving an empty shell.
That is the 40/60 rule. Apply it before you read further.
Category 01 — Crawlability (20 points)
Can a non-browser client reach and read your URLs at all? This is binary, and everything else is downstream of it.
| Check | Pts |
|---|---|
robots.txt returns HTTP 200 and contains no wildcard Disallow: /
|
3 |
OAI-SearchBot explicitly allowed (or deliberately blocked, and documented) |
3 |
Googlebot allowed on every route that matters |
3 |
PerplexityBot explicitly allowed (or deliberately blocked) |
2 |
Bingbot allowed — Copilot grounding depends on the Bing index |
2 |
Sitemap exists at /sitemap.xml and is referenced from robots.txt
|
2 |
| Sitemap contains only canonical, indexable, 200-status URLs | 2 |
| No WAF, bot challenge or CDN rule returns 403/429/503 to AI user agents | 3 |
The silent failure that costs entire channels
A CDN default that 403s PerplexityBot costs you the whole channel and produces zero errors anywhere. No deploy, no alert, no log line you'll notice. This is the most common cause of "we optimized for AI search and nothing happened."
Four lines of shell settles it:
for ua in "GPTBot/1.1" "OAI-SearchBot/1.0" \
"PerplexityBot/1.0" "ClaudeBot/1.0"; do
code=$(curl -s -o /dev/null -w "%{http_code}" \
-A "Mozilla/5.0 (compatible; $ua)" https://yoursite.com/)
echo "$ua -> $code"
done
Four 200s, or you have found your first problem.
Know which crawler does what
The robots.txt snippets circulating online are two years stale, and most conflate three different jobs into one decision. These are separate agents:
| Agent | Operator | Governs | Worth blocking? |
|---|---|---|---|
GPTBot |
OpenAI | Model training | Optional |
OAI-SearchBot |
OpenAI | ChatGPT search citations | Rarely |
ChatGPT-User |
OpenAI | User-triggered live fetch | Rarely |
ClaudeBot |
Anthropic | Training + crawling | Optional |
Claude-SearchBot |
Anthropic | Search grounding | Rarely |
PerplexityBot |
Perplexity | Search index | Rarely |
Googlebot |
Search + AI Overviews | Never | |
Google-Extended |
Gemini training only | Optional | |
Bingbot |
Microsoft | Bing + Copilot grounding | Rarely |
CCBot |
Common Crawl | Upstream model data | Optional |
Two corrections that change real decisions:
Blocking Google-Extended does not remove you from Google AI Overviews. AI Overviews are served from the ordinary Google index, governed by Googlebot. Google-Extended only controls Gemini training.
Blocking GPTBot does not remove you from ChatGPT search results. Citations come from OAI-SearchBot, and live URL fetches use ChatGPT-User. Three separate entries, three separate decisions.
A coherent default for most commercial sites is: block training, allow search. That is defensible and, more importantly, deliberate rather than copied.
Category 02 — Renderability (20 points)
Does the content exist in the delivered HTML, or does it appear only after JavaScript runs? This is where modern frameworks quietly lose 15 points.
| Check | Pts |
|---|---|
| Primary body copy present in raw HTML with JavaScript disabled | 8 |
h1–h3 headings present in raw HTML |
3 |
Internal links are real <a href> elements in raw HTML |
3 |
| JSON-LD present in raw HTML, not injected after hydration | 3 |
Content survives curl at 80%+ of the rendered word count |
3 |
# raw word count, tags stripped
curl -s https://yoursite.com/page | sed 's/<[^>]*>//g' | wc -w
Compare that against what the browser shows. Within 10% is fine. A 30% gap is a widget. A 3x gap means your content does not exist as far as retrieval is concerned — no matter how clean your markup is.
The hydration trap
This is the pattern that silently fails:
// ❌ invisible to most AI crawlers
useEffect(() => {
const s = document.createElement('script');
s.type = 'application/ld+json';
s.text = JSON.stringify(schema);
document.head.appendChild(s);
}, [schema]);
And this is the fix:
// ✅ present in the raw HTML response
const jsonLd = JSON.stringify(schema, null, 2);
return (
<script
type="application/ld+json"
dangerouslySetInnerHTML={{ __html: jsonLd }}
/>
);
Verify it in the only place that counts — not DevTools:
curl -s https://yoursite.com/page | grep -c 'application/ld+json'
0 means everything downstream of this section is theoretical.
The practical answer is SSR, SSG, or ISR for every content route. Client-side rendering belongs to application state, not to published content.
Category 03 — Entity clarity (18 points)
Retrieval systems resolve entities, not pages. Can a machine work out who you are, unambiguously, from any page on the site?
| Check | Pts |
|---|---|
Exactly one Organization (or Person) node per page, consistent across pages |
4 |
@id values used and cross-referenced between nodes |
3 |
sameAs lists 3–8 live, populated profiles (no 404s, no empty accounts) |
4 |
Same entity name in <title>, og:title, JSON-LD and visible body copy |
3 |
Organization node has a resolvable logo ImageObject at an absolute URL |
2 |
| No duplicate or conflicting entity blocks from competing plugins | 2 |
curl -s https://yoursite.com/ | grep -o '"@type":"Organization"' | wc -l
# expected: 1
A return of 2 means a plugin and a theme are both emitting an organization with different names or URLs. To a reconciling system that is two companies, and consolidation fails silently — no error, no warning, no dashboard.
On sameAs discipline: every URL must return 200 and carry real content. A dead profile is worse than no profile, because you asserted a fact that then failed verification. Three to eight links is the useful range. Forty reads as manipulation and gets discounted.
Category 04 — Structured data (15 points)
Validators check shape. They do not check meaning, recency, or whether the markup agrees with the page. That gap is where these 15 points live.
| Check | Pts |
|---|---|
| Every JSON-LD block parses with zero syntax errors on priority URLs | 3 |
datePublished / dateModified in ISO 8601 |
3 |
| Visible page content matches every claim the markup makes | 3 |
@type values correctly cased (WebPage, not Webpage) |
2 |
All image values are absolute https:// URLs |
2 |
FAQPage uses nested Question → acceptedAnswer → Answer
|
2 |
Values that parse but degrade output
| Property | Common value | Correct value |
|---|---|---|
datePublished |
Oct 24, 2026 |
2026-10-24 |
image |
/cover.png |
https://yoursite.com/cover.png |
@type |
Webpage |
WebPage |
name |
`Brand \ | Best Tools 2026` |
description |
400-character paragraph | 150–160 chars, one sentence |
"Oct 24, 2026" is ambiguous enough that some parsers drop the field entirely, and dateModified is one of the few freshness signals you fully control.
The row people skip is row three. If your markup asserts an answer the page does not visibly give, you have manufactured a confident wrong answer attributed to your brand. That is worse than silence, because it will be quoted.
And one tool used for the wrong job: the Rich Results Test validates syntax. Syntax was never the failure mode. Use it as a lint step, not a diagnosis.
Category 05 — Content chunking (12 points)
Retrieval does not read your page top to bottom. It splits the page along heading and paragraph boundaries, embeds each piece, and fetches whichever pieces match. Write for the piece.
| Check | Pts |
|---|---|
| Each H2 section is self-contained and makes sense in isolation | 4 |
| Each section opens with a direct answer in the first one or two sentences | 3 |
| No context-dependent openers ("as mentioned above", a bare "this") | 2 |
| Paragraphs under roughly 120 words | 2 |
| Key claims exist as text, not only inside images | 1 |
The isolation test
Copy a single H2 section into a fresh chat with no surrounding page. If it makes sense and answers something, it survives chunking. If it opens with a pronoun pointing three sections up, it does not — and that chunk will never be retrieved no matter how good the page is as a whole.
| Pattern | Survives | Does not |
|---|---|---|
| Opening | "JSON-LD works by placing a self-contained entity graph…" | "It works by wrapping the entity, as mentioned above…" |
| Scope | Names the subject in the first clause | Relies on the H1 for context |
| Length | One idea, under 120 words | Four ideas in one 300-word block |
This category costs nothing to fix and is the one most often skipped. It is a writing-discipline change, not an engineering one — no deploy, no ticket, no framework upgrade.
Category 06 — Freshness & trust (8 points)
| Check | Pts |
|---|---|
| Long-lived pages show a visible "last updated" date | 2 |
dateModified reflects real edits, not a mechanical bump |
2 |
| Author or organization attribution visible on-page | 2 |
| Fact-heavy sections cite external sources with working links | 1 |
| Nothing material is stale in a fast-moving topic | 1 |
A 2023 page about 2026 crawler behaviour is not merely unranked — it gets retrieved, and it is wrong. Archive or update. Do not leave it up hoping.
Category 07 — Answer formatting (5 points)
Deliberately the lowest-weighted category, because it is the easiest and everyone does it first.
| Check | Pts |
|---|---|
| Summary or TL;DR block near the top of long pages | 2 |
| Comparison tables for "X vs Y" queries | 2 |
| Procedural content uses ordered steps | 1 |
Formatting makes content easy to extract, so it feels productive. It is worth 5 points and will not rescue a site scoring 20 on the first two categories.
Category 08 — Measurement (2 points)
| Check | Pts |
|---|---|
| Server logs accessible, AI crawler user agents filterable | 1 |
| Five target questions probed weekly across ChatGPT, Perplexity and Google | 1 |
grep -iE "GPTBot|OAI-SearchBot|ClaudeBot|PerplexityBot" \
/var/log/nginx/access.log \
| awk '{print $9}' | sort | uniq -c
Three things fall out in ninety seconds: whether these crawlers visit at all, which URLs they actually pull, and whether they receive 200 or bounce off a 403. If the answer to the first is no, the rest of your plan is theoretical.
The whole audit in one script
Here is every automatable check above, in a single file. Save it as audit.sh, make it executable, and pass one URL.
It does not cover Categories 03 through 07 completely — those need human judgment — but it will find the blocking failures, which is where you should start.
#!/usr/bin/env bash
# ai-search-audit.sh — blocking checks only
# usage: ./audit.sh https://yoursite.com/page
URL="${1:?usage: ./audit.sh <url>}"
BASE=$(printf '%s' "$URL" | sed -E 's#(https?://[^/]+).*#\1#')
PASS=0; FAIL=0
ok() { echo " PASS $1"; PASS=$((PASS+1)); }
bad() { echo " FAIL $1"; FAIL=$((FAIL+1)); }
echo "== 01 CRAWLABILITY =="
ROBOTS=$(curl -s -o /dev/null -w "%{http_code}" "$BASE/robots.txt")
[ "$ROBOTS" = "200" ] && ok "robots.txt returns 200" \
|| bad "robots.txt returned $ROBOTS"
if curl -s "$BASE/robots.txt" | grep -qiE '^Disallow: *\*? *$|^Disallow: */$'; then
bad "robots.txt contains a wildcard Disallow"
else
ok "no wildcard Disallow in robots.txt"
fi
for ua in "GPTBot/1.1" "OAI-SearchBot/1.0" "PerplexityBot/1.0" \
"ClaudeBot/1.0" "Googlebot/2.1"; do
code=$(curl -s -o /dev/null -w "%{http_code}" \
-A "Mozilla/5.0 (compatible; $ua)" "$URL")
name=${ua%%/*}
if [ "$code" = "200" ]; then ok "$name reaches the page (200)"
else bad "$name got $code — blocked by WAF/CDN?"; fi
done
SM=$(curl -s -o /dev/null -w "%{http_code}" "$BASE/sitemap.xml")
[ "$SM" = "200" ] && ok "sitemap.xml returns 200" \
|| bad "sitemap.xml returned $SM"
echo
echo "== 02 RENDERABILITY =="
RAW=$(curl -s "$URL")
RAW_WORDS=$(printf '%s' "$RAW" | sed 's/<[^>]*>//g' | wc -w | tr -d ' ')
[ "$RAW_WORDS" -ge 300 ] && ok "raw HTML has $RAW_WORDS words" \
|| bad "raw HTML has only $RAW_WORDS words — JS-rendered?"
LD=$(printf '%s' "$RAW" | grep -c 'application/ld+json')
[ "$LD" -ge 1 ] && ok "JSON-LD present in raw HTML ($LD block(s))" \
|| bad "no JSON-LD in raw HTML — injected after hydration?"
H1=$(printf '%s' "$RAW" | grep -c '<h1')
[ "$H1" -ge 1 ] && ok "h1 present in raw HTML" \
|| bad "no h1 in raw HTML"
LINKS=$(printf '%s' "$RAW" | grep -o 'href="http' | wc -l | tr -d ' ')
[ "$LINKS" -ge 5 ] && ok "$LINKS absolute internal/external links in raw HTML" \
|| bad "only $LINKS links in raw HTML"
echo
echo "== 04 STRUCTURED DATA (syntax) =="
ORG=$(printf '%s' "$RAW" | grep -o '"@type":"Organization"' | wc -l | tr -d ' ')
if [ "$ORG" -eq 1 ]; then ok "exactly one Organization node"
elif [ "$ORG" -eq 0 ]; then bad "no Organization node found"
else bad "$ORG Organization nodes — duplicate entity data"; fi
printf '%s' "$RAW" | grep -qE '"date[A-Za-z]+":"[0-9]{4}-[0-9]{2}-[0-9]{2}"' \
&& ok "ISO 8601 dates present" || bad "no ISO 8601 date fields detected"
echo
echo "== RESULT =="
echo " passed: $PASS"
echo " failed: $FAIL"
[ "$FAIL" -eq 0 ] && echo " blocking categories look clear." \
|| echo " fix the FAILs above before touching schema."
Run it. If anything prints FAIL in the first two sections, you have your work order for the week and the other six categories can wait.
Your score sheet
Fill this in after running the script and scoring the human-judgment rows.
| Category | Max | Yours |
|---|---|---|
| 01 · Crawlability | 20 | |
| 02 · Renderability | 20 | |
| 03 · Entity clarity | 18 | |
| 04 · Structured data | 15 | |
| 05 · Content chunking | 12 | |
| 06 · Freshness & trust | 8 | |
| 07 · Answer formatting | 5 | |
| 08 · Measurement | 2 | |
| TOTAL | 100 |
What your score means
| Score | Meaning | Action |
|---|---|---|
| 85–100 | Eligible. Remaining gap is content quality, not technical | Stop auditing. Write better pages |
| 60–84 | Partially eligible. Citations will be inconsistent | Close 01–02 first, then 03–04 |
| 35–59 | Mostly invisible. Likely blocked, JS-only, or unparseable | Everything else is wasted until 01–02 pass |
| 0–34 | Not reachable. Do not buy tooling yet | Fix rendering and robots.txt this week |
Fastest wins, ranked by minutes-to-impact
| Fix | Time | Typical points |
|---|---|---|
| Unblock AI UAs in robots.txt / WAF | 15 min | up to 8 |
| ISO-format all dates | 20 min | 3 |
Remove display:none-only body content |
30 min | 3 |
| Deduplicate Organization blocks | 45 min | 4 |
Add an @graph with @id cross-references |
2 hrs | 7 |
| SSR/ISR the main content routes | 1–3 days | up to 20 |
The last row is the only expensive one, and it is also the only one that can move a site from "not reachable" to "eligible" on its own.
Three things that do not do what you think
1. The Rich Results Test. It validates syntax. Syntax was never the failure mode. Use it as a lint step before deploying, not as a diagnosis of why you are not cited.
2. llms.txt. A proposal published in late 2024, not a standard, with no major AI provider having confirmed publicly that it reads the file. Adding one costs twenty minutes and breaks nothing. Treating it as the fix is a mistake. If someone tells you otherwise, ask which provider confirmed it.
3. Adding more schema types. If you already have a clean Organization node and a correct Article node and are still uncited, more types will not help. At that point the constraint is whether your content is the clearest available answer to a specific question — which is a writing problem, not a markup problem.
FAQ
Does any of this guarantee I'll be cited?
No. It makes you eligible. Selection depends on whether your content is the clearest, most specific answer available for a given question. That is a writing problem, and no audit solves it for you.
Do AI crawlers run JavaScript?
Mostly no, or unreliably. Googlebot renders. The rest generally fetch raw HTML. Build on the assumption that JS-only content does not exist.
Should I block GPTBot?
Decide per purpose, not wholesale. Blocking GPTBot (training) does not affect ChatGPT search. Blocking OAI-SearchBot does. Blocking Google-Extended does not affect AI Overviews. Three separate decisions.
Is llms.txt supported by anyone?
Not confirmed by any major provider as of writing. Harmless to add, wrong to rely on.
How often do these crawlers revisit?
Popular, frequently updated pages within days. Low-traffic pages within weeks to months. Your own server logs are the only reliable answer.
What score should a small site target?
80+. Sites below 60 usually have one or two blocking issues, not a hundred small ones.
How long until structured data shows an effect?
Recrawl cadence is the limiter, not the markup. Expect a quarter, not a week, and track server-log bot hits rather than inferring from traffic.
What to do this week
Score all eight categories honestly, without giving yourself the benefit of the doubt. "Probably fine" is a zero.
If Categories 01 and 02 come in under 32, fix rendering and robots.txt before anything else. Deploy one Organization graph and verify it with curl, not with DevTools.
Then write down five questions your site should be the answer to, and probe them weekly in ChatGPT, Perplexity and Google. Log which URL gets cited. Over a month, that log tells you more than any audit tool will.
If you want this audit as a scored checklist you can actually fill in — 20 pages, all eight categories with every row and point value, plus three production-ready JSON-LD templates (Organization + WebSite, Article + Person, FAQPage) wired with @id cross-references — it's free: *100-Point AI Search Readiness Checklist***
And if you want the full build rather than the audit — the complete template set, the raw-vs-rendered diff tooling, the log-analysis queries, and the rewrite patterns for making each section survive chunking — that's the reference: *Structured for Answers →***
Earlier in this series: JSON-LD for AI Search: 7 Mistakes That Block Citations — why valid markup can still leave you uncited.
Top comments (0)