DEV Community: LovedByAI

The ChatGPT-User Bot Reads Your Site Less Now. Here's How to Check Your Logs

LovedByAI — Tue, 03 Nov 2026 18:56:00 +0000

If you grep your access log for ChatGPT-User every few weeks, you may have noticed the line count going down. You're not imagining it. Across 714 small business websites, ChatGPT-User fetched pages 3,344 times a day from 19 October to 1 November 2026, down 62% from 8,716 a day from 20 July to 2 August 2026. Over the same weeks, visitors arriving from ChatGPT more than doubled.

We build LovedByAI, and our WordPress plugin logs every request a recognised bot makes to the sites it runs on. That's where these numbers come from. The full study, with the weekly chart and method, is our Lab field note on ChatGPT reading less. This post is the developer version: what the two numbers are, why you should never add them together, and a standard-library Python script that splits them in your own log and flags a step change. We ran it before publishing; the output below is real (on a synthetic log, explained further down).

What is the ChatGPT-User bot, and how is it different from a ChatGPT visit?

Two different things show up in your log when ChatGPT is involved, and they mean opposite things for your business.

A read. A request whose user agent contains ChatGPT-User/1.0; +https://openai.com/bot. That's OpenAI's server opening one of your pages while ChatGPT answers someone. No human is looking at your site.
A visit. A request from an ordinary browser that came from a ChatGPT answer. ChatGPT usually adds utm_source=chatgpt.com to the link it shows; sometimes you only get chatgpt.com as the referrer. That is a person on your site.

There's a third OpenAI agent worth keeping apart: OAI-SearchBot, the crawler that builds OpenAI's search index ahead of time. It isn't a live fetch and it isn't a person.

What happened to ChatGPT-User traffic in 2026?

It fell in three steps, and OpenAI announced none of them:

20 August 2026, right after a short spike from 9 to 19 August.
17 to 18 September 2026. Comparing 4 to 17 September with 19 September to 2 October, the typical site lost 39% of its reads (393 sites down, 85 up) while visits from ChatGPT rose 13%.
28 October 2026. On the same 587 sites we follow every week, reads went from about 3,400 a day (19 to 26 October) to about 1,300 a day (28 October to 2 November). Week against week, the typical site fell 49%, with 154 sites down and 6 up. Visits held steady, 658 to 695 a day.

Late July against late October, the typical site lost 66% of its reads. ChatGPT still read most of the sites every week (450 to 497 of the 587 from July to mid October, 411 in the week of 26 October). It opens fewer pages on each one.

We also checked whether OpenAI's index crawler took over the reading. In total, OAI-SearchBot rose 41%, but at the typical site it fell 9% (148 sites down, 124 up), so we don't think it did. We don't know why ChatGPT reads less. The hub lists the outside evidence (Ahrefs on how rarely ChatGPT runs web searches, OpenAI's own description of answering from cached pages).

Why does counting reads and visits together hide the drop?

Because they move in opposite directions. If reads fall while visits rise, one combined "ChatGPT" number barely moves.

We learned this on our own data. Our tracker writes one ChatGPT-User line for every ChatGPT visitor, so the visit shows up next to the real fetches. Visits more than doubled over the window, so leaving those lines in hid about ten points of the drop. The study subtracts them first. We also confirmed the 28 October step was in the real fetches, not in that subtraction: the tracker's own lines were flat, about 1,040 a day.

If you use any plugin or dashboard that reports "ChatGPT traffic" as one figure, find out which of the two it's counting before you read anything into the trend.

How do I find ChatGPT-User reads and ChatGPT visits in my access log?

On most WordPress hosts you can download the raw access log from the control panel (cPanel calls it Raw Access). Nginx and Apache both write the combined format by default, which is what this script reads. It needs Python 3 and nothing else.

"""ChatGPT reads vs ChatGPT visits, per week, from an nginx/Apache combined log.

  python3 chatgpt_reads.py access.log [access.log.1 ...]

reads  = requests whose user agent is ChatGPT-User (OpenAI fetching a page
         while it answers someone). A bot, not a person.
visits = requests from a normal browser that came from ChatGPT: the URL has
         utm_source=chatgpt.com, or the referrer is chatgpt.com.
OAI-SearchBot (OpenAI's index crawler) is counted on its own line.
"lumped" is reads + visits, the single "ChatGPT" number some dashboards show,
printed only so you can see how much of a drop it hides.

"blocked" counts ChatGPT-User requests that got a 4xx or 5xx: if that column
climbs while reads fall, the drop is your firewall, not OpenAI.

A week is flagged when reads fall 40%+ against the average of the two weeks
before it. Standard library only.
"""
import re
import sys
from collections import defaultdict
from datetime import datetime, timedelta
from urllib.parse import urlsplit, parse_qs

LOG = re.compile(
    r'\S+ \S+ \S+ \[(?P<ts>[^\]]+)\] "(?:\S+) (?P<path>\S+)[^"]*" '
    r'(?P<status>\d{3}) \S+ "(?P<ref>[^"]*)" "(?P<ua>[^"]*)"'
)
BOTS = re.compile(r"bot|crawl|spider|slurp|fetch", re.I)
CHATGPT_HOSTS = {"chatgpt.com", "chat.openai.com"}


def kind(path, ref, ua):
    if "ChatGPT-User" in ua:
        return "read"
    if "OAI-SearchBot" in ua:
        return "searchbot"
    if BOTS.search(ua):
        return None
    if "chatgpt.com" in parse_qs(urlsplit(path).query).get("utm_source", []):
        return "visit"
    if urlsplit(ref).hostname in CHATGPT_HOSTS:
        return "visit"
    return None


weeks = defaultdict(lambda: defaultdict(int))
for name in sys.argv[1:]:
    with open(name, errors="replace") as f:
        for raw in f:
            m = LOG.match(raw)
            if not m:
                continue
            k = kind(m["path"], m["ref"], m["ua"])
            if not k:
                continue
            day = datetime.strptime(m["ts"].split()[0], "%d/%b/%Y:%H:%M:%S").date()
            monday = day - timedelta(days=day.weekday())
            weeks[monday][k] += 1
            if k == "read" and m["status"][0] in "45":
                weeks[monday]["blocked"] += 1

print(f"{'week of':<12}{'reads':>7}{'visits':>8}{'lumped':>8}{'searchbot':>11}{'blocked':>9}  note")
prev, prev_lumped = [], []
for wk in sorted(weeks):
    w = weeks[wk]
    lumped = w["read"] + w["visit"]
    note = ""
    if len(prev) >= 2:
        base = sum(prev[-2:]) / 2
        if base and w["read"] <= base * 0.6:
            lbase = sum(prev_lumped[-2:]) / 2
            note = (f"reads {100 * (w['read'] / base - 1):+.0f}% vs prior 2 weeks "
                    f"(lumped: {100 * (lumped / lbase - 1):+.0f}%)")
    print(f"{wk.isoformat():<12}{w['read']:>7}{w['visit']:>8}{lumped:>8}{w['searchbot']:>11}{w['blocked']:>9}  {note}")
    prev.append(w["read"])
    prev_lumped.append(lumped)

Run it on one file or several (rotated logs are fine, as long as they're unzipped):

python3 chatgpt_reads.py access.log access.log.1

Example output: what a step change looks like

We didn't want to publish a customer's log, so we generated an eight-week synthetic one (make_sample_log.py, seeded, in the same folder as the script) that mimics the shape we saw: reads drop on 28 October, visits keep climbing, OAI-SearchBot and Googlebot as noise. These are not study numbers. This is the script's real output on that file:

week of       reads  visits  lumped  searchbot  blocked  note
2026-09-07      121      27     148         33        0  
2026-09-14      118      33     151         30        0  
2026-09-21      126      35     161         30        0  
2026-09-28      123      40     163         31        0  
2026-10-05      123      48     171         34        0  
2026-10-12      121      54     175         32        0  
2026-10-19      119      56     175         32        0  
2026-10-26       72      61     133         35        0  reads -40% vs prior 2 weeks (lumped: -24%)

Three things to read off it:

The flag. Reads fell 40% against the two weeks before. The rule is deliberately blunt: a week at or below 60% of the previous two-week average. On a small site with a dozen reads a week, raise the bar or compare months instead, or noise will trip it.
The lumped column. The same week looks like a 24% dip if you add reads and visits together. That's the masking from the section above, in miniature.
The blocked column. All zeros. If ChatGPT-User requests start getting 403s or 429s, your firewall or CDN is turning OpenAI away and the drop is yours, not theirs. Requests your CDN blocks before they reach the server won't appear in this log at all, so a clean blocked column on the server is necessary but not proof.

Prefer to watch? The study in under six minutes:

What should you do when ChatGPT-User drops in your logs?

Mostly, stop watching it as a health metric. A quieter ChatGPT-User line is not lost ground on its own. Judge ChatGPT by the visits column: that's people. On our sites, visits from ChatGPT rose 160% between the same late-July and late-October windows, and 72% at the typical site (part of that jump is a week in mid October when ChatGPT's links lost their tracking tag and visits looked lower than they were).

What it does change is where you can win. If ChatGPT opens fewer pages live while answering, more of the answer comes from what it already holds about your business. That makes what your pages say when the bot does come by matter more than how often it comes.

And if the blocked column isn't zero, fix that first.

For the business side of the same data, without the code, see Does ChatGPT Still Crawl Websites? On Small Business Sites It Reads 62% Less on Medium.

Original study and full dataset: https://www.lovedby.ai/lab/chatgpt-reads-less

SPA SEO in 2026: Googlebot Renders Your App. ChatGPT Doesn't.

LovedByAI — Thu, 22 Oct 2026 12:43:53 +0000

Most SPA SEO advice answers one question: will Googlebot see my content? Googlebot fetches the HTML, queues the page, runs your JavaScript in a headless Chromium and indexes what comes out. Slow, sometimes flaky, but it gets there.

The AI crawlers don't get there. ChatGPT's crawlers fetch the HTML and stop. If your app mounts into an empty <div id="root">, that empty div is the whole page as far as ChatGPT search is concerned.

We build LovedByAI, an AI search optimization platform for small businesses. Our WordPress plugin logs every request a recognised bot makes to a site, which is where the crawler numbers below come from. This post is the developer version of our breakdown of dynamic bot rendering, a CDN feature now being pitched to marketers as the fix. It includes a standard-library Python script that shows what each crawler gets from a URL, and that spots dynamic rendering from the outside. We ran it before publishing; the output below is real.

Do AI crawlers render JavaScript?

No, not the ones that matter for ChatGPT. Vercel's December 2024 study found that none of the major AI crawlers executed JavaScript, including ChatGPT-User, OAI-SearchBot, GPTBot, ClaudeBot and PerplexityBot. ChatGPT and Claude did download JavaScript files (11.50% and 23.84% of their requests), but never ran them. The exceptions were Gemini, which uses Googlebot's rendering infrastructure, and AppleBot.

So for an SPA there are two audiences now:

Googlebot, which renders. Your existing SPA SEO work (rendering budget, <title> updates, history routing, hydration errors) still applies.
Everything OpenAI, Anthropic and Perplexity send, which reads the HTML you return and nothing after it.

Which AI crawlers should an SPA care about?

AI bots come in three kinds, and they do different jobs:

Kind	Examples	What it does	Share of AI bot requests
Training crawlers	GPTBot, ClaudeBot, Meta-ExternalAgent	Copy pages to train future models	69.9%
Search index crawlers	OAI-SearchBot, PerplexityBot, Claude-SearchBot	Crawl ahead of time to build the index an assistant searches when someone asks	18.8%
Live fetchers	ChatGPT-User, Claude-User, Perplexity-User	Open one page at the moment a person asks, to read it for the answer	11.3%

The shares are from LovedByAI's logs: the average across 924 WordPress sites running our plugin, nearly three in four of them small and mid-sized businesses, from 6 September to 5 October 2026. An app with a different audience will have a different mix, but the reach is worth knowing: in that window OAI-SearchBot crawled 90.6% of the sites and ChatGPT-User 91.7%.

The one to worry about is OAI-SearchBot. OpenAI uses it to surface websites in ChatGPT search. If it reads an empty shell, ChatGPT search has little more than your <title> to go on for that URL.

How do you see what each crawler gets from a page?

Fetch the page with each crawler's user agent and count the visible words in the raw HTML, before any JavaScript. That's what bot_view.py does. For every URL it:

fetches as a desktop browser, Googlebot, OAI-SearchBot, ChatGPT-User and GPTBot, using the user-agent strings Google and OpenAI publish;
reports the status, the first <h1> and the visible word count, skipping <script>, <style>, <noscript>, <template> and <svg>;
flags an empty shell: fewer than 50 visible words, with the app root (#root, #app, #__next, #__nuxt, #___gatsby) empty;
flags a page that differs by user agent: a different status, title, H1 or a big gap in word count compared with the browser fetch.

The second check matters more than it looks. A crawler getting a different page than a browser is what dynamic rendering, a bot-blocking firewall and cloaking all look like from outside. The script can't tell you which one it is, only that you should go and look.

#!/usr/bin/env python3
"""bot_view.py: what does each crawler get from your page before any JavaScript runs?

Fetches every URL as a desktop browser, Googlebot, OAI-SearchBot, ChatGPT-User and
GPTBot, using the user-agent strings Google and OpenAI publish, and reports the
status, the first <h1> and the number of visible words in the raw HTML (titles are
compared too).

It flags two things:
  1. EMPTY SHELL: the raw HTML has almost no visible text (an app root such as
     <div id="root"> with nothing inside, filled later by JavaScript). Crawlers
     that don't run JavaScript, which today means OpenAI's and most other AI
     crawlers, read exactly this.
  2. DIFFERS BY USER AGENT: a crawler gets a different title, H1, status or
     amount of text than the browser does. That's what dynamic rendering (a
     pre-rendered snapshot served to some bots) or a bot-blocking firewall looks
     like from the outside. Same content in a different shape is fine;
     different content is cloaking.

Standard library only. Python 3.9+.
Usage:  python3 bot_view.py https://example.com/ [more URLs] [--min-words 50]
Exit code 1 if anything was flagged, so it can run in CI.
"""
import argparse
import re
import sys
import urllib.error
import urllib.request
from html.parser import HTMLParser

# Published user-agent strings, October 2026. Google's docs write Googlebot's
# Chrome version as W.X.Y.Z (it follows current Chrome); any recent version works.
AGENTS = [
    ("Browser", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 "
                "(KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36"),
    ("Googlebot", "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 "
                  "(KHTML, like Gecko) Chrome/141.0.0.0 Mobile Safari/537.36 "
                  "(compatible; Googlebot/2.1; +http://www.google.com/bot.html)"),
    ("OAI-SearchBot", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 "
                      "(KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; "
                      "OAI-SearchBot/1.4; +https://openai.com/searchbot"),
    ("ChatGPT-User", "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; "
                     "ChatGPT-User/1.0; +https://openai.com/bot"),
    ("GPTBot", "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; "
               "GPTBot/1.4; +https://openai.com/gptbot"),
]

# Ids that single-page apps mount into (React, Vue, Next.js, Nuxt, Gatsby, Angular-ish).
APP_ROOTS = {"root", "app", "__next", "__nuxt", "___gatsby", "svelte", "main-app"}
SKIP = {"script", "style", "noscript", "template", "svg", "head", "title"}
VOID = {"area", "base", "br", "col", "embed", "hr", "img", "input", "link", "meta",
        "source", "track", "wbr"}


class PageText(HTMLParser):
    """Collects the title, first H1, visible text and any empty app root."""

    def __init__(self):
        super().__init__(convert_charrefs=True)
        self.stack = []          # open tags, to know when we're inside <script> etc.
        self.title, self.h1, self.text = [], [], []
        self.in_h1 = False
        self.h1_done = False
        self.scripts = 0
        self.root_id, self.root_depth, self.root_words = None, None, 0
        self.empty_root = None

    def handle_starttag(self, tag, attrs):
        if tag == "script":
            self.scripts += 1
        if tag in VOID:
            return
        self.stack.append(tag)
        if tag == "h1" and not self.h1_done:
            self.in_h1 = True
        el_id = dict(attrs).get("id")
        if el_id in APP_ROOTS and self.root_id is None:
            self.root_id, self.root_depth, self.root_words = el_id, len(self.stack), 0

    def handle_endtag(self, tag):
        if tag in VOID or tag not in self.stack:
            return
        while self.stack:                      # tolerate unclosed tags
            closed = self.stack.pop()
            if closed == "h1" and self.in_h1:
                self.in_h1, self.h1_done = False, True
            if self.root_depth is not None and len(self.stack) < self.root_depth:
                if self.root_words == 0:
                    self.empty_root = self.root_id
                self.root_depth = None
            if closed == tag:
                break

    def handle_data(self, data):
        if "title" in self.stack:
            self.title.append(data)
        if any(t in SKIP for t in self.stack):
            return
        words = data.split()
        if not words:
            return
        self.text.extend(words)
        if self.in_h1:
            self.h1.append(data)
        if self.root_depth is not None:
            self.root_words += len(words)


def clean(parts, limit=40):
    s = re.sub(r"\s+", " ", " ".join(parts)).strip()
    return s if len(s) <= limit else s[: limit - 1] + "…"


def fetch(url, ua):
    req = urllib.request.Request(url, headers={"User-Agent": ua, "Accept": "text/html,*/*"})
    try:
        with urllib.request.urlopen(req, timeout=20) as r:
            status, body, ctype = r.status, r.read(5_000_000), r.headers.get_content_charset()
    except urllib.error.HTTPError as e:
        status, body, ctype = e.code, e.read(5_000_000), e.headers.get_content_charset()
    except Exception as e:  # DNS, TLS, timeout: report it as a row, keep going
        return {"status": f"ERR {type(e).__name__}", "title": "", "h1": "", "words": 0,
                "scripts": 0, "empty_root": None}
    p = PageText()
    p.feed(body.decode(ctype or "utf-8", errors="replace"))
    p.close()
    return {"status": status, "title": clean(p.title, 10_000), "h1": clean(p.h1, 10_000), "words": len(p.text),
            "scripts": p.scripts, "empty_root": p.empty_root}


def check(url, min_words):
    rows = [(name, fetch(url, ua)) for name, ua in AGENTS]
    print(f"\n{url}")
    print(f"  {'agent':<14}{'status':<7}{'words':>6}  h1")
    for name, r in rows:
        print(f"  {name:<14}{str(r['status']):<7}{r['words']:>6,}  {clean([r['h1']]) or '(no h1)'}")

    flags = []
    shells = {}  # same reason -> the agents it applies to, so five bots make one line
    for name, r in rows:
        if isinstance(r["status"], int) and r["status"] < 400 and r["words"] < min_words:
            why = f"{r['words']} visible words"
            if r["empty_root"]:
                why += f', empty <div id="{r["empty_root"]}">'
            if r["scripts"]:
                why += f", {r['scripts']} script tag{'s' * (r['scripts'] != 1)}"
            shells.setdefault(why, []).append(name)
    for why, names in shells.items():
        who = "every agent" if len(names) == len(rows) else ", ".join(names)
        note = " Googlebot renders JavaScript later; OpenAI's crawlers don't." if "Googlebot" in names else ""
        flags.append(f"EMPTY SHELL for {who}: {why}.{note}")

    base = rows[0][1]
    for name, r in rows[1:]:
        diffs = []
        if r["status"] != base["status"]:
            diffs.append(f"status {base['status']} -> {r['status']}")
        if r["title"] != base["title"]:
            diffs.append("title")
        if r["h1"] != base["h1"]:
            diffs.append("h1")
        gap = abs(r["words"] - base["words"])
        if gap > 50 and gap > 0.2 * max(r["words"], base["words"]):
            diffs.append(f"words {base['words']:,} -> {r['words']:,}")
        if diffs:
            flags.append(f"DIFFERS BY USER AGENT for {name} vs Browser: {', '.join(diffs)}.")

    for f in flags:
        print(f"  ! {f}")
    if not flags:
        print("  ok: every agent gets the same page, with text in the HTML.")
    return bool(flags)


def main():
    ap = argparse.ArgumentParser(description=__doc__.split("\n")[0])
    ap.add_argument("urls", nargs="+")
    ap.add_argument("--min-words", type=int, default=50,
                    help="fewer visible words than this counts as an empty shell (default 50)")
    a = ap.parse_args()
    flagged = [check(u, a.min_words) for u in a.urls]
    sys.exit(1 if any(flagged) else 0)


if __name__ == "__main__":
    main()

What does bot_view.py show on a single-page app?

Here's the demo app we used, a plumber's homepage built the SPA way: index.html holds <div id="root"></div> and one script tag, and app.js fills in the headline, services and reviews. Served locally with python3 -m http.server 8790:

$ python3 bot_view.py http://localhost:8790/
http://localhost:8790/
  agent         status  words  h1
  Browser       200         0  (no h1)
  Googlebot     200         0  (no h1)
  OAI-SearchBot 200         0  (no h1)
  ChatGPT-User  200         0  (no h1)
  GPTBot        200         0  (no h1)
  ! EMPTY SHELL for every agent: 0 visible words, empty <div id="root">, 1 script tag. Googlebot renders JavaScript later; OpenAI's crawlers don't.

Every agent gets the same empty shell: a <title> and no text. In a browser, the page shows a headline, three service cards and a five-star review. Googlebot gets there in its rendering pass. ChatGPT-User, OAI-SearchBot and GPTBot never do.

And here is a server-rendered page, our own GEO checklist (a Next.js page rendered on the server):

$ python3 bot_view.py https://www.lovedby.ai/blog/geo-checklist
https://www.lovedby.ai/blog/geo-checklist
  agent         status  words  h1
  Browser       200     4,243  GEO Checklist: 30 On-Page Checks for AI…
  Googlebot     200     4,243  GEO Checklist: 30 On-Page Checks for AI…
  OAI-SearchBot 200     4,243  GEO Checklist: 30 On-Page Checks for AI…
  ChatGPT-User  200     4,243  GEO Checklist: 30 On-Page Checks for AI…
  GPTBot        200     4,243  GEO Checklist: 30 On-Page Checks for AI…
  ok: every agent gets the same page, with text in the HTML.

Same 4,243 words for every agent, and the H1 in the HTML. That's the result you want.

What does dynamic bot rendering look like from outside?

Dynamic bot rendering is the old dynamic rendering pattern pointed at AI assistants. A rule on your CDN (CloudFront, Cloudflare or Vercel) checks the user agent for a list of assistant tokens such as chatgpt-user, claude-user and perplexity-user. Matching requests get a stored, pre-rendered snapshot of the page. Everyone else, OAI-SearchBot included, gets the live page.

To see what that looks like to the script, we built a toy version: the same demo app, plus a server that hands a pre-rendered snapshot.html to the nine assistant tokens on the published list.

ASSISTANT_TOKENS = ("chatgpt-user", "codex", "claude-user", "claude-code", "perplexity-user",
                    "gemini-deep-research", "duckassistbot", "amzn-user", "mistralai-user")
ROOT = os.path.join(os.path.dirname(os.path.abspath(__file__)), "demo")


class Handler(http.server.SimpleHTTPRequestHandler):
    def __init__(self, *a, **kw):
        super().__init__(*a, directory=ROOT, **kw)

    def do_GET(self):
        ua = (self.headers.get("User-Agent") or "").lower()
        if self.path in ("/", "/index.html") and any(t in ua for t in ASSISTANT_TOKENS):
            self.path = "/snapshot.html"
        super().do_GET()

Run against it:

$ python3 dynamic_demo.py 8793 &
$ python3 bot_view.py http://localhost:8793/
http://localhost:8793/
  agent         status  words  h1
  Browser       200         0  (no h1)
  Googlebot     200         0  (no h1)
  OAI-SearchBot 200         0  (no h1)
  ChatGPT-User  200        64  Emergency plumber in Leeds, open 24/7
  GPTBot        200         0  (no h1)
  ! EMPTY SHELL for Browser, Googlebot, OAI-SearchBot, GPTBot: 0 visible words, empty <div id="root">, 1 script tag. Googlebot renders JavaScript later; OpenAI's crawlers don't.
  ! DIFFERS BY USER AGENT for ChatGPT-User vs Browser: h1, words 0 -> 64.

That's the whole trade in five rows. ChatGPT-User, the live fetcher, now reads the headline and 64 words. OAI-SearchBot, the crawler that builds ChatGPT search's index, still gets an empty <div id="root">. So does GPTBot, and so would PerplexityBot and Claude-SearchBot. A snapshot helps the fetch that happens after ChatGPT has already picked your page; it does nothing for the index it picks from.

Two more things the toy makes obvious:

Anyone can send that user agent. The routing trusts a header. On Wednesday 7 October 2026, 69% of the 404s logged under the ChatGPT-User user agent on small business websites in LovedByAI's logs were exploit probes, not ChatGPT. If you measure a snapshot's effect by counting requests under assistant names, check them against OpenAI's published IP ranges (chatgpt-user.json, searchbot.json, gptbot.json) first.
A user-agent test only sees user-agent routing. If a CDN also checks the request's IP, bot_view.py will see the live page for every agent. That isn't proof nothing is switched.

Is it cloaking to serve ChatGPT-User a snapshot?

Not by Google's definition, while the snapshot matches the page. Google's dynamic rendering documentation says similar content isn't cloaking, and its example of cloaking is a page about cats for users and dogs for crawlers. The risk is drift: a snapshot that still shows last month's price isn't the same content as the live page any more. When bot_view.py reports DIFFERS BY USER AGENT, compare the two responses by hand.

The same page warns that "dynamic rendering was a workaround and not a long-term solution."

How do you fix SPA SEO for AI crawlers?

Put the content in the HTML you send, for every visitor. Google recommends three ways to do that:

Server-side rendering. Next.js, Nuxt, SvelteKit, Remix and Angular's SSR all render the first response on the server, then hydrate. Every crawler gets the text; nobody gets a different page.
Static generation (prerendering at build time). For marketing pages, docs and anything that doesn't change per request, build HTML files. Astro, Next.js static export and Nuxt's generate mode all do this. It's the cheapest fix there is: the CDN serves files.
Hydration of server-sent HTML, the general form of both: send real markup, then let JavaScript take over.

If you can't change the app soon, serve a prerendered copy to every bot you care about, OAI-SearchBot, PerplexityBot and Claude-SearchBot included, not just the live fetchers. And give it an expiry: re-render on deploy and when prices or stock change, so the copy never drifts from the page.

Whatever you pick, put bot_view.py in CI against a staging URL. It exits with 1 when anything is flagged:

python3 bot_view.py https://staging.example.com/ https://staging.example.com/pricing

A deploy that ships an empty shell, or a CDN rule that starts sending bots a different page, then fails the build instead of quietly costing you ChatGPT search.

Where does WordPress fit?

Mostly outside this problem. WordPress builds each page in PHP and sends finished HTML, so the headings, text and schema are in the first response. The exceptions are widgets that load after the page (some review carousels, booking and menu embeds) and headless WordPress with a front end that renders in the browser.

By default, LovedByAI doesn't switch anything by user agent: people, ChatGPT-User, OAI-SearchBot and Googlebot get the same WordPress page at the same URL. Each page's machine-readable copy lives at its own public address (the page URL plus ?llm_view=1), linked from the site's footer with a plain HTML link, so any person or bot can open it.

The full breakdown of dynamic bot rendering, with the pros, the cons and how it compares with prerendering and server-side rendering: https://www.lovedby.ai/blog/dynamic-bot-rendering

For the business side, whether a small business should turn dynamic bot rendering on at all, see our analysis on Medium.

How to Run a GEO Audit on Any URL with One Python Script

LovedByAI — Tue, 20 Oct 2026 19:54:38 +0000

A GEO audit asks one thing of a web page: can ChatGPT search, Google AI Overviews, Copilot and Perplexity reach it, understand it and quote it? Most of the answer needs a human. Whether the opening two sentences actually answer the question is a judgment call.

About a third of it can be checked by a script, though, and that third is where the silent failures live: a robots.txt group that catches Claude-SearchBot by accident, a firewall that answers OAI-SearchBot with a 403, FAQ answers that exist only inside JSON-LD.

We build LovedByAI, a generative engine optimization platform for small-business WordPress sites that runs most of these checks automatically. This post is the do-it-yourself version: one standard-library Python script that audits a single URL against the machine-checkable items of our 30-point GEO checklist. We ran it before publishing; the real output is below.

What can a GEO audit script check, and what can't it?

Here's what the script checks, with the checklist number each one maps to:

AI search crawler access in robots.txt for OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and Bingbot (check 1). The training crawlers, GPTBot, ClaudeBot and Google-Extended, are reported but never fail the audit: blocking them is a separate decision and doesn't remove a page from search answers.
A status-code probe with each crawler's user-agent token, as a hint for firewall and CDN blocks (check 2).
Server response time, flagged above one second (check 5).
noindex, nosnippet and the canonical (checks 3 and 6).
Exactly one H1, a title that agrees with it, and a meta description of 160 characters or less (checks 8 to 10).
JSON-LD types, including an Organization or LocalBusiness node and its sameAs links (checks 26 and 27).
FAQPage answers present in the HTML, not only in the schema (check 23).
The URL listed in the XML sitemap, with a lastmod (check 7).

What it can't check is everything that needs reading: whether the page answers its question in the first two sentences, whether the headings are the questions customers ask, whether the proof is real. Those stay with a person.

How does robots.txt matching work for OAI-SearchBot and Claude-SearchBot?

RFC 9309 is short, and it's worth implementing properly. A crawler obeys the group that names its product token (case-insensitive), and falls back to the * group only if no group names it. Inside the group, the longest matching path wins, and on a tie, Allow beats Disallow. * matches anything and a trailing $ anchors the end.

We didn't use Python's built-in urllib.robotparser, because it reads rules differently. In Python 3.9 (the version we ran), it applies rules in file order and doesn't expand wildcards. Given the classic WordPress pair under User-agent: *, Disallow: /wp-admin/ followed by Allow: /wp-admin/admin-ajax.php, it disagrees with the longest-match rule:

>>> import urllib.robotparser
>>> rp = urllib.robotparser.RobotFileParser()
>>> rp.parse(open("robots.txt").read().splitlines())
>>> rp.can_fetch("OAI-SearchBot", "http://localhost:8765/wp-admin/admin-ajax.php")
False
>>> rp2 = urllib.robotparser.RobotFileParser()
>>> rp2.parse(["User-agent: *", "Disallow: /*.pdf$"])
>>> rp2.can_fetch("OAI-SearchBot", "http://localhost:8765/price-list.pdf")
True

The first result is wrong under longest-match: Allow: /wp-admin/admin-ajax.php is longer than Disallow: /wp-admin/, so the file allows it. The second ignores Disallow: /*.pdf$ entirely. The script's robots_verdict() gets both right, and it reports which line decided, so you can see why a crawler is blocked.

A mistake worth checking for is a pasted "block the AI bots" list that puts the search crawler in the same group as the training crawlers:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Claude-SearchBot
Disallow: /

That third line stops Claude-SearchBot from indexing the site for Claude's search, along with the training crawl. Anthropic's crawler documentation describes the three bots separately: ClaudeBot collects training data, Claude-SearchBot indexes for search, and Claude-User fetches pages when a person asks.

Why can a page return 200 to your browser and 403 to ChatGPT's crawler?

Because robots.txt isn't the only gate. Firewalls, CDNs and WordPress security plugins block crawlers by user-agent, and robots.txt never hears about it. In LovedByAI's study of 314 live websites in August 2026, 43% refused at least one AI crawler, and 112 of those refused it at the server while robots.txt said nothing.

So the script fetches the page once with a browser user-agent, then once per search crawler with Mozilla/5.0 (compatible; <Bot>/1.0), and warns when the status codes differ.

Read that probe as a hint, not proof. Real crawlers come from published IP ranges, and some firewalls check that, so a fake Googlebot from your laptop can get a 403 because the CDN is doing its job. Rules keyed on IP addresses don't show up in the probe at all. If the probe and the browser disagree for OAI-SearchBot or Claude-SearchBot, go and read the bot settings in your firewall or CDN; for the definitive answer, look for the crawler's requests and their status codes in your server logs.

How do you check noindex, nosnippet and the canonical?

The script reads <meta name="robots">, <meta name="googlebot"> and the X-Robots-Tag header, and fails on noindex, nosnippet, none or max-snippet:0. Google only uses pages that are indexed and eligible to show with a snippet for AI Overviews and AI Mode, and OpenAI uses noindex to keep pages out of ChatGPT search.

It also warns when the canonical points somewhere else. The canonical, the sitemap and the robots directives should all agree on which URL is the real one.

How do you check that the title, H1 and meta description agree?

Microsoft's guidance for Copilot and Bing asks for a title, H1 and description that agree. The script counts H1s (exactly one passes), then checks how many of the H1's significant words appear in the <title>, warning below half. It's a blunt measure on purpose: "Emergency plumber in Leeds, open 24/7" against "Emergency Plumber in Leeds, 24/7 | Joe's Plumbing" shares 5 of 6 words, while a homepage whose H1 is a slogan shares none.

How do you check FAQ answers are in the HTML, not only in the schema?

This check catches a failure that's easy to miss. Google's rule is that structured data has to match the visible text, and Microsoft notes that AI systems "may not render hidden content". An FAQ whose answers load by JavaScript when someone clicks a question will have complete FAQPage JSON-LD and an empty page.

The script walks every JSON-LD block (including @graph), collects each Question and its acceptedAnswer, normalizes the answer text and looks for its first 80 characters in the page's static text. Text inside a closed <details> element counts as present, because it is in the HTML. Text that arrives later doesn't.

How do you check the page is in your XML sitemap?

AI crawlers lean on sitemaps: across 82 million crawler requests on about 1,600 WordPress sites, they requested sitemap XML 177,353 times in May 2026 alone. The script reads the Sitemap: lines from robots.txt (or tries /sitemap.xml, /sitemap_index.xml and WordPress core's /wp-sitemap.xml), walks sitemap indexes up to 60 files, and checks the URL is listed with a lastmod date.

The GEO audit script

Standard library only, so it runs anywhere Python 3.9 or later does. Save it as geo_audit.py and run python3 geo_audit.py https://example.com/page/. It exits with code 1 if any check fails, so it can sit in CI or a cron job.

#!/usr/bin/env python3
"""geo_audit.py: check one URL against the machine-checkable items of an on-page GEO checklist.

Usage:
    python3 geo_audit.py https://example.com/some-page/

Checks robots.txt rules for the AI search crawlers (and what a plain user-agent
probe gets back), noindex/nosnippet, canonical, title/H1 agreement, one H1,
meta description, JSON-LD types, whether FAQPage answers are really in the
HTML, and whether the URL is listed in the XML sitemap.

Standard library only. Python 3.9+. Exit code 1 if any check FAILs.
"""
import gzip
import json
import re
import sys
import time
import xml.etree.ElementTree as ET
from html.parser import HTMLParser
from urllib.error import HTTPError, URLError
from urllib.parse import urljoin, urlsplit
from urllib.request import Request, urlopen

BROWSER_UA = ("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 "
              "(KHTML, like Gecko) Chrome/129.0 Safari/537.36")

# Crawlers that put a page into search answers. Blocking any of these is a FAIL.
SEARCH_BOTS = ["OAI-SearchBot", "Claude-SearchBot", "PerplexityBot", "Googlebot", "Bingbot"]
# Training crawlers: a separate decision, reported for information only.
TRAINING_BOTS = ["GPTBot", "ClaudeBot", "Google-Extended"]

ORG_TYPES = {"organization", "localbusiness", "corporation", "professionalservice",
             "store", "onlinestore", "restaurant", "dentist", "physician", "plumber",
             "electrician", "hvacbusiness", "roofingcontractor", "locksmith",
             "legalservice", "attorney", "autorepair", "medicalbusiness",
             "homeandconstructionbusiness", "generalcontractor", "realestateagent"}

STOPWORDS = set("a an and the of in on for to with by at from your our you we is are be "
                "how what why when who which do does it its this that or vs".split())

results = []  # (status, section, message)


def report(status, section, message):
    results.append((status, section, message))


def fetch(url, ua=BROWSER_UA, timeout=15):
    """Return (status, headers, body_text, seconds). Never raises on HTTP errors."""
    start = time.time()
    req = Request(url, headers={"User-Agent": ua, "Accept": "*/*"})
    try:
        with urlopen(req, timeout=timeout) as resp:
            raw, status, headers = resp.read(), resp.status, resp.headers
    except HTTPError as e:
        raw, status, headers = e.read() or b"", e.code, e.headers
    except URLError as e:
        return None, {}, str(e.reason), time.time() - start
    if raw[:2] == b"\x1f\x8b":
        raw = gzip.decompress(raw)
    charset = headers.get_content_charset() if hasattr(headers, "get_content_charset") else None
    return status, headers, raw.decode(charset or "utf-8", errors="replace"), time.time() - start


# ---------------------------------------------------------------- robots.txt (RFC 9309)

def parse_robots(text):
    """Return (groups, sitemaps). A group is (list_of_agents, list_of_(allow, pattern))."""
    groups, sitemaps = [], []
    agents, rules, in_agent_lines = [], [], False
    for raw in text.splitlines():
        line = raw.split("#", 1)[0].strip()
        if ":" not in line:
            continue
        key, value = (part.strip() for part in line.split(":", 1))
        key = key.lower()
        if key == "user-agent":
            if not in_agent_lines and (agents or rules):
                groups.append((agents, rules))
                agents, rules = [], []
            agents.append(value.lower())
            in_agent_lines = True
        elif key in ("allow", "disallow"):
            if agents:
                rules.append((key == "allow", value))
            in_agent_lines = False
        elif key == "sitemap":
            sitemaps.append(value)
    if agents:
        groups.append((agents, rules))
    return groups, sitemaps


def pattern_matches(pattern, path):
    if not pattern:
        return False  # "Disallow:" with no value blocks nothing
    regex = re.escape(pattern).replace(r"\*", ".*")
    if regex.endswith(r"\$"):
        regex = regex[:-2] + "$"
    return re.match(regex, path) is not None


def robots_verdict(groups, bot, path):
    """Longest matching rule wins; on a tie, Allow wins. Returns (allowed, reason)."""
    token = bot.lower()
    chosen = [g for g in groups if token in g[0]]
    agent_label = token
    if not chosen:
        chosen = [g for g in groups if "*" in g[0]]
        agent_label = "*"
    rules = [r for g in chosen for r in g[1]]
    best = None
    for allow, pattern in rules:
        if pattern_matches(pattern, path):
            if (best is None or len(pattern) > len(best[1])
                    or (len(pattern) == len(best[1]) and allow)):
                best = (allow, pattern)
    if best is None:
        return True, f'no rule matches (group "{agent_label}")' if chosen else "no group applies"
    verb = "Allow" if best[0] else "Disallow"
    return best[0], f'"{verb}: {best[1]}" in group "{agent_label}"'


def check_crawlers(url):
    parts = urlsplit(url)
    robots_url = f"{parts.scheme}://{parts.netloc}/robots.txt"
    path = (parts.path or "/") + (f"?{parts.query}" if parts.query else "")
    status, _, body, _ = fetch(robots_url)
    sitemaps = []
    if status == 200:
        groups, sitemaps = parse_robots(body)
        note = f"robots.txt {status}"
    elif status is not None and 400 <= status < 500:
        groups, note = [], f"robots.txt {status}: crawlers treat this as allow-all"
    else:
        got = "could not be fetched" if status is None else f"returned {status}"
        report("FAIL", "Crawler access", f"robots.txt {got}: crawlers may treat the "
               "whole site as blocked until it answers")
        return []
    report("INFO", "Crawler access", note)

    browser_status = fetch(url)[0]
    for bot in SEARCH_BOTS:
        allowed, reason = robots_verdict(groups, bot, path)
        probe = fetch(url, ua=f"Mozilla/5.0 (compatible; {bot}/1.0)")[0]
        line = f"{bot:<17} {'allowed' if allowed else 'BLOCKED'}: {reason}; UA probe {probe}"
        if not allowed:
            report("FAIL", "Crawler access", line)
        elif probe != browser_status:
            report("WARN", "Crawler access", line + f" (a browser gets {browser_status}: "
                   "check firewall/CDN bot rules)")
        else:
            report("PASS", "Crawler access", line)
    for bot in TRAINING_BOTS:
        allowed, reason = robots_verdict(groups, bot, path)
        report("INFO", "Crawler access", f"{bot:<17} {'allowed' if allowed else 'blocked'}: "
               f"{reason} (training crawler, your call)")
    return sitemaps


# ---------------------------------------------------------------- the page itself

class PageParser(HTMLParser):
    SKIP = {"script", "style", "noscript", "template", "svg"}

    def __init__(self):
        super().__init__(convert_charrefs=True)
        self.title, self.h1s, self.metas, self.canonical = "", [], {}, None
        self.jsonld, self.text, self.question_headings = [], [], 0
        self._stack, self._buf, self._skip_depth, self._in_jsonld = [], None, 0, False

    def handle_starttag(self, tag, attrs):
        a = {k.lower(): (v or "") for k, v in attrs}
        if tag == "meta" and a.get("name"):
            self.metas.setdefault(a["name"].lower(), []).append(a.get("content", ""))
        elif tag == "link" and "canonical" in a.get("rel", "").lower().split():
            self.canonical = a.get("href")
        elif tag == "script" and a.get("type", "").lower() == "application/ld+json":
            self._in_jsonld, self._buf = True, []
        if tag in ("title", "h1", "h2", "h3") and not self._skip_depth:
            self._stack.append((tag, []))
        if tag in self.SKIP:
            self._skip_depth += 1

    def handle_endtag(self, tag):
        if tag == "script" and self._in_jsonld:
            self.jsonld.append("".join(self._buf))
            self._in_jsonld, self._buf = False, None
        if tag in self.SKIP and self._skip_depth:
            self._skip_depth -= 1
        if self._stack and self._stack[-1][0] == tag:
            _, chunks = self._stack.pop()
            text = " ".join("".join(chunks).split())
            if tag == "title":
                self.title = self.title or text
            elif tag == "h1":
                self.h1s.append(text)
            if tag in ("h2", "h3") and text.endswith("?"):
                self.question_headings += 1

    def handle_data(self, data):
        if self._in_jsonld:
            self._buf.append(data)
            return
        if self._skip_depth:
            return
        for _, chunks in self._stack:
            chunks.append(data)
        self.text.append(data)


def words(s):
    return [w for w in re.findall(r"[a-z0-9]+", s.lower()) if w not in STOPWORDS]


def norm(s):
    return " ".join(re.findall(r"[a-z0-9]+", re.sub(r"<[^>]+>", " ", s).lower()))


def walk_jsonld(node, found):
    if isinstance(node, list):
        for item in node:
            walk_jsonld(item, found)
    elif isinstance(node, dict):
        types = node.get("@type", [])
        for t in ([types] if isinstance(types, str) else types):
            found.append((t, node))
        for key, value in node.items():
            if key != "@type":
                walk_jsonld(value, found)


def check_page(url, status, headers, html):
    p = PageParser()
    p.feed(html)
    page_text = norm(" ".join(p.text))

    # Indexing and snippets
    header_values = headers.get_all("X-Robots-Tag", []) if hasattr(headers, "get_all") else []
    directives = " ".join(p.metas.get("robots", []) + p.metas.get("googlebot", []) +
                          header_values).lower()
    bad = [d for d in ("noindex", "nosnippet", "none", "max-snippet:0") if d in directives]
    if bad:
        report("FAIL", "Indexing", f"found {', '.join(bad)} in meta robots / X-Robots-Tag")
    else:
        report("PASS", "Indexing", "no noindex or nosnippet in meta robots or X-Robots-Tag")
    if p.canonical:
        canon = urljoin(url, p.canonical)
        if canon.rstrip("/") == url.rstrip("/"):
            report("PASS", "Indexing", "canonical points to this URL")
        else:
            report("WARN", "Indexing", f"canonical points elsewhere: {canon}")
    else:
        report("WARN", "Indexing", "no canonical tag")

    # Title, description, H1
    if len(p.h1s) == 1:
        report("PASS", "Title and H1", f'one H1: "{p.h1s[0][:70]}"')
    else:
        report("FAIL", "Title and H1", f"{len(p.h1s)} H1s found (want exactly one)")
    title = p.title
    report("PASS" if 30 <= len(title) <= 65 else "WARN", "Title and H1",
           f'title ({len(title)} chars): "{title[:80]}"')
    if p.h1s:
        h1_words, title_words = set(words(p.h1s[0])), set(words(title))
        shared = h1_words & title_words
        share = len(shared) / max(1, len(h1_words))
        report("PASS" if share >= 0.5 else "WARN", "Title and H1",
               f"title and H1 share {len(shared)} of {len(h1_words)} H1 words")
    desc = (p.metas.get("description") or [""])[0]
    if not desc:
        report("WARN", "Title and H1", "no meta description")
    else:
        report("PASS" if len(desc) <= 160 else "WARN", "Title and H1",
               f"meta description {len(desc)} chars")

    # Structured data
    nodes, broken = [], 0
    for block in p.jsonld:
        try:
            walk_jsonld(json.loads(block), nodes)
        except json.JSONDecodeError:
            broken += 1
    if broken:
        report("FAIL", "Structured data", f"{broken} JSON-LD block(s) do not parse")
    types = sorted({t for t, _ in nodes})
    report("PASS" if types else "WARN", "Structured data",
           "JSON-LD types: " + (", ".join(types) if types else "none"))
    orgs = [n for t, n in nodes if t.lower() in ORG_TYPES or t.lower().endswith("business")]
    if orgs:
        same_as = orgs[0].get("sameAs", [])
        same_as = [same_as] if isinstance(same_as, str) else same_as
        report("PASS" if same_as else "WARN", "Structured data",
               f"organization node ({orgs[0].get('@type')}) with {len(same_as)} sameAs link(s)")
    else:
        report("WARN", "Structured data", "no Organization or LocalBusiness node")

    # FAQ answers really in the HTML
    faqs = []
    for t, node in nodes:
        if t == "Question":
            answer = node.get("acceptedAnswer") or {}
            answer = answer[0] if isinstance(answer, list) and answer else answer
            faqs.append((node.get("name", ""), answer.get("text", "") if isinstance(answer, dict) else ""))
    if faqs:
        missing = [q for q, a in faqs if norm(a)[:80] not in page_text]
        status_ = "PASS" if not missing else "WARN"
        msg = f"{len(faqs) - len(missing)} of {len(faqs)} FAQPage answers are in the page HTML"
        if missing:
            msg += "; not in the HTML: " + "; ".join(f'"{q}"' for q in missing)
        report(status_, "FAQ", msg)
    else:
        report("INFO", "FAQ", f"no FAQPage schema; {p.question_headings} question-shaped H2/H3s")
    return page_text


# ---------------------------------------------------------------- sitemap

def check_sitemap(url, sitemaps):
    parts = urlsplit(url)
    root = f"{parts.scheme}://{parts.netloc}"
    queue = sitemaps or [root + "/sitemap.xml", root + "/sitemap_index.xml", root + "/wp-sitemap.xml"]
    target, seen = url.rstrip("/"), 0
    while queue and seen < 60:
        sm = queue.pop(0)
        seen += 1
        status, _, body, _ = fetch(sm)
        if status != 200:
            continue
        try:
            tree = ET.fromstring(body.encode("utf-8"))
        except ET.ParseError:
            continue
        ns = {"s": "http://www.sitemaps.org/schemas/sitemap/0.9"}
        if tree.tag.endswith("sitemapindex"):
            queue += [e.text.strip() for e in tree.findall("s:sitemap/s:loc", ns) if e.text]
            continue
        for u in tree.findall("s:url", ns):
            loc = (u.findtext("s:loc", default="", namespaces=ns) or "").strip()
            if loc.rstrip("/") == target:
                lastmod = u.findtext("s:lastmod", default="", namespaces=ns)
                if lastmod:
                    report("PASS", "Sitemap", f"listed in {sm} (lastmod {lastmod.strip()})")
                else:
                    report("WARN", "Sitemap", f"listed in {sm} but without a lastmod date")
                return
    report("FAIL", "Sitemap", f"not found in {seen} sitemap file(s) checked")


def main():
    if len(sys.argv) != 2:
        sys.exit("usage: python3 geo_audit.py <url>")
    url = sys.argv[1]
    status, headers, html, secs = fetch(url)
    print(f"GEO audit: {url}")
    if status != 200:
        print(f"  page returned {status}: {html[:200]}")
        sys.exit(1)
    print(f"  fetched {status} in {secs:.2f}s, {len(html) / 1024:.1f} KB of HTML")
    report("PASS" if secs < 1 else "WARN", "Indexing", f"server answered in {secs:.2f}s")
    sitemaps = check_crawlers(url)
    check_page(url, status, headers, html)
    check_sitemap(url, sitemaps)

    for sec in ("Crawler access", "Indexing", "Title and H1", "Structured data", "FAQ", "Sitemap"):
        print(f"\n{sec}")
        for st, s, msg in results:
            if s == sec:
                print(f"  {st:<4}  {msg}")
    counts = {s: sum(1 for r in results if r[0] == s) for s in ("PASS", "WARN", "FAIL")}
    print(f"\n{counts['PASS']} pass, {counts['WARN']} warn, {counts['FAIL']} fail")
    sys.exit(1 if counts["FAIL"] else 0)


if __name__ == "__main__":
    main()

GEO audit output on a demo page and a live page

First, a demo. We built a page for a fictional business, Joe's Plumbing, served it locally with python3 -m http.server, and planted two mistakes: the robots.txt group above, which blocks Claude-SearchBot along with the training crawlers, and an FAQ answer ("Do you fix boilers on Sundays?") that exists only in the JSON-LD.

$ python3 geo_audit.py http://localhost:8765/emergency-plumber-leeds/
GEO audit: http://localhost:8765/emergency-plumber-leeds/
  fetched 200 in 0.01s, 2.5 KB of HTML

Crawler access
  INFO  robots.txt 200
  PASS  OAI-SearchBot     allowed: no rule matches (group "*"); UA probe 200
  FAIL  Claude-SearchBot  BLOCKED: "Disallow: /" in group "claude-searchbot"; UA probe 200
  PASS  PerplexityBot     allowed: no rule matches (group "*"); UA probe 200
  PASS  Googlebot         allowed: no rule matches (group "*"); UA probe 200
  PASS  Bingbot           allowed: no rule matches (group "*"); UA probe 200
  INFO  GPTBot            blocked: "Disallow: /" in group "gptbot" (training crawler, your call)
  INFO  ClaudeBot         blocked: "Disallow: /" in group "claudebot" (training crawler, your call)
  INFO  Google-Extended   allowed: no rule matches (group "*") (training crawler, your call)

Indexing
  PASS  server answered in 0.01s
  PASS  no noindex or nosnippet in meta robots or X-Robots-Tag
  PASS  canonical points to this URL

Title and H1
  PASS  one H1: "Emergency plumber in Leeds, open 24/7"
  PASS  title (49 chars): "Emergency Plumber in Leeds, 24/7 | Joe's Plumbing"
  PASS  title and H1 share 5 of 6 H1 words
  PASS  meta description 145 chars

Structured data
  PASS  JSON-LD types: Answer, FAQPage, Plumber, PostalAddress, Question
  PASS  organization node (Plumber) with 2 sameAs link(s)

FAQ
  WARN  2 of 3 FAQPage answers are in the page HTML; not in the HTML: "Do you fix boilers on Sundays?"

Sitemap
  PASS  listed in http://localhost:8765/sitemap.xml (lastmod 2026-10-14)

14 pass, 1 warn, 1 fail

Both planted mistakes show up, and the training-crawler blocks are reported without failing the audit. Note the UA probe 200 next to the blocked Claude-SearchBot: http.server ignores robots.txt, which is the point. robots.txt is a request to crawlers, and the probe shows what the server itself does.

Then we pointed it at the GEO checklist page itself, live on 20 October 2026:

$ python3 geo_audit.py https://www.lovedby.ai/blog/geo-checklist
GEO audit: https://www.lovedby.ai/blog/geo-checklist
  fetched 200 in 0.19s, 182.8 KB of HTML

Crawler access
  INFO  robots.txt 200
  PASS  OAI-SearchBot     allowed: "Allow: /" in group "oai-searchbot"; UA probe 200
  PASS  Claude-SearchBot  allowed: "Allow: /" in group "*"; UA probe 200
  PASS  PerplexityBot     allowed: "Allow: /" in group "perplexitybot"; UA probe 200
  PASS  Googlebot         allowed: "Allow: /" in group "*"; UA probe 200
  PASS  Bingbot           allowed: "Allow: /" in group "*"; UA probe 200
  INFO  GPTBot            allowed: "Allow: /" in group "gptbot" (training crawler, your call)
  INFO  ClaudeBot         allowed: "Allow: /" in group "claudebot" (training crawler, your call)
  INFO  Google-Extended   allowed: "Allow: /" in group "google-extended" (training crawler, your call)

Indexing
  PASS  server answered in 0.19s
  PASS  no noindex or nosnippet in meta robots or X-Robots-Tag
  PASS  canonical points to this URL

Title and H1
  PASS  one H1: "GEO Checklist: 30 On-Page Checks for AI Search"
  PASS  title (58 chars): "GEO Checklist: 30 On-Page Checks for AI Search | LovedByAI"
  PASS  title and H1 share 7 of 7 H1 words
  PASS  meta description 152 chars

Structured data
  PASS  JSON-LD types: Answer, BlogPosting, Brand, BreadcrumbList, CollegeOrUniversity, ContactPoint, EntryPoint, FAQPage, ImageObject, ListItem, Organization, Person, Place, PostalAddress, QuantitativeValue, Question, SearchAction, WebPage, WebSite
  PASS  organization node (Organization) with 10 sameAs link(s)

FAQ
  PASS  6 of 6 FAQPage answers are in the page HTML

Sitemap
  PASS  listed in https://www.lovedby.ai/sitemap/blog.xml (lastmod 2026-10-20T00:00:00.000Z)

16 pass, 0 warn, 0 fail

What can't a GEO audit script tell you?

The part that decides most citations. A script can confirm a page has one H1; it can't tell you whether the first two sentences name the business, the service and the place in the customer's own words. Wording matters here: in our test of 50 local questions on 13 October 2026, 117 of the 135 businesses ChatGPT recommended for a specific need were described with exactly that need.

It also can't see off-page signals. In the same test, 283 of the 284 local businesses ChatGPT recommended came with a listing card or a Google Maps link, so an up-to-date Google Business Profile matters as much as anything in the HTML.

[[VIDEO: planned youtube video | How to rank in ChatGPT: the human half of a GEO audit, the 14 parts of a small business page that a script can't judge, shown on one example page in under seven minutes.]]

And it can't tell you whether any of it worked. For that, measure AI referral visits per page: ChatGPT adds utm_source=chatgpt.com to its links, so Google Analytics 4 shows those visits by landing page, and Google counts AI Overviews and AI Mode traffic in Search Console's Performance report under the Web search type.

LovedByAI runs these checks across a whole WordPress site rather than one URL at a time, automates the on-page fixes (question headings, answer-first sections, FAQs with FAQPage schema, Organization schema), and reports AI mentions and AI referral traffic back. This script is the one-URL version you can read end to end.

For the business owner's side of this, without the code: SEO for AI search: what to change on a small business page, top to bottom.

The full 30-point checklist with sources: https://www.lovedby.ai/blog/geo-checklist

utm_source=chatgpt.com: how to see which pages ChatGPT visitors land on (GA4 API and server logs)

LovedByAI — Tue, 20 Oct 2026 19:54:04 +0000

If you've spotted utm_source=chatgpt.com in your analytics or your access logs, you've found the people ChatGPT sends you. The useful next question is where they land, and on most small business sites we measured, the answer isn't mainly the homepage.

We build LovedByAI, a generative engine optimization platform for small businesses built around WordPress, and its plugin logs every ChatGPT visit with the page it lands on. Across 23,740 ChatGPT visits to 323 small business websites between 14 September and 13 October 2026 (sites with at least 10 ChatGPT visits, each site counted once), the homepage took 42% of ChatGPT's visitors on the average service business site and 23% on the average online store, where product pages took 37%. The full study is ChatGPT referral traffic: where visitors land.

This post covers how to get the same list for your own site, or for every client site you look after: from the GA4 interface, from the GA4 Data API, and from raw access logs, with a script we ran before publishing.

What does utm_source=chatgpt.com mean?

utm_source=chatgpt.com is a query parameter ChatGPT appends to the links in its answers. When a person clicks one, the request for your page carries it, and Google Analytics 4 records the visit with the session source chatgpt.com. The browser usually sends https://chatgpt.com/ as the referrer too.

Three things that look similar but aren't the same:

A ChatGPT-User request is not a visitor. That's OpenAI's fetcher reading a page during a conversation. It shows up in logs with a bot user agent and no tag.
Paid clicks can carry the tag too. A link tagged utm_medium=paid or cpc is an ad click, not an answer. The study left those out, along with two other ad-click parameters we saw in our data (oppref and olref), and the script below does the same.
Some organic ChatGPT clicks now carry nothing. Since 8 October 2026, part of ChatGPT's traffic arrives with no tag and no referrer, so GA4 files it under Direct and a log can't tell it from a typed URL. Everything below is a sample, not a total.

How to find ChatGPT landing pages in the GA4 interface

Open Reports, then Engagement, then Landing page.
Add filter: dimension Session source, exactly matches, chatgpt.com.
Set the range to the last 90 days and sort by Sessions.

GA4's default channel group now also has an AI Assistant channel (Google's channel definitions), and in our own property ChatGPT's tagged sessions show up there with the medium ai-assistant. We still filter on Session source. Google's rule puts a session in that channel when its medium is ai-assistant, which GA4 sets when the referrer matches its list of assistants, and in our property some chatgpt.com sessions were filed under Organic Search instead. The source is what the tag sets directly.

How to pull ChatGPT landing pages with the GA4 Data API

The same report as a runReport request. POST it to https://analyticsdata.googleapis.com/v1beta/properties/<PROPERTY_ID>:runReport (the "Try this method" panel on that page works, as does any Google client library):

{
  "dateRanges": [{ "startDate": "90daysAgo", "endDate": "yesterday" }],
  "dimensions": [{ "name": "landingPage" }],
  "metrics": [{ "name": "sessions" }],
  "dimensionFilter": {
    "filter": {
      "fieldName": "sessionSource",
      "stringFilter": { "matchType": "EXACT", "value": "chatgpt.com" }
    }
  },
  "orderBys": [{ "metric": { "metricName": "sessions" }, "desc": true }],
  "limit": 1000
}

We checked this request against the Data API on our own GA4 property before publishing. Two things to expect in the response: landingPage is the path without the query string (use landingPagePlusQueryString if you want it), and some rows come back as (not set), sessions GA4 couldn't attach a page to. Save the response as a .json file; the script reads it as is, in either the REST shape (dimensionValues) or the client library's (dimension_values).

How to find ChatGPT visits in server logs

A ChatGPT visit in a combined-format access log is a request whose query string carries utm_source=chatgpt.com, or whose referrer is chatgpt.com. The second page of the same visit has your own site as its referrer, so it doesn't match, which means each match is a landing. These demo lines (invented, from the documentation IP ranges) show what counts and what doesn't:

203.0.113.10 - - [06/Oct/2026:08:01:12 +0000] "GET /?utm_source=chatgpt.com HTTP/1.1" 200 18233 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 18_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.6 Mobile/15E148 Safari/604.1"
192.0.2.80 - - [06/Oct/2026:12:45:00 +0000] "GET /contact/ HTTP/1.1" 200 9001 "https://chatgpt.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36"
192.0.2.91 - - [06/Oct/2026:13:20:33 +0000] "GET /?utm_source=chatgpt.com&utm_medium=cpc HTTP/1.1" 200 18233 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36"
203.0.113.10 - - [06/Oct/2026:08:02:40 +0000] "GET /services/burst-pipe-repair/ HTTP/1.1" 200 15002 "https://demo-plumber.example/?utm_source=chatgpt.com" "Mozilla/5.0 (iPhone; CPU iPhone OS 18_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.6 Mobile/15E148 Safari/604.1"
192.0.2.150 - - [06/Oct/2026:14:40:02 +0000] "GET /services/burst-pipe-repair/ HTTP/1.1" 200 15002 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot"

In order: a tagged landing on the homepage (counted); a landing on /contact/ with only the referrer (counted); a paid click (skipped); the visitor from the first line moving to a second page, with your own site as referrer (skipped); ChatGPT-User fetching a page (skipped, it's a bot).

Two caveats for production logs. If a CDN serves cached HTML, those hits never reach your origin log, so read the CDN's logs instead. And a reload logs twice, where GA4 counts one session.

How to split ChatGPT landing pages by page type with a Python script

chatgpt_landings.py takes one or more sources, one per website: an access log, or a saved GA4 runReport response. For each it prints the split by page type and the top pages, then compares the average across sites with the pooled total. Standard library only, Python 3.8 or later.

#!/usr/bin/env python3
"""Where do ChatGPT's visitors land? One split per website.

    python3 chatgpt_landings.py SOURCE [SOURCE ...]

Each SOURCE is one website, either:
  - an access log in the combined format (the nginx and Apache default), or
  - a saved GA4 Data API runReport response (.json) with landingPage as the
    first dimension and sessions as the first metric.

Prints each site's split by page type and its top pages, then the average
across sites (every site counts once) next to the pooled split.
Standard library only, Python 3.8 or later.
"""
import json
import re
import sys
from collections import Counter
from urllib.parse import parse_qs, unquote, urlsplit

LINE = re.compile(
    r'(?P<ip>\S+) \S+ \S+ \[[^\]]+\] "(?P<method>\S+) (?P<target>\S+)[^"]*" '
    r'(?P<status>\d{3}) \S+ "(?P<referer>[^"]*)" "(?P<ua>[^"]*)"'
)
CHATGPT_HOSTS = {"chatgpt.com", "chat.openai.com"}
# Clicks on paid placements: the study left these out.
PAID = re.compile(r"[?&]utm_medium=(paid|cpc|ads?)(&|$)|[?&](oppref|olref)=", re.I)
NOT_A_PERSON = re.compile(r"bot|crawler|spider|ChatGPT-User", re.I)
ASSET = re.compile(r"\.(css|js|map|png|jpe?g|gif|webp|svg|ico|woff2?)$", re.I)

# Rough URL rules, checked in order. The study typed pages with an AI model;
# edit these to match your own site's URL structure.
RULES = [
    ("homepage", r"^/([a-z]{2}(-[a-z]{2})?/)?$"),
    ("product", r"^/(product|products|item)/"),
    ("category", r"^/(product-category|shop|collections?)(/|$)"),
    ("blog post", r"^/(blog|news|articles?|guides?)/.+|^/\d{4}/\d{2}/"),
    ("service", r"^/(services?|treatments?|practice-areas?)/.+"),
    ("location", r"^/(locations?|areas?|service-areas?)/.+"),
    ("about", r"^/(about|about-us|team|our-story)/?$"),
    ("contact/booking", r"^/(contact|contact-us|book|booking|appointments?|quote|get-a-quote)/?$"),
    ("pricing", r"^/(pricing|prices|rates|plans)/?$"),
]
RULES = [(name, re.compile(rx, re.I)) for name, rx in RULES]


def page_type(path):
    for name, rx in RULES:
        if rx.search(path):
            return name
    return "other"


def from_chatgpt(target, referer):
    """A person who clicked a link in a ChatGPT answer (tag or referrer)."""
    if "chatgpt.com" in parse_qs(urlsplit(target).query).get("utm_source", []):
        return True
    return (urlsplit(referer).hostname or "") in CHATGPT_HOSTS


def read_log(path):
    pages, dead = Counter(), Counter()
    with open(path, encoding="utf-8", errors="replace") as log:
        for raw in log:
            m = LINE.match(raw)
            if not m or m["method"] != "GET" or NOT_A_PERSON.search(m["ua"]):
                continue
            page = unquote(urlsplit(m["target"]).path).lower() or "/"
            if ASSET.search(page) or PAID.search(m["target"]):
                continue
            if from_chatgpt(m["target"], m["referer"]):
                pages[page] += 1
                if m["status"] == "404":
                    dead[page] += 1
    return pages, dead, 0


def read_ga4(path):
    """A runReport response, REST (camelCase) or client-library (snake_case)."""
    report = json.load(open(path, encoding="utf-8"))
    pages, not_set = Counter(), 0
    for row in report.get("rows", []):
        dims = row.get("dimensionValues") or row.get("dimension_values")
        mets = row.get("metricValues") or row.get("metric_values")
        page, sessions = dims[0]["value"], int(mets[0]["value"])
        if page == "(not set)":
            not_set += sessions
            continue
        pages[page.lower()] += sessions
    return pages, Counter(), not_set


def split(pages):
    by_type = Counter()
    for page, n in pages.items():
        by_type[page_type(page)] += n
    total = sum(by_type.values())
    return {t: n / total for t, n in by_type.items()}, total


def main(sources):
    shares, pooled = [], Counter()
    for src in sources:
        pages, dead, not_set = read_ga4(src) if src.endswith(".json") else read_log(src)
        if not pages:
            print(f"\n{src}: no ChatGPT visits found")
            continue
        site, total = split(pages)
        shares.append(site)
        for page, n in pages.items():
            pooled[page_type(page)] += n
        print(f"\n{src}: {total} ChatGPT landings")
        for t, s in sorted(site.items(), key=lambda kv: -kv[1]):
            print(f"  {100 * s:5.1f}%  {t}")
        top = pages.most_common(3)
        print(f"  top 3 pages take {100 * sum(n for _, n in top) / total:.0f}%: "
              + ", ".join(p for p, _ in top))
        if not_set:
            print(f"  ({not_set} sessions with landing page '(not set)' left out)")
        if dead:
            print(f"  landed on a 404: {sum(dead.values())} ({', '.join(dead)})")

    if len(shares) > 1:
        types = sorted({t for s in shares for t in s}, key=lambda t: -sum(s.get(t, 0) for s in shares))
        pooled_total = sum(pooled.values())
        print(f"\nAcross {len(shares)} sites    average of sites   pooled visits")
        for t in types:
            avg = sum(s.get(t, 0) for s in shares) / len(shares)
            print(f"  {t:17s}  {100 * avg:14.1f}%  {100 * pooled[t] / pooled_total:13.1f}%")


if __name__ == "__main__":
    if len(sys.argv) < 2:
        sys.exit(__doc__)
    main(sys.argv[1:])

The page types come from rough URL rules at the top. Our study sorted 7,211 landing pages with an AI model instead; for one site, editing RULES to match your URL structure is quicker and good enough.

We ran it on two demo sites: an access log from an invented plumber (plumber-access.log, 12 lines) and a saved GA4 response from an invented shop (shop-ga4-runreport.json, 8 rows):

$ python3 chatgpt_landings.py demo/plumber-access.log demo/shop-ga4-runreport.json

demo/plumber-access.log: 7 ChatGPT landings
   28.6%  homepage
   28.6%  service
   14.3%  blog post
   14.3%  contact/booking
   14.3%  other
  top 3 pages take 71%: /, /services/burst-pipe-repair/, /blog/how-to-shut-off-your-water-main/
  landed on a 404: 1 (/emergency-plumber-24-7/)

demo/shop-ga4-runreport.json: 125 ChatGPT landings
   50.4%  product
   24.0%  homepage
   12.0%  category
    9.6%  blog post
    2.4%  about
    1.6%  contact/booking
  top 3 pages take 74%: /product/linen-apron/, /, /product/canvas-tote/
  (4 sessions with landing page '(not set)' left out)

Across 2 sites    average of sites   pooled visits
  homepage                     26.3%           24.2%
  product                      25.2%           47.7%
  service                      14.3%            1.5%
  blog post                    11.9%            9.8%
  contact/booking               7.9%            2.3%
  other                         7.1%            0.8%
  category                      6.0%           11.4%
  about                         1.2%            2.3%

The plumber's 12 log lines become 7 landings: the paid click, the Google visit, the stylesheet, the second page of a visit and the ChatGPT-User fetch all drop out. The 404 is a link ChatGPT handed out to a page that doesn't exist, which is a separate problem we covered in why ChatGPT sends visitors to broken links.

Why should you average ChatGPT landing pages per site instead of pooling visits?

If you look after several client sites, the last table is the one to read. Pooled, the shop's 125 sessions swamp the plumber's 7, and product pages look like 47.7% of ChatGPT traffic. Averaged per site, they're 25.2%. Neither site looks like the pooled number.

The study hit the same thing at scale. Across the 333 sites with at least 10 ChatGPT visits between 14 September and 13 October 2026, the five biggest held 26% of all visits. Pooled, listing pages (directory entries) came out at 16.4% of ChatGPT traffic, mostly from one directory; averaged per site, 2.2%. Pooling also pushed the homepage down to 22.7%, against 32.7% when every site counts once.

Medians don't rescue it either. A typical site gets ChatGPT visitors on only three or four kinds of page, so the median share for most page types is zero and can't rank anything. Use the per-site average, plus a count of how many sites had any visit on each type.

What does a normal ChatGPT landing page split look like?

Benchmarks from LovedByAI's data, per-site averages, 14 September to 13 October 2026, sites with at least 10 ChatGPT visits. "Service business" means no ChatGPT visitor landed on a product, category or listing page; "online store" means at least one landed on a product or category page.

Page type	Service business (175 sites)	Online store (148 sites)
Homepage	42%	23%
Product	0%	37%
Product category	0%	13%
Service	18%	6%
Blog post	18%	11%
About	4%	2%
Location	4%	1%
Contact or booking	3%	1%
Listing	0%	2%
Pricing	1.4%	0.3%
Other	10%	4%

The spread matters as much as the split. On the median service business site, ChatGPT visitors landed on 7 different pages in the month, and the top 3 took 78% of them. On the median store it was 16 pages, with the top 3 taking 53%. And blog posts are lopsided: on 47% of the service sites no ChatGPT visitor landed on a blog post, while on 14% blog posts took half or more.

If your own split is far from these, that's not a problem in itself. It tells you which pages ChatGPT uses to introduce you, and those are the pages to look at first.

[[VIDEO: planned youtube video | How to track AI traffic in GA4: the landing page report for ChatGPT visitors, the AI Assistant channel, and what 23,740 ChatGPT visits say you'll find.]]

What can't utm_source=chatgpt.com data tell you?

What visitors did next. Landing pages only. Nothing here says one page type converts better than another.
The untagged share. Since 8 October 2026 some ChatGPT visits carry no tag and no referrer. Neither GA4 nor a log can attribute them, and we can't check whether they land on the same pages.
Perfect page types. The study typed pages from the URL with an AI model, and the script uses regex rules. Both will misfile some pages.
Every kind of site. The benchmarks come from WordPress sites running our plugin, mostly small businesses, with at least 10 ChatGPT visits in the month.

For the owner's side of this, without the code: ChatGPT traffic: why most of it lands past your homepage.

LovedByAI's plugin keeps this log on every WordPress site it runs on, which is where the 23,740 visits above come from, and the platform automates the on-page optimization across those pages and the rest of the site. If you're not on WordPress, the script and the GA4 request get you the same list by hand.

Original study and full dataset: https://www.lovedby.ai/blog/chatgpt-referral-traffic

ChatGPT-User in your logs: how to tell real ChatGPT traffic from a scanner using its name

LovedByAI — Sat, 10 Oct 2026 20:29:35 +0000

If you grep your access logs for ChatGPT-User to see what ChatGPT can't find on your site, most of what comes back won't be ChatGPT.

We looked at one day of bot traffic, Wednesday 7 October 2026, across the small business websites running LovedByAI, a generative engine optimization platform for small-business WordPress sites. Requests carrying the ChatGPT-User user agent got a 404 1,022 times across 111 sites, and 703 of those (69%) were exploit probes, not ChatGPT. A vulnerability scanner is wearing the name.

Meanwhile the 404s that do matter, real people ChatGPT sent to pages that don't exist, never show up under that user agent at all. This post covers how to tell the two apart, with a script you can run on your own logs. The full study behind the numbers is ChatGPT broken links: why visitors land on 404s.

What is the ChatGPT-User user agent?

ChatGPT-User is the user agent OpenAI uses when ChatGPT fetches a page because a person asked it something mid-conversation. It's one of three OpenAI crawlers, and they do different jobs:

User agent	What it does	OpenAI's published IP ranges
`GPTBot`	Collects training data	`https://openai.com/gptbot.json`
`OAI-SearchBot`	Builds ChatGPT's search index	`https://openai.com/searchbot.json`
`ChatGPT-User`	Fetches a page for a user, on demand	`https://openai.com/chatgpt-user.json`

A ChatGPT-User request looks like this in a combined-format log (a demo line, using an address from OpenAI's list):

104.208.184.197 - - [07/Oct/2026:10:14:02 +0000] "GET /services/ HTTP/1.1" 200 512 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot"

The user agent string is trivially easy to copy. The IP ranges are not, which is why OpenAI publishes them.

How many ChatGPT-User 404s are fake?

On 7 October 2026, 5.8% of the 823,928 bot requests to small business websites in our logs got a 404. For user agents claiming to be an AI assistant fetching a page for a user (ChatGPT-User, Claude-User, Perplexity-User and the like), the 404 rate was 43.7% (4,597 of 10,528). For AI search crawlers it was 10.0%, search engines 7.2% and AI training crawlers 4.4%.

The gap is the scanner. Of the 1,022 ChatGPT-User 404s that day, 703 matched a strict exploit pattern:

/cgi-bin/php-cgi with an allow_url_include argument, the PHP-CGI argument injection tracked as CVE-2024-4577
.env files
/proc/self reads and path traversal
Vite dev server /@fs/ file reads
Kubernetes service account tokens

Most of the remainder asked for config and debug files (runtime-config.js, /debug/pprof) that no small business website serves. The same probes also arrive dressed as GrokBot, MistralAI-User and DuckAssistBot. Some examples, as they appear in a log (demo lines, addresses from the documentation ranges):

203.0.113.24 - - [07/Oct/2026:10:14:02 +0000] "GET /cgi-bin/php-cgi.exe?%ADd+allow_url_include%3d1+%ADd+auto_prepend_file%3dphp://input HTTP/1.1" 404 512 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot"
203.0.113.24 - - [07/Oct/2026:10:14:02 +0000] "GET /.env HTTP/1.1" 404 512 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot"
198.51.100.7 - - [07/Oct/2026:10:14:02 +0000] "GET /@fs/etc/passwd?raw HTTP/1.1" 404 512 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot"
198.51.100.7 - - [07/Oct/2026:10:14:02 +0000] "GET /var/run/secrets/kubernetes.io/serviceaccount/token HTTP/1.1" 404 512 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot"

On a WordPress site these should all 404, which is the correct answer. The problem is only in the reporting: count them as ChatGPT and your "ChatGPT can't find our pages" number is mostly noise.

How do you verify a ChatGPT-User request?

Check the client IP against OpenAI's published ranges. This script does that, separates real ChatGPT-User fetches from fakes, and then does the more useful job: it finds the human visits ChatGPT sent that landed on a 404.

#!/usr/bin/env python3
"""Split an access log's ChatGPT traffic into what's real and what isn't.

    python3 chatgpt_404s.py access.log

Reads the combined log format (nginx and Apache default) and downloads
OpenAI's published ChatGPT-User IP ranges. Standard library only.
"""
import ipaddress
import json
import re
import sys
import urllib.request
from collections import Counter
from urllib.parse import parse_qs, urlsplit

RANGES_URL = "https://openai.com/chatgpt-user.json"

LINE = re.compile(
    r'(?P<ip>\S+) \S+ \S+ \[[^\]]+\] "(?P<method>\S+) (?P<path>\S+)[^"]*" '
    r'(?P<status>\d{3}) \S+ "(?P<referer>[^"]*)" "(?P<ua>[^"]*)"'
)

# Paths no small business website serves. These are what the scanner wearing
# the ChatGPT-User name asked for in our logs.
PROBE = re.compile(
    r"php-cgi|allow_url_include|/\.env|/proc/self|/@fs/|serviceaccount/token"
    r"|\.\./|%2e%2e|/debug/pprof|runtime-config\.js",
    re.IGNORECASE,
)


def load_ranges(url=RANGES_URL):
    req = urllib.request.Request(url, headers={"User-Agent": "chatgpt-404s/1.0"})
    with urllib.request.urlopen(req, timeout=20) as resp:
        prefixes = json.load(resp)["prefixes"]
    return [ipaddress.ip_network(p.get("ipv4Prefix") or p["ipv6Prefix"]) for p in prefixes]


def from_openai(ip, nets):
    try:
        addr = ipaddress.ip_address(ip)
    except ValueError:
        return False
    return any(addr in net for net in nets if net.version == addr.version)


def sent_by_chatgpt(path, referer):
    """A person clicking a link in a ChatGPT answer, not a bot."""
    sources = parse_qs(urlsplit(path).query).get("utm_source", [])
    return "chatgpt.com" in sources or urlsplit(referer).hostname in ("chatgpt.com", "chat.openai.com")


def main(log_path):
    nets = load_ranges()
    bots = Counter()
    bot_404s = Counter()
    visitor_404s = Counter()
    visitors = 0

    with open(log_path, encoding="utf-8", errors="replace") as log:
        for raw in log:
            m = LINE.match(raw)
            if not m:
                continue
            page = urlsplit(m["path"]).path
            if "ChatGPT-User" in m["ua"]:
                if not from_openai(m["ip"], nets):
                    bots["fake (not from OpenAI's IP ranges)"] += 1
                    if PROBE.search(m["path"]):
                        bots["  of which exploit probes"] += 1
                else:
                    bots["real ChatGPT-User"] += 1
                    if m["status"] == "404":
                        bot_404s[page] += 1
            elif sent_by_chatgpt(m["path"], m["referer"]):
                visitors += 1
                if m["status"] == "404":
                    visitor_404s[page] += 1

    print("ChatGPT-User requests")
    for label, n in bots.items():
        print(f"  {n:>5}  {label}")
    print("\nPages real ChatGPT-User fetched that returned 404")
    for page, n in bot_404s.most_common(20):
        print(f"  {n:>5}  {page}")
    lost = sum(visitor_404s.values())
    print(f"\nVisitors ChatGPT sent you: {visitors}, landed on a 404: {lost}")
    for page, n in visitor_404s.most_common(20):
        print(f"  {n:>5}  {page}")


if __name__ == "__main__":
    main(sys.argv[1])

Standard library only, Python 3.8 or later. Run it against a log:

python3 chatgpt_404s.py /var/log/nginx/access.log

On a small demo log (invented lines: one real ChatGPT-User address from OpenAI's list, two fake ones from the documentation ranges, and a handful of visitors), it prints:

ChatGPT-User requests
      2  real ChatGPT-User
      6  fake (not from OpenAI's IP ranges)
      5    of which exploit probes

Pages real ChatGPT-User fetched that returned 404
      1  /experiences/

Visitors ChatGPT sent you: 5, landed on a 404: 3
      2  /contact-us/
      1  /product/discontinued-perfume/

Three things to know before you trust it on production logs:

Behind Cloudflare or another proxy, log the real client IP. Otherwise every request comes from the proxy's addresses and everything looks fake. On nginx that means real_ip_header CF-Connecting-IP with Cloudflare's ranges in set_real_ip_from.
Fetch the ranges fresh. OpenAI updates chatgpt-user.json; the copy we fetched on 10 October 2026 was dated 7 October.
Gzipped, rotated logs need decompressing first (zcat access.log.2.gz > access.log.2).

How do you find the 404s real ChatGPT visitors hit?

Look for the click, not the bot. When a person clicks a link in a ChatGPT answer, their browser makes the request, with a normal browser user agent, and ChatGPT adds utm_source=chatgpt.com to the URL (the referrer is often https://chatgpt.com/ too). That's what sent_by_chatgpt() matches.

Those are the ones that cost you something. Across 323 small business websites that received a ChatGPT visit between 30 September and 9 October 2026, 36 of 4,053 ChatGPT visits (about 1 in 113) landed on a 404. When we rechecked each address on 10 October against the live site and the Wayback Machine's CDX index, they split like this:

The four "garbled" ones are worth a developer's attention: the page existed, but the link ChatGPT wrote had its percent-encoding doubled (%25 where a % belonged), one byte changed, or bytes missing from the middle so it was no longer valid UTF-8. Long, encoded slugs are the ones a model breaks, because it writes a URL one token at a time instead of copying it.

What should each kind of ChatGPT 404 return?

What the 404 is	What to serve
Page moved (old URL structure, renamed slug)	301 to the new address
Product, listing or job removed	301 to the replacement or the category, or keep a "no longer available" page with alternatives. Not the homepage: Google treats a mass redirect to the homepage as a soft 404
Address ChatGPT invented that keeps getting visits	Build a page there, or 301 to the closest real one
Garbled version of a real URL	301 the broken variant to the real page
Spam or test pages that should stay gone	410 Gone
Exploit probes (`php-cgi`, `.env`, `/@fs/`)	Leave the 404, or block at the WAF

A threshold that keeps this manageable: redirect a dead address once it has had more than two or three human visits and you have a live page that answers the same need, or immediately if other sites link to it.

The same fixes, walked through in under five minutes:

What we couldn't prove

Our 69% is by path, not by IP. We classified the 7 October probes by what they requested; we didn't check each one's address against OpenAI's ranges. The script above is what we recommend, not how the study counted.
Ten days is a first read. 36 dead-page visits shows the pattern, not a precise rate.
CDN redirects are invisible to us. Redirects done by Cloudflare or the web server never reach WordPress, so a site that fixed things there looks clean.

For the owner's side of this, without the code, our Medium piece walks through it: Dead links from ChatGPT: it's still sending customers to pages you deleted.

LovedByAI's plugin records the status code of every AI referral visit on WordPress sites, which is where the 4,053 visits above come from. If you're not on WordPress, the script gets you the same list from raw logs.

Original study and full dataset: ChatGPT broken links: why visitors land on 404s

How to measure AI crawler traffic in your server logs (and why the average lies)

LovedByAI — Wed, 07 Oct 2026 18:27:23 +0000

If you look after websites, AI crawlers are probably already your busiest bots, and almost nothing in your analytics shows them. Bots don't run JavaScript, so Google Analytics never sees them. You have to read the server logs.

We do this every day across the sites running LovedByAI, a generative engine optimization platform for small-business WordPress sites. On 924 of them, between 6 September and 5 October 2026, the median site got 3.1 AI crawler requests for every Googlebot request. Here's how to measure the same thing on your own sites, and the mistake that makes most published numbers look much bigger than what a typical site sees.

How do you identify AI crawlers in server logs?

By user agent token. Every major AI company publishes the token its crawler sends. The ones that matter, with what each operator says it's for:

import re
from collections import Counter
from statistics import median

# User-agent token -> (company, purpose). Purpose follows each operator's docs.
AI_CRAWLERS = {
    "Meta-ExternalAgent": ("Meta", "training"),
    "Meta-WebIndexer":    ("Meta", "search"),
    "ClaudeBot":          ("Anthropic", "training"),
    "Claude-SearchBot":   ("Anthropic", "search"),
    "Claude-User":        ("Anthropic", "user"),
    "GPTBot":             ("OpenAI", "training"),
    "OAI-SearchBot":      ("OpenAI", "search"),
    "ChatGPT-User":       ("OpenAI", "user"),
    "PerplexityBot":      ("Perplexity", "search"),
    "Perplexity-User":    ("Perplexity", "user"),
    "Amazonbot":          ("Amazon", "training"),
    "Bytespider":         ("ByteDance", "training"),
    "CCBot":              ("Common Crawl", "training"),
}

UA = re.compile(r'"[^"]*" \d{3} \S+ "[^"]*" "([^"]*)"$')  # combined log format

def classify(user_agent):
    ua = user_agent.lower()
    if "googlebot" in ua:
        return "Googlebot"
    for token in AI_CRAWLERS:
        if token.lower() in ua:
            return token
    return None

def count_site(lines):
    counts = Counter()
    for line in lines:
        m = UA.search(line.rstrip())
        if m and (bot := classify(m.group(1))):
            counts[bot] += 1
    return counts

def ai_per_googlebot(counts):
    ai = sum(n for bot, n in counts.items() if bot in AI_CRAWLERS)
    return ai / counts["Googlebot"] if counts["Googlebot"] else None

A few tokens look like crawlers but aren't. Google-Extended and Applebot-Extended are robots.txt switches with no bot behind them, so they never appear in logs. anthropic-ai and Claude-Web are retired. Don't count any of them.

Which AI crawlers are for training and which answer questions?

The purpose column matters more than the volume. On the average site in our data, 69.9% of AI crawler requests were for training, 18.8% for AI search indexes and 11.3% were fetches triggered by a person asking an assistant a question.

If you're reporting to a client, split them. "Meta crawled you 200 times" and "ChatGPT fetched your pricing page while someone was choosing a supplier" are not the same news.

Why is the average AI crawler share misleading?

Because AI crawling is extremely concentrated. In our logs, the busiest 1% of sites received between 75% and 87% of the requests from each of the four biggest training crawlers. For Googlebot that figure is 42%.

So if you add every request across your sites together and divide, a couple of hammered sites decide the answer. Here's the effect on three toy sites, two normal and one crawled hard by a single bot:

sites = [count_site(lines) for lines in (site_a, site_b, site_c)]
ratios = [r for r in map(ai_per_googlebot, sites) if r is not None]
ai_total = sum(n for s in sites for bot, n in s.items() if bot in AI_CRAWLERS)
google_total = sum(s["Googlebot"] for s in sites)

print(median(ratios))           # 3.0   what a typical site sees
print(ai_total / google_total)  # 58.6  what the "total" says

Compute the ratio per site first, then take the median across sites. That's what every headline number in our data does. Pooled across all 924 sites, AI crawlers were 63% of identified bot traffic; per site, it was 40.7%.

What can't your server logs see?

Three blind spots worth knowing before you trust any of this.

Cached requests. If a page cache or CDN answers the request, WordPress never sees it. Every count from origin logs is a floor.

Spoofed user agents. Anyone can send GPTBot in a header. In our logs more than 99% of requests from the big crawlers carried each operator's exact published format, which rules out crude fakes. To go further, check the source IP against the ranges OpenAI, Anthropic and others publish.

Visitors that look like bots. A person clicking through from ChatGPT arrives as a normal browser, usually tagged utm_source=chatgpt.com. Count those as referrals, not crawls, and keep the two apart.

What does AI crawler traffic look like on a typical small business site?

From the same 924 sites, 6 September to 5 October 2026:

About 1,050 AI crawler page requests on the median site, roughly 35 a day, from 11 different AI crawlers. Googlebot: 345.
AI crawlers out-requested Googlebot on 79% of sites.
Meta-ExternalAgent was the most active single AI crawler, at 21.6% of the average site's AI requests.
OpenAI's crawlers made 33.7 page requests for every visitor ChatGPT sent, on the median site with at least one ChatGPT visit.

The full tables, the industry split and the methodology are in the AI crawler statistics post. If you'd rather not build this yourself, LovedByAI's Bot Activity and AI Analytics screens show the same split for each WordPress site automatically.

Server logs from 924 websites running the LovedByAI WordPress plugin, 6 September to 5 October 2026. No site is identified.

How to check if your site is blocking AI crawlers (robots.txt won't tell you)

LovedByAI — Wed, 23 Sep 2026 20:32:49 +0000

We build LovedByAI, a generative engine optimization platform for small business WordPress sites, which means we run crawler-access checks on live sites all day. In August we pointed that at 314 of them, fetching each one as 38 different AI crawlers and comparing the result against four weeks of their server logs.

One finding is a genuine observability trap, and I have not seen it written up anywhere.

A site blocking OAI-SearchBot at the origin logged a page request from it 84.6% of the time. A site with no block at all logged one 87.3% of the time.

Three points apart. So if you are checking your own logs to find out whether AI crawlers can reach you, a working block and no block look the same.

Reading robots.txt does not help either, and that turns out to be the more common failure: of those 314 sites, 135 refused at least one AI crawler and only 23 had written a rule about it.

That is the gap, as our own crawlability view reports it: what robots.txt declares on one side, what the server actually answered on the other. A site can carry one of each at the same time, and only one of them is something a human ever typed.

Here is why that happens, and how to test it properly.

Edge vs origin: where an AI crawler block lives

A block at the edge happens before the request reaches your server. A block at the origin happens after. That single distinction decides everything about what you can observe.

Edge. Cloudflare, a CDN firewall rule, a WAF. The crawler's request is terminated at the network boundary. Your application never runs. Your access log never gets a line. The crawler gets a 403 and leaves.

Origin. A security plugin, an nginx map on $http_user_agent, an Apache RewriteCond. The request completes a TCP handshake, hits your server, gets logged, then gets refused.

So the origin-blocked request appears in your logs as a visit. Your crawler-stats plugin counts it. Your dashboard shows GPTBot stopping by.

It left with a 403 and nothing else.

Here's the same comparison for the edge case:

OAI-SearchBot blocked at the origin: logged on 84.6% of sites
OAI-SearchBot not blocked at all: logged on 87.3% of sites
OAI-SearchBot blocked at the edge: logged on 0.0% of sites

The edge column is the only one that's legible. Zero requests, because nothing ever arrived. An edge block is loud in your logs by being completely silent. An origin block hides in plain sight.

Why robots.txt will not tell you if AI crawlers are blocked

The obvious instinct is to skip logs and read robots.txt. That's worse.

Of the 135 sites refusing a crawler, only 23 declared it. The other 112 refused at the server with nothing in the site's own config to show for it. The request arrives, the server says 403, and robots.txt still reads like a welcome mat.

And a written rule isn't reliable in the other direction either, because robots.txt is a convention, not an enforcement mechanism:

33.3% of sites disallowing GPTBot still logged real GPTBot page requests
17.6% of sites disallowing ClaudeBot still logged ClaudeBot

Neither figure counts robots.txt or sitemap fetches, which aren't ingested. Every request counted is a crawler asking for a page it was told not to take.

So robots.txt tells you what you meant. Your logs tell you what arrived. Neither tells you what the crawler got.

How to check if your site is blocking AI crawlers

Send the request yourself, as the crawler, and read the status code.

The part people skip is the control. If you fire ten requests at a site from one IP and get ten 403s, you've learned nothing. It could be your IP, a rate limiter, or the site being down. You need an unbranded request from the same address to prove the site is up and that the refusal is attributable to the user-agent string.

#!/usr/bin/env bash
SITE="${1:?usage: ./crawlcheck.sh https://example.com}"

code() { curl -s -o /dev/null -w "%{http_code}" -A "$1" --max-time 15 "$SITE"; }

CONTROL=$(code "crawlcheck-control/1.0")
echo "control          $CONTROL"
[ "$CONTROL" = "200" ] || { echo "control failed - results below are meaningless"; exit 1; }

for UA in \
  "GPTBot/1.2" \
  "OAI-SearchBot/1.0" \
  "ChatGPT-User/1.0" \
  "PerplexityBot/1.0" \
  "Claude-SearchBot/1.0" \
  "ClaudeBot/1.0" \
  "Bytespider" \
  "Amazonbot/0.1"
do
  printf '%-18s %s\n' "$(echo "$UA" | cut -d/ -f1)" "$(code "$UA")"
  sleep 1
done

200 means reachable. 403 means you found one.

That covers eight crawlers. There are 38 tokens worth checking, including consent switches like Google-Extended and Applebot-Extended that are not crawlers at all and can only be evaluated against robots.txt. Keeping that list current as vendors add and rename tokens is the tedious part. LovedByAI automates the check across all 38 with the unbranded control request built in, and names each crawler that gets refused.

That sleep 1 matters more than it looks. Which brings me to the thing that nearly broke our own numbers.

Why rate limiting breaks an AI crawler check

Our check fires around 30 requests at a site in quick succession. That is enough to trip a rate limiter on its own.

A 429 therefore tells you about request volume, not about how the site treats AI crawlers. It's a measurement artefact of the tool doing the measuring.

We excluded every rate-limit response from every figure in the study. Including them would have added 107 refusals across 39 sites and made the headline considerably bigger.

If you're building anything like this, space your requests out and throw away the 429s. Otherwise you're measuring your own burst.

What an AI crawler block looks like: 403, 406 and challenges

Across 558 refusals where robots.txt explicitly allowed the crawler:

509 were a plain 403 Forbidden, 93% of the total
24 were a Cloudflare browser challenge, which to a bot is an unsolvable puzzle and therefore a locked door
20 were 406 Not Acceptable, typically a user-agent rule at the web server
5 were a Cloudflare managed-rule 403

That 93% is the useful part. It's the signature of a rule keyed to the user-agent string and nothing else. Not behaviour, not rate, not fingerprinting. Just string matching against a list somebody pasted in years ago.

Which is exactly why this catches crawlers nobody meant to catch. A "block bad bots" toggle switched on in 2021 has no idea ChatGPT-User would exist in 2026.

Which AI crawlers actually crawl your site?

Once the control returns 200 and nothing returns 403, you know crawlers can reach you. The next question is which ones actually do, and that needs request logs rather than a probe.

The two answer different things. The probe tells you who can get in. The logs tell you who bothered, and in what proportion. Raw crawler totals are usually dominated by training crawlers like Meta-ExternalAgent, which will bury the assistant crawlers you care about if you read the totals without splitting them.

Training crawlers vs assistant crawlers: GPTBot vs OAI-SearchBot

Not all of these are the same thing, and the naming does nothing to help:

Training crawlers. GPTBot, CCBot, Bytespider, Amazonbot, Google-Extended. They take content to train models. Blocking them is a legitimate, common position and costs you nothing in AI search.
Assistant crawlers. OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot, DuckAssistBot. They fetch a page because someone just asked a question. The response ends in a citation with your link in it.

GPTBot and OAI-SearchBot are both OpenAI. They do completely different jobs. A UA rule matching GPT catches one; a rule matching OpenAI or a blanket bot filter catches both.

In our sample, 102 of the 135 affected sites block only training crawlers, deliberately and consistently. 33 sites block at least one assistant crawler, and 31 of them never wrote it down anywhere. That's not a policy. That's a side effect.

Does blocking AI crawlers cost you AI traffic?

The thing you'd most want to know is whether blocked sites get less AI traffic. We can't show it.

Median AI referral visits, June to August: 12 for the 269 reachable sites, 6 for the 11 blocking OpenAI at the edge, 5 for the 17 blocking at the origin. Points the right way. Then the 17 sites disallowing OpenAI in robots.txt came in at 19, above the reachable group, which is the opposite.

Groups of 11 to 17 sites, no control for site size or niche. That's noise, and I'd rather say so than dress it up.

What this data does establish: a large minority of sites refuse AI crawlers, most of them can't see it from their own configuration, and the most common kind of block is the kind your logs can't distinguish from success.

Run the script against your own domain. It takes ten seconds. If the control returns 200 and anything else returns 403, you've found something you didn't know was there.

Method, full numbers and the ones we threw out: the 314-site study.

If you'd rather not maintain a token list by hand, LovedByAI is a generative engine optimization platform for small businesses built around WordPress. It automates the optimization work, checks all 38 crawler tokens with the control request built in, and reports AI mentions and AI referral traffic back. The crawler check is free to run.

First-party crawlability checks and bot logs. 314 websites, 38 crawler tokens, 27 to 30 August 2026, validated against logs from 1 to 26 August. No site identified.

WordPress SEO Plugins Weren’t Built for AI Crawlers. Here’s What Changes in 2026

LovedByAI — Mon, 14 Sep 2026 08:19:55 +0000

For years, the WordPress SEO stack has been pretty straightforward.

Install Yoast, Rank Math, AIOSEO, or something similar. Set your titles and descriptions. Generate a sitemap. Add schema. Fix your redirects. Make Google happy.

That still matters.

But ChatGPT, Gemini, Perplexity and other AI answer engines introduce a slightly different problem:

What does an AI crawler actually receive when it visits your page?

We recently compared five WordPress plugins through that lens:

Rank Math
Yoast
AIOSEO
LovedByAI
Squirrly

Full comparison of the best GEO plugins for WordPress

Disclosure before getting into it: I'm part of the team behind LovedByAI, so don't treat this as an independent ranking. What I think is more interesting than the ranking itself is what we found when comparing how these tools approach AI discovery.

SEO and GEO are solving different problems

Traditional SEO plugins are primarily designed around search engines indexing a page and returning a link.

You optimize things like:

Meta titles
Canonicals
XML sitemaps
Structured data
Redirects
Internal linking
Keyword usage

GEO, or Generative Engine Optimization, is trying to influence a different outcome.

Instead of:

"Can Google rank this URL?"

The question becomes:

"Can an LLM understand this page well enough to use the business as part of an answer?"

That means page structure, entity clarity and machine readability become much more important.

The architectural difference I found most interesting

Most WordPress SEO plugins are what I'd call passive optimization.

They improve the existing page.

Rank Math might inject schema. Yoast adds metadata. AIOSEO adds LocalBusiness information.

But when an AI crawler visits the site, it still receives the underlying WordPress page.

Depending on the site, that can mean a pretty messy combination of:

Theme HTML
Page-builder markup
Navigation
Popups
JavaScript
Tracking scripts
Widgets
Cookie banners
Actual content
More JavaScript

For a human browser, that's fine. The browser knows how to render it.

For a machine trying to quickly determine:

What does this company do?
Who is it for?
Where does it operate?
What services does it provide?
Why is it different?

there's potentially a lot of noise.

The alternative is an active rendering layer.

Instead of only modifying the original WordPress HTML, you create a machine-oriented representation of the same page.

Something closer to:

<h1>Acme Accounting</h1>

<p>
Acme Accounting provides bookkeeping and tax services
for small businesses in Austin, Texas.
</p>

<h2>What services does Acme Accounting offer?</h2>

<ul>
  <li>Bookkeeping</li>
  <li>Business tax preparation</li>
  <li>Payroll management</li>
</ul>

alongside structured entity data.

The information hasn't fundamentally changed.

The representation has.

That is the approach we took with LovedByAI's LLM view.

So do you replace Yoast or Rank Math?

I don't think so.

This was probably the biggest conclusion from the comparison.

A GEO plugin and an SEO plugin shouldn't necessarily compete for the same slot.

Rank Math is very good at things like:

redirects
sitemap management
schema
on-page SEO
404 monitoring

Yoast remains extremely strong for:

editorial workflows
readability
metadata
mature SEO infrastructure

AIOSEO has particularly useful features for local businesses.

Those problems don't disappear because ChatGPT exists.

The stack increasingly looks more like:

WordPress
    ↓
Traditional SEO layer
    ↓
Yoast / Rank Math / AIOSEO
    ↓
AI visibility layer
    ↓
Machine-readable content + entity data

In other words, SEO hygiene + GEO infrastructure, rather than SEO vs. GEO.

llms.txt has gained popularity, but it's not the right answer

Another thing worth mentioning.

There has been a lot of attention around llms.txt.

It's an emerging convention, but generating an llms.txt file doesn't suddenly make a site optimized for AI.

The actual page still matters.

An AI system needs to extract clear facts, relationships and answers from your content.

Things like:

clean semantic structure
descriptive headings
direct answers
Organization and Service schema
consistent entity information
crawlable content

are much harder to fake with one text file.

How I'd evaluate a WordPress GEO plugin

After working on this space, I think the useful questions are less about how many "AI features" appear on the pricing page.

I'd look at:

1. What does an AI crawler actually receive?

Inspect the output, not the marketing copy.

2. Does it improve machine readability or just generate content?

An AI writing assistant and an AI-search optimization layer aren't the same thing.

3. Does it produce useful structured data?

Not just generic Article schema.

4. Does it work alongside the existing SEO stack?

Replacing a mature SEO setup just to add GEO rarely makes sense.

5. How much work does it require after installation?

For SMBs especially, a dashboard containing 47 recommendations isn't necessarily a solution.

The bigger shift

The web has historically been optimized primarily for two consumers:

Humans
Googlebot

We're adding a third:

Humans
Search crawlers
AI crawlers / answer engines

And they don't necessarily want information presented in exactly the same way.

I suspect the WordPress stack will increasingly reflect that.

SEO plugins will continue handling technical search hygiene. A new layer of tools will focus specifically on making sites easy for AI systems to parse, understand and cite.

I also created a free GEO auditor that you can run and see the type of suggestions that we are talking about.