DEV Community: AURA GEO

Structured Data That Matches the Page: Check Your JSON-LD Against Visible Content with Python

AURA GEO — Sun, 25 Oct 2026 17:14:03 +0000

Structured data is a promise about the page it sits on. A JSON-LD block that says your app costs 12 USD, that support is at a certain address and that you're the same company as a given GitHub account is a set of claims. Search engines and AI assistants can only use those claims safely if the page backs them up.

Google is explicit about this. Its general structured data guidelines say: "Don't mark up content that is not visible to readers of the page. For example, if the JSON-LD markup describes a performer, the HTML body must describe that same performer." The same page adds that "your structured data must be a true representation of the page content."

Most validators don't check that part. They check syntax and required properties. This post shows how to write Organization and SoftwareApplication JSON-LD from facts that are already on the page, and how to catch drift between the two with about 90 lines of standard-library Python.

Start from the visible facts, not the schema

The usual order goes wrong in a predictable way. Someone opens schema.org, fills in every property that looks relevant, and ends up with markup that says more than the page does. An email address that only exists in the JSON-LD. A price from last quarter. A sameAs link to a profile nobody maintains.

Go the other way. List the facts the page actually shows a visitor, then mark up only those:

the organization or product name, exactly as written in the heading or logo text
the canonical URL of the site
prices with currency, as shown in the pricing table
a support email or phone number, if the page shows one
profile links that appear on the page (often in the footer)

If a fact isn't on the page and you want it in the markup, put it on the page first.

Organization: who you are

Google's Organization documentation says there are no required properties and recommends adding the ones that apply. It suggests placing the markup on the home page "or a single page that describes your organization, for example the about us page," and notes you "don't need to include it on every page of your site." For name, it says to "use the same name and alternateName that you're using for your site name."

sameAs is described as "the URL of a page on another website with additional information about your organization", for example a social or review profile. That's the property assistants and search engines can use to connect your site to your profiles elsewhere, so it should only list profiles that are really yours and still active.

A minimal block for a small SaaS company:

{
  "@type": "Organization",
  "@id": "https://example.com/#org",
  "name": "Example Labs",
  "url": "https://example.com/",
  "email": "support@example.com",
  "sameAs": [
    "https://github.com/example-labs",
    "https://x.com/examplelabs"
  ]
}

SoftwareApplication: what you sell

For apps, Google's Software App documentation lists three required properties for the rich result: name, offers.price, and a rating or review (aggregateRating or review). If the app is free, it says to set offers.price to 0. If the price is above zero, it recommends adding offers.priceCurrency. applicationCategory (for example BusinessApplication) and operatingSystem are recommended.

Be careful with the rating requirement. Don't invent an aggregateRating to qualify for stars. The general guidelines specifically warn against marking up "fake reviews," and the review snippet guidelines say that when an entity controls the reviews about itself, its pages using Organization-type markup are ineligible for star reviews. If you have no genuine, visible ratings, leave the property out. The markup is still valid schema.org and still describes your product. It just won't be eligible for that particular rich result.

{
  "@type": "SoftwareApplication",
  "name": "Example Notes",
  "url": "https://example.com/",
  "applicationCategory": "BusinessApplication",
  "operatingSystem": "Web",
  "publisher": { "@id": "https://example.com/#org" },
  "offers": { "@type": "Offer", "price": 12.0, "priceCurrency": "USD" }
}

The publisher reference uses the @id from the Organization node. Putting both nodes in one @graph keeps a single source of truth for the company and avoids repeating it per page.

If you sell a physical product, Google's Product and merchant listing docs apply instead, with their own required fields. The checking approach below works the same way.

The full block, in the page head

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@graph": [
    { "@type": "Organization", "@id": "https://example.com/#org",
      "name": "Example Labs", "url": "https://example.com/",
      "email": "support@example.com",
      "sameAs": ["https://github.com/example-labs", "https://x.com/examplelabs"] },
    { "@type": "SoftwareApplication", "name": "Example Notes",
      "url": "https://example.com/", "applicationCategory": "BusinessApplication",
      "operatingSystem": "Web", "publisher": { "@id": "https://example.com/#org" },
      "offers": { "@type": "Offer", "price": 12.0, "priceCurrency": "USD" } }
  ]
}
</script>

And the visible part of the page, which is where things drift:

<h1>Example Notes</h1>
<p>Shared notes for small teams, made by Example Labs.</p>
<p>Team plan: 12 USD per user per month.</p>
<footer>
  <a href="https://github.com/example-labs">GitHub</a> Â·
  <a href="/">Home</a>
</footer>

Two claims in the markup have no visible counterpart: the support email and the X profile. Neither is a syntax error, so a validator won't flag them.

A small checker: does the markup match the page?

The script below fetches a page (or reads a saved HTML file), pulls out every application/ld+json block, and walks the JSON for a set of fact-like keys: names, URLs, sameAs, email, phone, price, currency and address parts. Then it checks each value against the page:

text facts must appear in the visible text, meaning anything outside <script>, <style>, <noscript>, <template>, <svg> and elements marked hidden or aria-hidden="true"
url and sameAs values must appear as a link (<a href>) on the page
missing sameAs profiles produce a WARN rather than a failure, since profile links are something to review, not always an error

It only uses the Python standard library.

#!/usr/bin/env python3
"""Check that JSON-LD facts on a page also appear in its visible HTML."""
import json, re, sys, urllib.request
from html.parser import HTMLParser
from urllib.parse import urljoin, urlsplit

CHECK_KEYS = {"name", "legalName", "alternateName", "url", "sameAs", "email",
              "telephone", "price", "priceCurrency", "streetAddress",
              "addressLocality", "postalCode"}
SKIP_TAGS = {"script", "style", "noscript", "template", "svg"}

class Page(HTMLParser):
    VOID = {"area", "base", "br", "col", "embed", "hr", "img", "input",
            "link", "meta", "source", "track", "wbr"}
    def __init__(self, base):
        super().__init__(convert_charrefs=True)
        self.base, self.stack, self.in_ld = base, [], False
        self.text, self.links, self.ld = [], set(), []
    def handle_starttag(self, tag, attrs):
        a = dict(attrs)
        if tag == "a" and a.get("href"):
            self.links.add(norm(urljoin(self.base, a["href"])))
        if tag == "script" and a.get("type", "").lower() == "application/ld+json":
            self.in_ld = True
            self.ld.append("")
        if tag in self.VOID:
            return
        hidden = tag in SKIP_TAGS or "hidden" in a or a.get("aria-hidden") == "true"
        self.stack.append((tag, hidden))
    def handle_startendtag(self, tag, attrs):
        if tag == "a":
            self.handle_starttag(tag, attrs)   # records the link
            self.handle_endtag(tag)
    def handle_endtag(self, tag):
        if tag == "script":
            self.in_ld = False
        for i in range(len(self.stack) - 1, -1, -1):   # pop to the matching tag
            if self.stack[i][0] == tag:
                del self.stack[i:]
                break
    def handle_data(self, data):
        if self.in_ld:
            self.ld[-1] += data
        elif not any(h for _, h in self.stack):
            self.text.append(data)

def norm(u):
    p = urlsplit(u.strip())
    return f"{p.netloc.lower().removeprefix('www.')}{p.path.rstrip('/')}"

def walk(node, path=""):
    """Yield (path, key, value) for every scalar we want to check."""
    if isinstance(node, dict):
        for k, v in node.items():
            yield from walk(v, f"{path}.{k}" if path else k)
    elif isinstance(node, list):
        for i, v in enumerate(node):
            yield from walk(v, f"{path}[{i}]")
    elif isinstance(node, (str, int, float)):
        if isinstance(node, float) and node.is_integer():
            node = int(node)                  # 19.0 -> "19"
        key = re.sub(r"\[\d+\]$", "", path).split(".")[-1]
        if key in CHECK_KEYS and not re.search(r"(logo|image)", path):
            yield path, key, str(node)

def main(url, saved=None):
    if saved:                                 # test a local build before deploy
        html = open(saved, encoding="utf-8").read()
    else:
        req = urllib.request.Request(url, headers={"User-Agent": "ld-check/0.1"})
        html = urllib.request.urlopen(req, timeout=20).read().decode("utf-8", "replace")
    page = Page(url)
    page.feed(html)
    text = re.sub(r"\s+", " ", " ".join(page.text)).lower()
    problems = 0
    for i, raw in enumerate(page.ld):
        try:
            data = json.loads(raw)
        except json.JSONDecodeError as e:
            print(f"block {i}: INVALID JSON ({e})"); problems += 1; continue
        for path, key, value in walk(data):
            if key in {"url", "sameAs"}:
                ok = norm(value) in page.links or norm(value) == norm(url)
                where = "link"
            else:
                ok = value.lower() in text
                where = "text"
            soft = key == "sameAs"          # profile links: review, don't fail
            status = "OK  " if ok else ("WARN" if soft else "MISS")
            problems += status == "MISS"
            print(f"{status} block {i} {path} = {value!r} ({where})")
    print(f"\n{len(page.ld)} JSON-LD block(s), {problems} problem(s)")
    return 1 if problems else 0

if __name__ == "__main__":
    sys.exit(main(*sys.argv[1:3]))

Run it against a URL, or against a local build with the URL as the base for relative links:

python3 ldcheck.py https://example.com/
python3 ldcheck.py https://example.com/ dist/index.html

For the example page above, the output is:

OK   block 0 @graph[0].name = 'Example Labs' (text)
OK   block 0 @graph[0].url = 'https://example.com/' (link)
MISS block 0 @graph[0].email = 'support@example.com' (text)
OK   block 0 @graph[0].sameAs[0] = 'https://github.com/example-labs' (link)
WARN block 0 @graph[0].sameAs[1] = 'https://x.com/examplelabs' (link)
OK   block 0 @graph[1].name = 'Example Notes' (text)
OK   block 0 @graph[1].url = 'https://example.com/' (link)
OK   block 0 @graph[1].offers.price = '12' (text)
OK   block 0 @graph[1].offers.priceCurrency = 'USD' (text)

1 JSON-LD block(s), 1 problem(s)

The fix is on the page side, not the markup side. Show the support email in the footer, add the X link if the account is real and maintained, or remove both from the JSON-LD. The script exits with status 1 when anything is MISS, so you can run it in CI against a preview build and fail the job when someone changes a price in the template but not in the markup.

What the script doesn't do

It's a drift detector, not a validator. Know its limits:

Substring matching. A price of 12 also matches 120. For pricing pages, tighten the check to look for the formatted price string you actually render.
Raw HTML only. It reads the HTML the server returns. If your JSON-LD or your visible text is injected by JavaScript, the script won't see it. Google can process JavaScript-generated structured data and documents how, but not every crawler runs JavaScript, so server-rendering both the facts and the markup is the safer default.
CSS hiding. It skips hidden and aria-hidden elements, but not elements hidden with CSS classes. If you hide content in collapsed panels or display: none rules, check those by hand.
Image URLs. logo and image paths are skipped on purpose, since a logo URL isn't something a visitor reads as text.
Formatting. Phone numbers in E.164 (+1-555-...) won't match a display format like (555) .... Normalize them if you check phones.

Validate the syntax and eligibility too

The checker covers the "matches the page" rule. Two official tools cover the rest:

Schema.org Validator parses JSON-LD, Microdata and RDFa and shows the extracted items against the schema.org vocabulary. Use it to catch typos in types and property names.
Google Rich Results Test checks whether markup is eligible for Google's rich result types and reports missing required or recommended properties. Google recommends the URL input over the code input, because the code input has JavaScript limitations.

After deploying, Google's docs suggest using the URL Inspection tool in Search Console to see how Google renders the page, then requesting a recrawl. Even then, Google says it "does not guarantee that features that consume structured data will show up in search results."

Why this matters beyond rich results

AI assistants with web search read and quote pages. When the markup and the visible text agree, every reader gets the same facts: a crawler parsing JSON-LD, a model reading page text, and a person checking the source. When they disagree, you've published two versions of your company and can't control which one gets picked up.

Keep the check cheap and automatic. Run it on the home page, the pricing page and the about page whenever any of them change.

If you want this kind of check as part of a bigger workflow, AURA GEO can generate JSON-LD from the facts it fetched from your site, and its URL verification stores the raw HTML and JSON-LD a page actually served, so you can compare them. The MCP guide shows how to connect your own agent to run those checks and pick up the resulting tasks.

Where the AI Money Actually Goes: What Builders Can Learn From 2025–2026 Spending Data

AURA GEO — Sun, 25 Oct 2026 13:45:48 +0000

Every week a new AI wrapper launches, and every week someone on a forum asks: "Will anyone actually pay for this?" Instead of guessing, I went through the public spending reports from Menlo Ventures, Sensor Tower and Appfigures and pulled out the numbers that matter if you build software. This post is a builder's reading of that data: what people pay for, what problem they are buying a fix for, and what that means for the thing you are shipping.

TL;DR

Consumer AI spend went from an estimated $12B (2025) to ~$40B (2026), while US adoption only moved from 61% to 64% (Menlo, 2026). Growth came from existing users paying more, not new users.
Enterprise gen-AI spend hit $37B in 2025, and coding alone was $4.0B, the largest application category (Menlo, Enterprise 2025).
27% of enterprise AI app spend arrives through product-led growth, vs 7% in traditional software. Individual developers swiping a card is a real go-to-market.
Users now rank accuracy (45%) and trustworthiness (40%) above ease of use (32%). That changes what "good UX" means for an AI product.

1. People pay to save time on work they already do

The 2026 Menlo consumer survey (5,067 US adults) lists six activities where more than half of the people doing them now use AI:

Activity	Share using AI
Writing support	61%
Coding projects	58%
School / work assignments	56%
Image creation	53%
Presentations	52%
Game creation	52%

Look at the list: none of it is a new behaviour. People already wrote, coded, and made slides. AI money flows to existing, frequent jobs where the output is easy to check. If your product idea needs users to adopt a brand-new habit and trust AI, you are fighting two battles.

2. Coding is the first "killer use case", and devs are price-insensitive

On the enterprise side, Menlo estimates departmental AI at $7.3B in 2025, with coding at $4.0B, 55% of it. Next come IT ops ($700M), marketing ($660M) and customer success ($630M).

The report also notes that "for frontier use cases like coding, users are actually quite price-insensitive and will pay more for performance." That matches what many of us feel personally: if a tool reliably saves an hour a day, $20 vs $40 a month barely registers.

Implication for builders: developer tools can price on value and still convert, if the quality gap is obvious within the first session.

3. A small group of power users funds the whole market

This is the most useful number in the 2026 consumer report for anyone designing pricing:

55% of AI users pay for at least one AI product.
The typical payer spends $20–49/month.
But the 14% who spend $100+/month generate 60% of consumer AI spending.
Payers are 5x more likely to use agents (65% vs 13%).

Pricing models reflect this. Half of payers are on flat monthly plans, 26% annual, 15% mix a subscription with usage charges, and 17% are on usage-based pricing.

Implication: a single $9 tier leaves most of the money on the table. A common pattern that fits the data: free tier for discovery → predictable subscription → metered overage for heavy users. Menlo's own advice to founders is blunt: "Keep the free tier." It's where habits form before people pay.

4. Thin wrappers make money, but the bulldozer is coming

Appfigures found that third-party "General Assistant" apps (thin wrappers on top of model APIs) earned $726M in gross revenue since 2023 among the top 1,000 AI apps, more than ChatGPT's own app category in their dataset at the time (Appfigures).

But Menlo's 2025 consumer report warns that each new model release "absorbs more intelligence and specialized features, making entire categories obsolete overnight," which they compare to "building in front of a moving bulldozer."

The 2026 data adds nuance: specialized-tool usage rose from 69% to 75% of AI users, and 72% now use both general and specialized products. Users don't replace ChatGPT, they add tools that do one job materially better. Menlo calls this the difference between being in the "rotation" (competing for queries) and being in the "stack" (owning a workflow).

Implication: a wrapper can earn, but durable revenue comes from owning a workflow: data the general assistant doesn't have, integrations, state, and memory.

5. Trust is now a feature you have to ship

In 2025, ease of use ranked ahead of trust. In 2026 the order flipped: accuracy 45%, trustworthiness 40%, security and privacy 36%, ease of use 32%. Among people who only use general assistants, 27% cite privacy concerns with niche tools and 21% say they don't know which tool to trust. Only 11% say they're unaware of a specialized product.

So the problem isn't discovery. It's credibility. Practical things builders can do:

Show sources and confidence for generated output.
Ask for the minimum permissions your core job needs, then expand.
Make output easy to verify (diffs, previews, citations).
Publish clear data-handling docs.

6. Distribution is shifting into AI answers themselves

One more data point that affects how dev tools get discovered. In the 2026 survey, 50% of shoppers who use AI used it to compare prices, 43% to narrow a shortlist, and 37% made a final purchase decision based on its recommendation. Menlo also reports that ChatGPT launched ads in February 2026 and, according to OpenAI, passed $1B in run-rate ad revenue about 200 days later.

For builders, "ask ChatGPT which tool to use" is now a real acquisition channel, next to search and word of mouth. It's worth occasionally asking the main assistants about your own category and seeing whether your product shows up, and what they say about it. We build AURA GEO around exactly this kind of check, but you can start manually: ask five assistants the question your user would ask and write down the answers.

A builder's checklist

Pick a job people already do weekly and where output is checkable.
Be 10x better at one thing than the general assistant your user already has open.
Price for the power-user tail: subscription + metered overage, plus a free tier.
Ship trust: citations, permissions hygiene, verifiable output.
Plan for PLG: individuals adopt first, teams contract later. Menlo notes Cursor reached $200M in revenue before hiring an enterprise sales rep.
Check how AI assistants describe your product, because a growing share of buying decisions start there.

Closing thought

The data says people aren't paying for "AI". They're paying to get back time on work they already do, and they keep paying when they trust the result. That's a much more boring, and much more buildable, target than most launch posts suggest.

Sources

Menlo Ventures, 2026: The State of Consumer AI — https://menlovc.com/perspective/2026-the-state-of-consumer-ai/
Menlo Ventures, 2025: The State of Consumer AI — https://menlovc.com/perspective/2025-the-state-of-consumer-ai/
Menlo Ventures, 2025: The State of Generative AI in the Enterprise — https://menlovc.com/perspective/2025-the-state-of-generative-ai-in-the-enterprise/
Appfigures, AI Apps are a $2B Opportunity and Thin Wrappers Make More Than You Think — https://appfigures.com/resources/insights/20250131?f=1

Allow AI Search, Opt Out of Training: A Granular robots.txt You Can Test in Python

AURA GEO — Sat, 24 Oct 2026 16:15:11 +0000

Most robots.txt advice about AI comes in two flavors: "block all AI bots" or "allow everything." Both skip the useful middle option. You can let AI search and user-requested fetches reach your pages while telling the training crawlers to stay out.

That only works if you know which token does which job, and if your file actually matches the way crawlers read groups. This post covers both, then tests the result with about 30 lines of Python and a couple of curl calls.

Everything below comes from the vendors' own documentation as of October 2026. Tokens and policies change, so check the linked pages before you deploy.

Three jobs, three kinds of bots

Each large AI operator separates crawling by purpose. Blocking one token doesn't block the others.

Search and indexing (they decide whether you can be shown and linked in answers):

OAI-SearchBot: OpenAI says it is used "to surface websites in search results in ChatGPT's search features," and sites opted out of it "will not be shown in ChatGPT search answers" (OpenAI crawlers).
Claude-SearchBot: per Anthropic, blocking it "may reduce your site's visibility and accuracy in user search results."
PerplexityBot: "designed to surface and link websites in search results on Perplexity," and not used to crawl content for AI foundation models (Perplexity crawlers).
Bingbot: Bing's standard crawler (Bing crawlers).
Googlebot: Google says robots.txt rules for Googlebot are the control for Search, including AI Overviews and AI Mode (AI features and your website).
Applebot: powers search in Spotlight, Siri and Safari (About Applebot).

User-triggered fetchers (they load a page because a person asked a question just now):

ChatGPT-User: OpenAI says that "because these actions are initiated by a user, robots.txt rules may not apply."
Perplexity-User: Perplexity says that "since a user requested the fetch, this fetcher generally ignores robots.txt rules."
Claude-User: Anthropic describes it as a way for site owners to control user-initiated access, and says its bots honor robots.txt.

Training and dataset controls:

GPTBot: disallowing it "indicates a site's content should not be used in training" OpenAI's foundation models.
ClaudeBot: restricting it signals that future content "should be excluded from our AI model training datasets."
Google-Extended: not a crawler. It's a control token with no user agent of its own. It governs use of Google-crawled content for training future Gemini models and for grounding in Gemini Apps and Vertex AI, and Google says it "does not impact a site's inclusion in Google Search" (Google's common crawlers).
Applebot-Extended: also doesn't crawl. Disallowing it opts you out of training Apple's foundation models, and those pages "can still be included in search results."
CCBot: the crawler of Common Crawl, a non-profit that publishes an open crawl archive anyone can use. Blocking it keeps future pages out of that archive.

Two operators don't fit this pattern. Microsoft has no separate training token in robots.txt. Its controls for Copilot and training are the noarchive and nocache robots meta values (Bing robots meta tags). Apple has a fallback rule: "If robots instructions don't mention Applebot but mention Googlebot, the Apple robot will follow Googlebot instructions." A Googlebot-only group therefore quietly governs Applebot too.

How crawlers pick a group (RFC 9309)

RFC 9309 is the robots.txt standard. Four rules from it cause most of the surprises:

A crawler obeys the group that names its product token. Matching is case-insensitive. The * group applies only "if no matching group exists."
A named group doesn't inherit anything from *. If OAI-SearchBot has its own group, every line under User-agent: * is irrelevant to it.
Groups naming the same token are merged. Two User-agent: GPTBot blocks combine into one rule set (the RFC has an example). Many homemade parsers get this wrong, so don't depend on it. Keep one group per token.
Within a group, the most specific path wins, meaning the longest match. If an Allow and a Disallow are equivalent, Allow should win.

Here's the bug rule 2 produces, and it's common:

User-agent: *
Disallow: /admin/

User-agent: OAI-SearchBot
Allow: /

The author wanted "everyone stays out of /admin/, and OpenAI search is explicitly welcome." What they got: OAI-SearchBot obeys only its own group, so /admin/ is allowed for it. Every named group has to repeat the private paths.

A granular file

Here is a file that allows search and user fetches, keeps private paths private for everyone, and opts out of training:

# 1) Training and dataset opt-outs
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /

# 2) Search and user-triggered fetchers: allowed, same private paths
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Bingbot
User-agent: Googlebot
User-agent: Applebot
Disallow: /admin/
Disallow: /cart/

# 3) Everyone else
User-agent: *
Disallow: /admin/
Disallow: /cart/

Sitemap: https://example.com/sitemap.xml

A few notes:

Several User-agent lines in a row form one group. That's valid under the RFC and saves repetition.
The training group comes first on purpose. That's not because of the RFC, where order doesn't matter, but because of a Python quirk covered below.
Opting out of training is a business decision, not a GEO requirement. If you're happy to be used for training, delete group 1 and the * group covers those bots.
Remember what robots.txt can't do. ChatGPT-User and Perplexity-User say they may not follow it. To actually stop a client, you need server or WAF rules.

Test it with Python

Python's standard library includes urllib.robotparser. This script prints an allow/block matrix for every token against a few paths. It works on a local file or a live URL:

import sys
import urllib.robotparser

# python3 check_robots.py robots.txt
# python3 check_robots.py https://example.com/robots.txt
src = sys.argv[1]
rp = urllib.robotparser.RobotFileParser()
if src.startswith("http"):
    rp.set_url(src)
    rp.read()
else:
    with open(src, encoding="utf-8") as f:
        rp.parse(f.read().splitlines())

AGENTS = [
    "OAI-SearchBot", "ChatGPT-User", "GPTBot",
    "Claude-SearchBot", "Claude-User", "ClaudeBot",
    "PerplexityBot", "Perplexity-User",
    "Bingbot", "Googlebot", "Google-Extended",
    "Applebot", "Applebot-Extended", "CCBot",
    "SomeOtherBot",
]
PATHS = ["/", "/pricing", "/blog/post", "/admin/", "/cart/checkout"]

width = max(len(a) for a in AGENTS)
print(" " * width, *[p.ljust(14) for p in PATHS])
for agent in AGENTS:
    row = ["allow" if rp.can_fetch(agent, p) else "BLOCK" for p in PATHS]
    print(agent.ljust(width), *[r.ljust(14) for r in row])

Running it against the file above (Python 3.13) prints:

                  /              /pricing       /blog/post     /admin/        /cart/checkout
OAI-SearchBot     allow          allow          allow          BLOCK          BLOCK
ChatGPT-User      allow          allow          allow          BLOCK          BLOCK
GPTBot            BLOCK          BLOCK          BLOCK          BLOCK          BLOCK
Claude-SearchBot  allow          allow          allow          BLOCK          BLOCK
Claude-User       allow          allow          allow          BLOCK          BLOCK
ClaudeBot         BLOCK          BLOCK          BLOCK          BLOCK          BLOCK
PerplexityBot     allow          allow          allow          BLOCK          BLOCK
Perplexity-User   allow          allow          allow          BLOCK          BLOCK
Bingbot           allow          allow          allow          BLOCK          BLOCK
Googlebot         allow          allow          allow          BLOCK          BLOCK
Google-Extended   BLOCK          BLOCK          BLOCK          BLOCK          BLOCK
Applebot          allow          allow          allow          BLOCK          BLOCK
Applebot-Extended BLOCK          BLOCK          BLOCK          BLOCK          BLOCK
CCBot             BLOCK          BLOCK          BLOCK          BLOCK          BLOCK
SomeOtherBot      allow          allow          allow          BLOCK          BLOCK

Run it against the buggy two-group example and you'll see OAI-SearchBot allowed into /admin/. That's the inheritance bug made visible.

Know the parser's limits

urllib.robotparser is handy, but it's not an RFC 9309 implementation. Reading its source shows three differences:

It uses the first matching group and matches substrings. It checks whether a group's token appears inside the name you pass in. When I first put the search group (which contains Applebot) above the training group, Applebot-Extended matched Applebot and came out as "allow." Putting the training group first fixed the test. Real crawlers match product tokens, but the lesson stands: test exactly the tokens you care about.
It uses the first matching rule, not the longest one. Overlapping Allow/Disallow pairs can give a different answer than the RFC.
It doesn't support * and $ path wildcards. Google documents those and many sites use them.

So use it as a fast smoke test for files with one group per token and no overlapping or wildcard rules. For anything more complex, also test with a parser that implements the standard, such as Google's open-source robots.txt library.

Test the live file with curl

The parser tells you what the file means. curl tells you what crawlers actually receive:

# 1. Status, content type, redirects
curl -sSI https://example.com/robots.txt

# 2. The exact body, as a crawler gets it
curl -s -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" \
  https://example.com/robots.txt

# 3. Compare with a browser user agent
diff <(curl -s https://example.com/robots.txt) \
     <(curl -s -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)" https://example.com/robots.txt)

What to look for, according to RFC 9309:

A 5xx response is the worst case. The file is "unreachable," and crawlers must assume complete disallow. A flaky origin or an edge rule that errors on bots can silently block everything.
A 4xx response means "unavailable," and crawlers may access anything. Your carefully written opt-outs simply don't exist.
Redirects: crawlers should follow at least five. A redirect to an HTML page (a login, a 404 template) gives them something that isn't a robots file.
Size: parsers must handle at least 500 KiB. Keep the file far below that.
Different body per user agent: a CDN or bot-protection layer may be rewriting or challenging the request.

Finally, allow for delay. The RFC says crawlers shouldn't use a cached copy for more than 24 hours. OpenAI says search changes can take "~24 hours." Perplexity says "up to 24 hours." Check again the next day.

Checklist

One group per token. No duplicated User-agent: X blocks.
Private paths repeated in every named group.
A deliberate decision on each training token. Don't rely on copied defaults.
Googlebot rules reviewed with Applebot in mind (Apple's fallback).
The live file returns 200, text/plain, and the same body for bots and browsers.
The matrix script run against the live URL after every deploy.

If you manage AI visibility with an agent, AURA GEO can generate a robots.txt patch from your site's real state. The tool description is explicit that it must be merged into your existing policy, never blindly swapped in. Run the script above on the merged result before you ship it. For background on the tokens themselves, see AURA's knowledge-base note on robots.txt for AI search.

Moved to Cloudflare? Check It Isn't Quietly Blocking AI Crawlers

AURA GEO — Fri, 23 Oct 2026 16:19:02 +0000

You moved your site onto Cloudflare. DNS resolved, the padlock is green, every page loads in your browser. Job done?

Not quite. Cloudflare now decides, at the edge, what each client gets, and several of its features treat AI crawlers differently from people. If you never opened those settings, you don't actually know whether ChatGPT search, Claude or Perplexity can still reach your pages.

This post walks through the settings that matter (as Cloudflare documents them in October 2026), a curl test you can run in a few minutes, and how to read what comes back.

First, know which bot does what

"AI bots" is not one thing. The big operators run separate user agents for training, for search, and for fetching a page because a person asked a question. Blocking one does not block the others.

Operator	Training	Search / indexing	User-triggered fetch
OpenAI	`GPTBot`	`OAI-SearchBot`	`ChatGPT-User`
Anthropic	`ClaudeBot`	`Claude-SearchBot`	`Claude-User`
Perplexity	(none listed)	`PerplexityBot`	`Perplexity-User`

Sources: OpenAI crawlers, Anthropic's crawler FAQ, Perplexity crawlers.

Two details from those pages matter here. OpenAI says sites opted out of OAI-SearchBot won't be shown in ChatGPT search answers. And both OpenAI and Perplexity say their user-triggered fetchers (ChatGPT-User, Perplexity-User) may not follow robots.txt, because a person asked for the page. So "should I let AI bots in?" is really three questions: training, search, and live fetches.

The Cloudflare settings that can affect AI crawlers

There are five places to look. They overlap, so check all of them.

1. AI bot policies: Search, Agent, Training

Cloudflare now groups AI traffic by behavior (Bots: AI bots):

Search: collects or indexes your content so it can answer questions later.
Agent: acts in real time on a person's behalf, "such as chat fetch bots and browser-use agents."
Training: crawls content to train or fine-tune a model.

You'll find them under Security Settings > Configure AI bot policies. For each one you can choose Block, Block on pages with ads, or Allow (Block AI Bots docs). Since September 15, 2026, Training also has a Disallow AI Training option. According to Cloudflare's announcement, it publishes a no-training preference in robots.txt, keeps crawlers Cloudflare calls "Accountable" mixed-use crawlers (Applebot, Bingbot, Googlebot) allowed for search, and blocks other training crawlers.

Two things in that announcement deserve attention:

Block now reaches search engines too. Block and Block on pages with ads for Training now also apply to mixed-use crawlers, "including Applebot, Bingbot, and Googlebot, so either setting impacts search as well as training."
The old switch is gone. The legacy "Block AI bots" toggle is deprecated, and existing settings were migrated to the new controls. If you had the old toggle on, check what it turned into.

For new domains, Cloudflare proposes presets during onboarding. Per the same post, a site that doesn't monetize with ads gets Search, Training and Agent all set to Allow. A site that monetizes with ads gets Search Allow, Training Disallow AI Training and Agent Block on pages with ads. You can change any of them later. Don't assume. Open the page and read what's actually set.

2. Bot Preference Sync (formerly managed robots.txt)

Bot Preference Sync writes your policy choices into robots.txt. When it's on and at least one policy is set to Block, Block on pages with ads or (for Training) Disallow, Cloudflare prepends managed directives and its Content Signals Policy to your existing file. If you have no robots.txt, Cloudflare creates one. Cloudflare's blog says it's on by default for new customers.

Content Signals are three machine-readable categories: search, ai-input and ai-train. Cloudflare's docs are clear that robots.txt compliance is voluntary. It states a preference, and enforcement happens elsewhere.

So your live robots.txt may no longer be the file you deployed.

3. AI Crawl Control

AI Crawl Control (formerly AI Audit, available on all plans) shows which AI crawlers request your content, and lets you Allow or Block each one (Manage AI crawlers). Blocking a crawler there creates or updates a WAF custom rule. On paid plans you can choose whether blocked crawlers get 403 Forbidden or 402 Payment Required.

One detail matters for testing. On the Free plan, AI Crawl Control identifies AI crawlers "based on their user agent strings." Paid plans can use Bot Management detection instead.

4. Bot Fight Mode

Bot Fight Mode is a free toggle that identifies traffic matching known bot patterns and issues computationally expensive challenges. Cloudflare notes it "may challenge API or mobile app traffic". Also, you can't bypass it with WAF custom rules or Page Rules: Skip, Bypass and Allow have no effect on it. If it gets in your way, the documented options are turning it off or moving to Super Bot Fight Mode, which supports Skip rules.

5. Your own WAF custom rules

Country blocks or a user-agent blocklist copied from a forum can catch crawlers too. The Block action returns 403 for most security features (or 429 for rate limiting). Challenge actions require the client to pass a Cloudflare challenge first. Anthropic, for example, says its bots won't try to bypass CAPTCHAs.

If you write allow rules, Cloudflare exposes cf.client.bot (a request from a known good bot) and cf.verified_bot_category. That's safer than trusting the User-Agent header, which anyone can fake. Perplexity's own docs recommend combining user-agent matching with its published IP ranges.

The curl test

Send the same request with different user agents and compare. The strings below come from the vendors' docs; version numbers may change. Anthropic doesn't publish a full UA string on that page, so the ClaudeBot line is a minimal test string containing the product token.

URL="https://example.com/pricing"

declare -A UA=(
  [browser]="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"
  [GPTBot]="Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot"
  [OAI-SearchBot]="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot"
  [ChatGPT-User]="Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot"
  [PerplexityBot]="Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)"
  [Perplexity-User]="Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)"
  [ClaudeBot]="Mozilla/5.0 (compatible; ClaudeBot/1.0)"
)

for name in "${!UA[@]}"; do
  curl -s -o "out_$name.html" -D "hdr_$name.txt" -A "${UA[$name]}" \
    -w "$name\t%{http_code}\t%{size_download} bytes\n" "$URL"
  grep -i -E "^(cf-mitigated|cf-ray):" "hdr_$name.txt" | sed 's/^/    /'
done

Then check what robots.txt says now:

curl -s https://example.com/robots.txt | sed -n '1,80p'

Look for a BEGIN Cloudflare block, Content-signal: lines, and Disallow: / under any user agent you care about.

Reading the result

What you see	What it means
Same status and similar size as the browser	No user-agent-based block on this URL
`403` with a `cf-ray` header (Cloudflare adds it to responses, see HTTP headers)	Blocked at the edge: AI Crawl Control, an AI bot policy, or a custom rule
`402`	AI Crawl Control block configured as "Payment Required"
`cf-mitigated: challenge` header	A Cloudflare challenge page; per the docs this header is set on every challenge page type
`429`	Rate limiting
`200` but robots.txt disallows the bot	No enforcement, just a stated preference that compliant crawlers follow

An important limit: your curl comes from your IP with a borrowed user agent. Cloudflare verifies real bots by other means: Web Bot Auth signatures, published IP lists, or reverse DNS (Verified bots). So this test catches user-agent-based rules (including AI Crawl Control on the Free plan), but it can't perfectly reproduce how verified bot traffic is treated, in either direction. Confirm in the dashboard: Security > Analytics > Events shows which service acted on a request, and the AI Crawl Control Crawlers tab shows requests and unsuccessful requests per crawler.

Choosing deliberately

A reasonable starting point for a business that wants to be found in AI answers:

Search: Allow. This is how OAI-SearchBot, Claude-SearchBot and PerplexityBot build what they cite.
Agent: think before blocking. Cloudflare's Agent category includes chat fetch bots, the kind that open your page when someone asks about you.
Training: your call. Disallow AI Training keeps Accountable mixed-use crawlers for search. Plain Block now affects Googlebot and Bingbot as well.

Common mistakes

Testing only in a browser. Humans and crawlers can get different answers from the same URL.
Treating robots.txt as a firewall. It's a preference. Cloudflare's own docs say so.
Choosing "Block" for Training to "just stop AI". After September 15, that also stops Googlebot and Bingbot.
Forgetting Bot Fight Mode. Your WAF allow rule won't override it.
Allowing by user agent alone. Use cf.client.bot or IP ranges, or anyone can claim to be GPTBot.

If you'd rather have this checked continuously, AURA GEO audits what your server actually returns and keeps synthetic user-agent tests separate from real crawler evidence. There's a related explainer on checking CDN and WAF rules that block AI bots.

Can AI Crawlers See Your JavaScript? A 10-Minute curl Test for the HTML Bots Actually Get

AURA GEO — Thu, 22 Oct 2026 16:09:01 +0000

Your page looks perfect in Chrome. Headline, pricing, FAQ, a nice comparison table. But an AI crawler doesn't open Chrome. It sends one HTTP request, gets back whatever your server returns, and moves on.

If your important text only appears after JavaScript runs, there's a real chance that crawler saw an empty shell. This post shows how to check that in about ten minutes with curl and a couple of one-liners.

Why this matters for AI search

There are two versions of every page:

The initial HTML - what the server sends in response to a request.
The rendered page - what you see after scripts run, APIs return data and widgets load.

Google documents a separate rendering stage for JavaScript (JavaScript SEO basics). Other clients don't have to copy Google's capabilities. In December 2024, Vercel and MERJ published an analysis of AI crawler traffic on Vercel's network (The rise of the AI crawler). They found that OpenAI's crawlers, ClaudeBot, PerplexityBot and several others fetched JavaScript files but didn't execute them, while Gemini (via Google's infrastructure) and Applebot did render pages. Crawler behavior can change over time, so treat this as a reason to test, not as a permanent rule.

The safe assumption: if a fact isn't in the initial HTML, don't count on an AI crawler seeing it.

Step 1: Pick three pages, not your whole site

Don't start with a full crawl. Choose:

a service or pricing page,
one article or guide,
one product or listing page.

For each one, write down 3-5 facts that must be visible: the H1, the main description, a price or condition, a key claim, and the important internal links.

Step 2: Fetch the raw HTML

URL="https://example.com/pricing"

curl -sL "$URL" -o raw.html
wc -c raw.html

Ignore the file size. A 300 KB response can be all navigation and inline JSON; a 20 KB response can contain the full answer. What matters is meaning.

To see the response a specific crawler would get, you can also send its user agent string (useful when a CDN or WAF treats bots differently):

curl -sL -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.1; +https://openai.com/gptbot)" \
  -o raw_gptbot.html -w "%{http_code}\n" "$URL"

If this returns 403, 429 or a challenge page while a normal request returns 200, you have a different problem: your firewall, not your rendering. Fix that first.

Step 3: Search for your must-have facts

for phrase in "Pricing" "per month" "Free trial" "How it works"; do
  if grep -qi "$phrase" raw.html; then
    echo "FOUND:   $phrase"
  else
    echo "MISSING: $phrase"
  fi
done

A match isn't the whole story. Text sitting inside a JSON blob in a <script> tag (for example __NEXT_DATA__ or a hydration payload) is technically "in the HTML," but a normal content extractor may not treat it as the main content. (The Vercel study notes that models can sometimes interpret JSON in the initial response, but you shouldn't rely on it.) Check where the match lives:

grep -o -i ".\{80\}per month.\{80\}" raw.html | head

If every hit is inside a script, mark it as "present in data, not in markup."

Step 4: Strip the markup and read it like a bot

A quick way to see the visible text without a browser:

python3 - <<'PY'
import re, html
t = open("raw.html", encoding="utf-8", errors="ignore").read()
t = re.sub(r"(?is)<(script|style|noscript|svg)[^>]*>.*?</\1>", " ", t)
t = re.sub(r"(?s)<[^>]+>", " ", t)
t = re.sub(r"\s+", " ", html.unescape(t)).strip()
print(len(t.split()), "words")
print(t[:1500])
PY

Read the first 1,500 characters. Does it say what the page is about and what you offer? Or is it "Loading... Menu Sign in Cookie settings"?

Step 5: Compare with the rendered page

Open the same URL in your browser, open DevTools and compare the rendered DOM with raw.html. Note which important elements only exist after load: tabs, accordions filled by API calls, pricing tables from a widget, reviews from a third-party script.

Also test direct URL access. Single-page apps sometimes work when you click from the homepage but return a generic shell (or a 404) when the route is requested directly. Crawlers always request directly.

Step 6: Fix delivery, not the content

If the useful text is missing from the initial HTML, the usual options are:

Server-side rendering (SSR) for pages that change often.
Static generation / prerendering for pages that change rarely.
Moving critical facts into the server-rendered template (H1, intro, price, key FAQ answers) while keeping interactive extras client-side.

Don't build a separate "bot version" with different claims. Visitors and crawlers should get the same meaning; serving different content to bots is the kind of thing that backfires.

Step 7: Save before and after

When the fix ships, re-run the same curl and the same checks. Keep both files:

curl -sL "$URL" -o "after_$(date +%F).html"

A fix is done when the main text is present in a direct request, the page still works for humans, and the required links are there. That removes a technical barrier. It doesn't make the page relevant to every question or guarantee a citation, so track discovery and citations separately.

A quick checklist

[ ] Three representative pages chosen
[ ] Must-have facts listed per page
[ ] Raw HTML fetched (normal UA and crawler UA)
[ ] Status codes match between the two
[ ] Facts found in markup, not only in script data
[ ] Direct URL access tested for app routes
[ ] Fix verified with a before/after snapshot

If you want this check done automatically, AURA GEO audits the HTML your server actually returns, flags pages where little readable text comes through, and turns that into tasks you can hand to your own agent over MCP. There's a longer explainer here: how to tell whether a crawler can see the text on a JavaScript website.

Is That Really GPTBot? How to Find and Verify AI Crawlers in Your Server Logs

AURA GEO — Wed, 21 Oct 2026 16:14:49 +0000

You've allowed AI search bots in robots.txt. Great. But are they actually reaching your pages, and are the requests in your logs really from them?

The User-Agent header is just a string. Anyone can send GPTBot in a curl request. So "I saw GPTBot in my logs" proves very little until you check two things: what status code the bot got and whether the IP really belongs to the vendor.

This guide shows how to do both with standard tools: grep, awk and a short Python script.

1. Know which bots you're looking for

Major AI vendors now separate bots by purpose. This matters, because blocking the wrong one can quietly remove you from AI search answers. According to the vendors' own documentation:

Vendor	Bot	What it's for
OpenAI	`OAI-SearchBot`	Surfacing sites in ChatGPT search results
OpenAI	`GPTBot`	Crawling content that may be used for model training
OpenAI	`ChatGPT-User`	Visits triggered by a user's request in ChatGPT
Perplexity	`PerplexityBot`	Surfacing and linking sites in Perplexity search
Perplexity	`Perplexity-User`	Fetches triggered by a user's question
Anthropic	`Claude-SearchBot`	Improving Claude's search results
Anthropic	`ClaudeBot`	Collecting content that may be used for training
Anthropic	`Claude-User`	Fetches when a user asks Claude a question

Two practical notes from the docs:

OpenAI says sites opted out of OAI-SearchBot won't be shown in ChatGPT search answers (except as navigational links), and that a robots.txt change can take about 24 hours to take effect for search. Perplexity also mentions up to 24 hours.
User-triggered fetchers (ChatGPT-User, Perplexity-User) are documented as acting on behalf of a user, so robots.txt rules may not apply to them in the same way.

Sources: OpenAI crawlers, Perplexity crawlers, Anthropic crawlers.

2. Find AI bot hits in your access log

The examples assume the standard nginx/Apache "combined" log format, where the client IP is the first field, the status code is the ninth, and the user agent is the sixth field when you split on double quotes.

Count requests per bot:

grep -oE 'OAI-SearchBot|GPTBot|ChatGPT-User|PerplexityBot|Perplexity-User|Claude-SearchBot|ClaudeBot|Claude-User' \
  /var/log/nginx/access.log | sort | uniq -c | sort -rn

See which pages a specific bot requested:

grep 'OAI-SearchBot' /var/log/nginx/access.log \
  | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

If your logs are rotated and gzipped, use zgrep on access.log.*.gz the same way.

3. Check the status codes — this is where problems hide

A bot that "visits" but receives 403 or 429 isn't reading your content. This is very common on sites behind a CDN or WAF with aggressive bot protection.

grep -E 'OAI-SearchBot|PerplexityBot|Claude-SearchBot' /var/log/nginx/access.log \
  | awk '{print $9}' | sort | uniq -c | sort -rn

What to look for:

200 on real content pages — good.
301/302 — fine if they resolve to a 200; check that redirects don't loop or land on a login page.
403 — usually a firewall or bot-protection rule. Perplexity's docs, for example, describe allowing their bots in Cloudflare or AWS WAF by combining a user-agent match with their published IP ranges.
429 — rate limiting. Bots may back off and come back less often.
5xx — your server failed; the bot got nothing useful.

Important: if a CDN blocks a request at the edge, it may never reach your origin server logs at all. In that case, check the CDN's own firewall/security events too.

4. Verify the IP, not just the user agent

OpenAI and Perplexity publish JSON files with the IP ranges their bots use:

https://openai.com/searchbot.json
https://openai.com/gptbot.json
https://openai.com/chatgpt-user.json
https://www.perplexity.ai/perplexitybot.json
https://www.perplexity.ai/perplexity-user.json

Each file contains a prefixes list with ipv4Prefix or ipv6Prefix entries. Here's a small script that checks IPs from your log against them:

#!/usr/bin/env python3
"""verify_ai_bots.py — check which logged AI-bot IPs match official ranges."""
import ipaddress, json, re, sys, urllib.request

SOURCES = {
    "OAI-SearchBot": "https://openai.com/searchbot.json",
    "GPTBot": "https://openai.com/gptbot.json",
    "ChatGPT-User": "https://openai.com/chatgpt-user.json",
    "PerplexityBot": "https://www.perplexity.ai/perplexitybot.json",
    "Perplexity-User": "https://www.perplexity.ai/perplexity-user.json",
}

def load_ranges(url):
    req = urllib.request.Request(url, headers={"User-Agent": "range-check"})
    data = json.load(urllib.request.urlopen(req, timeout=20))
    nets = []
    for p in data.get("prefixes", []):
        cidr = p.get("ipv4Prefix") or p.get("ipv6Prefix")
        if cidr:
            nets.append(ipaddress.ip_network(cidr))
    return nets

ranges = {bot: load_ranges(url) for bot, url in SOURCES.items()}
seen = {}

with open(sys.argv[1], errors="replace") as log:
    for line in log:
        for bot in ranges:
            if re.search(rf"\b{re.escape(bot)}\b", line):
                ip = line.split()[0]
                seen.setdefault((bot, ip), 0)
                seen[(bot, ip)] += 1

for (bot, ip), hits in sorted(seen.items()):
    addr = ipaddress.ip_address(ip)
    ok = any(addr in net for net in ranges[bot])
    print(f"{'OK  ' if ok else 'FAKE?'} {bot:16} {ip:40} {hits} hits")

Run it:

python3 verify_ai_bots.py /var/log/nginx/access.log

Anything marked FAKE? claims to be an AI bot but comes from an IP outside the published ranges. Don't panic: the ranges change over time, so re-download them before drawing conclusions. But if a "GPTBot" is hammering your site from a random hosting provider, it's very likely not OpenAI.

Note: if your server sits behind a reverse proxy or CDN, the first log field may be the proxy's IP. Make sure you log the real client IP (for example via X-Forwarded-For / CF-Connecting-IP with a trusted-proxy config).

5. For Googlebot and Bingbot: reverse DNS

Google and Bing also publish IP lists (googlebot.json, bingbot.json), and both support the classic reverse-then-forward DNS check:

host 66.249.66.1
# → ...googlebot.com
host crawl-66-249-66-1.googlebot.com
# → must resolve back to 66.249.66.1

If the reverse lookup doesn't end in a Google-owned domain (such as googlebot.com or google.com), or the forward lookup doesn't return the same IP, it isn't Googlebot. The same idea works for Bingbot with search.msn.com.

6. What a verified visit does — and doesn't — prove

A verified OAI-SearchBot hit with a 200 tells you one thing: the bot could fetch that page. It does not prove that:

the page is indexed in that assistant's search,
the page will be cited in an answer,
anyone will click through.

Those are separate stages, and each needs its own evidence: a saved answer that links to your page, and a real referral visit in your analytics. Treat bot logs as the first link in that chain, not the finish line.

Quick checklist

[ ] List AI bots in your logs and count hits per bot
[ ] Check status codes for search bots — no 403/429 on public pages
[ ] Check CDN/WAF events for edge blocks that never reach your origin
[ ] Verify IPs against the official JSON ranges (re-download them regularly)
[ ] Use reverse DNS for Googlebot and Bingbot
[ ] Don't treat a crawl as a citation

If you'd rather not script this by hand, AURA GEO runs these accessibility checks for your site, keeps a crawl history for AI bots and shows real referral visits from AI assistants — and your own agent can pick up the fixes over MCP. There's also an English write-up on why a bot visit isn't a citation.

How to Get Your Site Ready for AI Search: robots.txt, llms.txt, Schema and AI Bots — A Practical Checklist

AURA GEO — Tue, 20 Oct 2026 17:07:16 +0000

More and more people skip the ten blue links and search directly in ChatGPT, Gemini, Perplexity or Claude. An assistant with web search finds pages on its own, reads them and builds an answer with links to its sources. If your site is blocked, slow or hard to parse for these systems, it won't make it into the answer — even if it ranks well on Google.

Preparing a site for this kind of search is called GEO (Generative Engine Optimization). The good news: the technical foundation of GEO overlaps almost entirely with good SEO. The bad news: there are a few places where sites break all the time, and a standard SEO audit doesn't catch them.

Below is a checklist you can get through in an evening. Every item can be checked with a terminal command or a couple of lines of code.

1. Allow AI bots in robots.txt

The first thing any crawler does is read robots.txt. Over the past few years, plenty of sites copied "protective" rules and accidentally locked out AI search bots.

It's important to tell two kinds of bots apart:

Training bots (e.g. GPTBot, ClaudeBot, Google-Extended) collect data to train models.
Search and user-triggered bots (e.g. OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) show up when an assistant is looking for an answer for a specific user. These are the ones that bring citations and clicks.

You can block training and still allow search. Example:

# AI search and user-triggered bots — allow
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-User
Allow: /

# Classic search engines
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

User-agent: *
Allow: /
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml

Check: make sure the file returns a 200 and doesn't contain a blanket Disallow: / for the agents you care about.

curl -s -o /dev/null -w "%{http_code}\n" https://example.com/robots.txt
curl -s https://example.com/robots.txt | grep -iE "user-agent|disallow"

Don't forget your CDN and WAF

A common trap: robots.txt allows everything, but Cloudflare or another WAF serves bots a 403 or a captcha. Check the server response with different User-Agents:

for ua in "OAI-SearchBot" "ChatGPT-User" "PerplexityBot" "Googlebot"; do
  code=$(curl -s -o /dev/null -w "%{http_code}" -A "$ua" https://example.com/)
  echo "$ua -> $code"
done

If anything isn't a 200, look at your bot-protection rules.

2. Serve text in HTML, not just via JavaScript

Many AI crawlers don't execute JavaScript, or only do so in a limited way. If your content only appears after an SPA hydrates, the bot sees an empty <div id="root">.

A simple check — download the page and see whether the main text is there:

curl -s https://example.com/pricing \
  | sed -e 's/<[^>]*>//g' | tr -s ' \n' | head -c 1500

If you see prices, descriptions and the FAQ, you're good. If all you get is the menu and empty space, you need SSR, SSG or prerendering (Next.js, Nuxt, Astro, Remix, etc.).

3. Add llms.txt

llms.txt is a proposed format: a Markdown file at the site root that briefly explains what the site is and lists key pages with descriptions. It isn't a standard, and no major assistant officially promises to use it. But it's cheap, useful for agents and developers, and it forces you to put the essentials of your product into words.

Minimal example:

# Example Inc.

> Example is an online booking service for small yoga studios.
> Operates in Latvia and Lithuania, interface in Russian and English.

## Main pages
- [Pricing](https://example.com/pricing): three plans, free trial.
- [API docs](https://example.com/docs/api): REST API for client bookings.
- [About](https://example.com/about): team, contacts, company details.

## FAQ
- [FAQ](https://example.com/faq): refunds, payments, integrations.

A good live example is AURA GEO's llms.txt: besides the list of pages, it honestly states what the service does not do (for instance, that it doesn't "index sites in AI directly"). Explicit limitations like this are useful — models have less to fill in on their own.

4. Schema.org markup (JSON-LD)

Structured data helps search engines — and, indirectly, AI — understand who you are and what you sell. The minimum set for a SaaS or service:

Organization — name, logo, sameAs with links to external profiles;
WebSite — site name;
SoftwareApplication or Product — for the product and pricing;
FAQPage — for a Q&A block;
Article — for blog posts and knowledge-base articles.

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "Organization",
  "name": "Example Inc.",
  "url": "https://example.com",
  "logo": "https://example.com/logo.png",
  "sameAs": [
    "https://github.com/example",
    "https://www.producthunt.com/products/example"
  ]
}
</script>

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "FAQPage",
  "mainEntity": [{
    "@type": "Question",
    "name": "Is there a free plan?",
    "acceptedAnswer": {
      "@type": "Answer",
      "text": "Yes, you can connect one site for free."
    }
  }]
}
</script>

The golden rule: markup must match the visible text. If your JSON-LD contains an FAQ that isn't on the page, that's worse than having no markup at all.

Check it with Google's Rich Results Test or validator.schema.org.

5. Sitemap, canonical and noindex

AI assistants with search lean heavily on the Google and Bing indexes. A page that isn't indexed has almost no chance of making it into an answer. Check:

# sitemap is reachable
curl -s -o /dev/null -w "%{http_code}\n" https://example.com/sitemap.xml

# no accidental noindex, and canonical points to itself
curl -s https://example.com/pricing | grep -iE 'noindex|rel="canonical"'
curl -sI https://example.com/pricing | grep -i x-robots-tag

Also go into Google Search Console and Bing Webmaster Tools and review the report on non-indexed pages. Bing matters especially: several AI assistants rely on its index.

6. IndexNow: report updates right away

IndexNow is a protocol for notifying Bing, Yandex and other participating search engines about new or changed URLs. Google doesn't use it, but for Bing it's the fastest route.

# the key must be available at https://example.com/<KEY>.txt
curl -X POST "https://api.indexnow.org/indexnow" \
  -H "Content-Type: application/json; charset=utf-8" \
  -d '{
    "host": "example.com",
    "key": "<KEY>",
    "urlList": ["https://example.com/pricing", "https://example.com/faq"]
  }'

Keep in mind: a successful submission only means "notification received" — not "page indexed", and certainly not "AI is citing it". These are separate stages, and each is worth checking on its own.

7. Write pages that are easy to cite

Technical work opens the door, but content is what gets cited. What works:

Answer in the first paragraph. Question in the heading, a direct 2–3 sentence answer right below it.
Specifics. Prices, timelines, countries, limitations, versions. "Free for one site" beats "affordable pricing".
Verifiable facts. Last-updated date, author, source of the numbers.
Structure. H2/H3, lists, comparison tables.
One page per question. A "How much does X cost" page gets cited more than a paragraph buried in a giant landing page.
Don't pass off demo data as facts. If your landing page has nice example numbers in an interface mockup, label them as examples — otherwise a model may repeat them as real.

8. Measure: bot logs and AI referrals

Without measurement, GEO turns into guesswork. The minimum:

Server logs. Look for AI bot visits:

grep -iE "GPTBot|OAI-SearchBot|ChatGPT-User|PerplexityBot|ClaudeBot|Claude-User" \
  /var/log/nginx/access.log | awk '{print $1, $7, $12}' | tail -50

Referrers in analytics. Visits come from domains like chatgpt.com, perplexity.ai, gemini.google.com, copilot.microsoft.com. Create a separate segment for them.
Manual answer checks. Ask your customers' neutral questions in fresh chats with search enabled, save the answer and the list of sources, and repeat after making changes. A single check proves nothing — the trend is what matters.

You can stitch all of this together with scripts, or keep it in one dashboard. For example, AURA GEO runs a technical audit (robots.txt, accessible HTML, noindex, sitemap), counts recognized AI referrals, lets you hand tasks to your own AI agent via MCP, and submits updated pages through IndexNow. The basic plan for one site is free.

Final checklist

Item	How to check
robots.txt allows AI search bots	`curl robots.txt`, grep for User-agent
WAF doesn't block bots	`curl -A "ChatGPT-User"` → 200
Text is in HTML without JS	`curl` + strip tags
llms.txt at the root	`curl /llms.txt`
JSON-LD: Organization, FAQPage	Rich Results Test
Sitemap, canonical, no noindex	grep + Search Console / Bing WMT
IndexNow configured	200/202 response from the API
Pages answer specific questions	content review
Bot logs and AI referrers are tracked	log grep, analytics segment

FAQ

What is GEO in plain terms?

GEO (Generative Engine Optimization) is preparing a site so that AI assistants with web search can find, read and cite its pages. SEO is about rankings in regular search results; GEO is about showing up in answers from ChatGPT, Gemini, Perplexity and others.

Should I block GPTBot to protect my content?

That's your call. Blocking training bots (GPTBot, ClaudeBot, Google-Extended) doesn't affect search bots like OAI-SearchBot and ChatGPT-User. If you want citations, don't block search and user-triggered bots.

Is llms.txt required?

No. It's an optional format, and major assistants don't officially promise to use it. But it's cheap, useful for AI agents and developers, and helps you describe your product clearly.

Does schema.org help you get into AI answers?

There are no direct guarantees, but markup helps search engines understand the entities on a page, and AI search relies heavily on their indexes. The key is that markup matches the visible text.

Does submitting to IndexNow mean the page will show up in ChatGPT?

No. IndexNow only notifies search engines. Indexing, a bot visit, a citation and a click are separate stages, and each needs to be checked separately.

How do I know AI is already sending traffic?

Look for chatgpt.com, perplexity.ai and gemini.google.com referrers in your analytics and AI bot visits in your logs. And regularly check how assistants answer your customers' questions.