DEV Community: Ali Baizhanov

Can ChatGPT access your website? A 2-minute check (robots.txt, firewall, HTML)

Ali Baizhanov — Mon, 09 Nov 2026 12:49:39 +0000

Originally published on pagelens.io.

If ChatGPT can't read your pages, it can't quote them, recommend them or link to them. Most sites never check. Three things decide whether an AI assistant gets in, and you can test all three for free in a couple of minutes.

1. robots.txt: are AI bots allowed?

Open yoursite.com/robots.txt. Look for groups that name AI user agents, or a User-agent: * group with Disallow: /, which blocks everyone. The ones that matter for answers today:

OAI-SearchBot builds the index ChatGPT search uses. Blocked here, you're unlikely to be cited in ChatGPT answers.
ChatGPT-User, Claude-User and Perplexity-User fetch a page when a person asks the assistant about it.
GPTBot and ClaudeBot collect training data. Blocking them is a choice about training, not about appearing in answers.

Reading robots.txt by hand is easy to get wrong: a specific group overrides *, and the most specific path wins. A free AI crawler checker reads it the way crawlers do and shows each AI bot as allowed or blocked.

2. Your CDN or firewall

robots.txt is a request. A firewall is a wall. Many sites block AI bots at the CDN without knowing it:

Cloudflare has a one-click setting that blocks AI bots, and since mid-2025 it's offered by default to new sites. Check the bot and AI crawler settings in your dashboard.
Bot protection on other CDNs and WAFs, and some WordPress security plugins, challenge unknown agents with a JavaScript or CAPTCHA page that AI fetchers can't pass.

Quick test: ask ChatGPT (search on) or Claude to "open https://yoursite.com/pricing and summarise it". If it can't access the page, or summarises something generic, the request was probably stopped before it reached your content. Try a deep page, not only the home page.

3. Is the content in the HTML?

AI crawlers download the HTML and mostly don't run JavaScript. If your page is a single-page app that renders everything in the browser, the bot may see an empty <div id="root">.

curl -s -A "ChatGPT-User" https://yoursite.com/pricing | grep -i "your headline"

Nothing back? Render that page on the server or pre-render it at build time (Next.js, Nuxt, Astro and SvelteKit all can).

4. Check that they actually come

Allowed doesn't mean visited. AI bots never show up in JavaScript analytics, so you need server or CDN logs. I wrote up five ways to see AI crawlers from grep to automatic, and what the first week looked like on a brand new site.

Checklist

Check	How	Fix
robots.txt allows search and user agents	a robots.txt checker	allow `OAI-SearchBot`, `*-User` agents, `PerplexityBot`
CDN/firewall lets them through	ask ChatGPT to open a deep URL	turn off AI bot blocking, or allow verified AI bots
Content is in the HTML	`curl` and look for your text	SSR or pre-render
They actually visit	server or CDN logs	link new pages, submit a sitemap

I build PageLens Analytics (pagelens.io), which shows AI crawler reads per page. The checks above work with any stack and cost nothing.

Do AI bots actually read llms.txt? 8 days of server logs

Ali Baizhanov — Mon, 09 Nov 2026 12:43:53 +0000

Originally published on pagelens.io.

Every GEO guide now says "add an llms.txt". I wanted to know if any AI system actually fetches it, so I checked the server logs of a new site that had one from day one.

Setup: one small site launched 7 Oct 2026, every request logged on the server (so non-JS bots count), user agents matched to AI companies, IPs checked against published ranges where they exist. Window: 8–15 Oct.

Numbers

File	AI bot requests	Share
All pages	567	100%
`/robots.txt`	73	13%
`/sitemap.xml`	50	9%
`/llms.txt`	6	1%

The six llms.txt requests: ClaudeBot, GPTBot, meta-externalagent, ChatGPT-User, PerplexityBot, OAI-SearchBot. Once each. The four OpenAI/Perplexity ones were IP-verified.

What I take from it

It does get fetched. Every major AI company's bot grabbed it within a week.
It's not treated like robots.txt. ClaudeBot asked for robots.txt 46 times that week and llms.txt once.
A fetch isn't usage. Can't tell from logs whether anything followed its links. The most-read pages were linked from everywhere anyway.
ChatGPT-User fetched it once. That agent acts for a person mid-conversation, so the only hint of answer-time use is a single event.

So: add one, it's ten minutes and free. Don't expect it to get you into AI answers. Robots.txt access, CDN not blocking bots, content in the HTML and links from other sites still do the work.

Check your own logs:

grep -i "llms.txt" access.log | grep -Ei "GPTBot|ClaudeBot|PerplexityBot|OAI-SearchBot|ChatGPT-User"

Curious if people with bigger sites see the same pattern.

Cheaper Peec AI alternatives in 2026: 6 AI visibility tools compared, with prices

Ali Baizhanov — Thu, 22 Oct 2026 11:41:37 +0000

Originally published on pagelens.io.

Peec AI is one of the best-known tools for tracking whether ChatGPT, Perplexity and Gemini name your brand. It is built for marketing teams and agencies, and priced for them: third-party reviews put the Starter plan at around $95 a month for 50 prompts (Peec's own pricing page didn't show prices when I checked). If you run one product or a small site, that can be a lot for a first look at AI search.

Disclosure: I make PageLens Analytics, which is on this list, and it is the narrowest tool here for tracking AI answers. I've tried to say plainly where the others are the better pick. Prices are the cheapest plan on each vendor's pricing page on 22 October 2026; they change often, so check before you buy.

Quick comparison

Tool	From	Prompts	AI engines	Trial
Peec AI (for reference)	about $95/mo	50	3 of your choice, more as add-ons	Free trial
Rankscale	$17/mo (yearly only); Pro $99/mo	Credit-based	Pro: ChatGPT, Perplexity, AI Mode, AI Overviews, Gemini, Claude, Copilot and more	7 days on Pro
Otterly.AI	$29/mo	15	ChatGPT, AI Overviews, Perplexity, Copilot; others as add-ons	Free trial
Promptmonitor	$39/mo	25, checked twice a week	ChatGPT, Gemini, Perplexity	7 days, no card
AIclicks	$59/mo	30	3 engines	3 days
LLMrefs	$79/mo	500	All major engines, no add-on fees	7 days, plus a free account
PageLens Analytics	$5/mo	3, asked daily (10 on $12, 25 on $29)	Perplexity	7 days, no card

Rankscale

The lowest sticker price on this list: an Essentials plan at $17 a month, billed yearly. The pricing page doesn't spell out what Essentials includes, so the plan to compare is Pro at $99 a month, which covers the widest set of engines here, including Claude, DeepSeek, Mistral and Copilot, with usage paid in credits. Best for: people who want many engines and don't mind credit-based billing.

Otterly.AI

The Lite plan is $29 a month for 15 prompts across ChatGPT, Google AI Overviews, Perplexity and Microsoft Copilot, with Gemini, AI Mode and Claude as add-ons. The next plan jumps to $189 for 100 prompts. Best for: a small, focused set of questions on the engines most people use.

Promptmonitor

$39 a month for 25 prompts on ChatGPT, Gemini and Perplexity, checked twice a week rather than daily, with a 7-day trial that doesn't ask for a card. Best for: a cheap start with a no-card trial.

AIclicks

The Starter plan is $59 a month for 30 prompts on three engines, with a 3-day trial. Best for: teams that want a bit more room than the $29–39 plans.

LLMrefs

$79 a month buys 500 prompts with every major engine included and no add-on fees, which makes it by far the most prompts per dollar here. There is also a free account to try it. Best for: tracking a lot of questions across every engine without paying per model.

PageLens Analytics

PageLens Analytics is web analytics first, and it covers AI search from the other side. It asks your questions in Perplexity every day (3 on the $5 plan, 10 on $12, 25 on $29), shows who is named instead of you and which pages the answer was built from, and turns that into a list of places to get mentioned. It only tracks Perplexity, so if you need ChatGPT, Gemini and Claude answers, pick one of the tools above.

What it adds is what happens on your site: which pages AI crawlers read, which ones ChatGPT, Claude and Perplexity open while answering someone, the visits that come from AI assistants, and whether those visitors sign up or pay, next to cookieless analytics with funnels and session replay. Best for: founders and small sites that want web analytics and a first look at AI search in one tool. PageLens Analytics vs Peec AI in detail.

How to choose

You need several engines and many prompts: LLMrefs gives the most for the money; Rankscale Pro if you want the widest engine list.
You want to test a handful of questions cheaply: Otterly Lite or Promptmonitor.
You want to know if AI sends you visitors and customers, not only whether it names you: PageLens Analytics, alone or next to one of the trackers above.

Whatever you pick, start with the questions your customers actually ask, and check whether AI bots can reach your site at all: the free AI crawler checker tests that in a few seconds.

Which of these have you tried, and what made you stay or leave?

How to see which AI crawlers read your website (5 ways, with copy-paste commands)

Ali Baizhanov — Tue, 13 Oct 2026 09:59:02 +0000

Originally published on pagelens.io.

AI crawlers read websites all day, and almost no site owner can see it. The reason is simple: analytics tools count visitors with JavaScript that runs in a browser, and AI crawlers download the HTML without running any JavaScript. To see them you have to look where the request arrives: your server or your CDN.

Here are five ways to do that, starting with the ones that cost nothing.

First: which bots to look for

There are three kinds, and they mean different things:

Kind	Examples	What a request means
Training crawlers	GPTBot, ClaudeBot, CCBot	The page was collected for training a model
AI search crawlers	OAI-SearchBot, Claude-SearchBot, PerplexityBot	The page went into the index an assistant searches
Fetches for a person	ChatGPT-User, Claude-User, Perplexity-User	Someone asked an assistant about this right now, and it opened your page

The third kind is the one to watch. It is the closest thing to "my page was used in an answer".

1. Server access logs (free, if you run the server)

Nginx and Apache already write every request to a log. Count requests per AI bot:

grep -oE "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User" /var/log/nginx/access.log \
  | sort | uniq -c | sort -rn

See which pages one of them asked for (the path is the 7th field in the default log format):

grep "ChatGPT-User" /var/log/nginx/access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

And the requests that ended in a 404, which are pages a bot expected to find and didn't (the status is the 9th field):

grep -E "GPTBot|ClaudeBot|PerplexityBot" /var/log/nginx/access.log | awk '$9 == 404 {print $7}' | sort | uniq -c | sort -rn | head

Good for: a one-off look on a VPS. Limits: logs rotate after days or weeks, a CDN cache in front of the server hides requests from it, and you get a number, not a trend.

2. Your CDN's dashboard (free, if your site runs through one)

A CDN sees every request before your server does, including the ones it answers from cache. Cloudflare has an AI crawler section in its dashboard that lists AI crawlers requesting your site and lets you allow or block each. Other CDNs and hosts (Vercel, Netlify, Fastly) show requests in their logs, where you can filter by user agent.

Good for: a quick answer to "which AI companies crawl me, and how much". Limits: usually no view per page over time, and nothing about what happened next (did anyone visit, did they buy).

3. A few lines of middleware (free, for Next.js and similar)

If your framework lets you run code on every request, log the bots there. In Next.js:

// middleware.ts (proxy.ts in Next.js 16)
import { NextResponse, type NextRequest } from 'next/server'

const AI = /GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User/i

export function middleware(req: NextRequest) {
  const bot = req.headers.get('user-agent')?.match(AI)?.[0]
  if (bot) console.log(JSON.stringify({ bot, path: req.nextUrl.pathname }))
  return NextResponse.next()
}

export const config = { matcher: ['/((?!_next/static|_next/image|favicon.ico).*)'] }

The lines land in your host's function logs. The same idea works in Express, a Cloudflare Worker or any server framework.

Limits: you still have to store, count and chart them, and middleware runs before the page, so it doesn't know the response status.

4. WordPress

WordPress runs PHP on every uncached request, so a short snippet in a small plugin or your child theme can log AI bots:

add_action('template_redirect', function () {
    $ua = $_SERVER['HTTP_USER_AGENT'] ?? '';
    if (preg_match('/GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|PerplexityBot|Perplexity-User/i', $ua, $m)) {
        error_log('AI bot ' . $m[0] . ' ' . ($_SERVER['REQUEST_URI'] ?? '/'));
    }
});

The catch: page caches (WP Rocket, LiteSpeed, a CDN) serve cached pages without running PHP, so cached pages go uncounted. Exclude bot user agents from the cache, or count at the CDN instead.

5. Analytics that reads bots from your server (automatic)

The four ways above give you raw requests. Turning them into something you can act on means storing them, grouping them by page and by company, and joining them with what people did afterwards. I built this into PageLens: a small snippet on your server reports bot requests, and the dashboard shows which pages AI crawlers read, which ones assistants opened to answer people, the 404s they hit, and the visits and revenue that came from AI assistants.

Check that the bots are real

Anyone can send a request that says GPTBot. OpenAI, Anthropic and Perplexity publish the IP ranges their bots use, so a request with a bot's name from another address is an impostor. When I checked 740 top websites, 9% turned away requests carrying AI bot names even though their robots.txt allowed those bots.

What to do with the numbers

Read a lot, never opened for an answer: AI knows the page exists and doesn't find it useful for questions. Put the question in the heading and the answer in the first two sentences.
Opened for answers, no visits: you are being used as a source without being clicked. Check how the page describes you and whether it gives a reason to click through.
Important page never read: link to it from your home page and sitemap, and check robots.txt and your CDN.
Bots hitting 404s: they expect a page there, often /pricing, /docs or /llms.txt. Create it or redirect.

The field numbers in the commands are for the default "combined" log format of Nginx and Apache. If your format differs, adjust them.

What do you use to watch AI crawlers on your sites?

We checked 740 of the world's top websites for AI bots. Most that block GPTBot also block ChatGPT search

Ali Baizhanov — Mon, 12 Oct 2026 08:27:34 +0000

Originally published on pagelens.io.

Most advice about AI search starts with "check your robots.txt". I wanted to know what the biggest sites actually do, so I ran an AI visibility check on the top 1,500 domains of the Tranco list (list 4672X), a research ranking of the most visited domains, on October 12, 2026.

Many of those domains are APIs and CDNs with no home page (gstatic.com, akamai.net and so on). 740 served a home page to a normal browser request, and every number below is out of those 740 websites.

November update: the same check a month later

On November 9, 2026 I ran the same check on the newest Tranco list (8P8NV). 720 of its top 1,500 domains served a home page. The list changes a little every month, so treat moves of a point or two as noise.

Signal	Oct 12 (740 sites)	Nov 9 (720 sites)
robots.txt blocks GPTBot	14.7%	15.0%
robots.txt blocks ClaudeBot	13.6%	14.6%
robots.txt blocks meta-externalagent	12.6%	14.0%
robots.txt blocks Applebot-Extended	9.3%	10.7%
robots.txt blocks OAI-SearchBot (ChatGPT search)	7.8%	8.2%
robots.txt blocks ChatGPT-User	9.7%	9.4%
Blocks at least one agent that answers people	13.0%	12.5%
GPTBot blockers that also block ChatGPT search or browsing	63% (69 of 109)	61% (66 of 108)
Publishes an llms.txt (Markdown with a title)	about 14%	17.9%
Structured data (JSON-LD) on the home page	45%	46%

llms.txt is the one clear change: up about four points in a month, from roughly one site in seven to almost one in five.
Training blocks creep up a little (ClaudeBot, Meta and Apple), while blocks of the agents that answer people stay flat.
Server-side blocks: 85 sites (11.8%) turned away requests named after AI bots while a made-up crawler got through. October's 9.3% came after a second re-test that removed one-off failures, which this run didn't do, so the two aren't directly comparable.

The rest of this post describes the October run.

The short version

15% block GPTBot in robots.txt, and about as many block ClaudeBot (14%) and Google-Extended (13%).
13% block at least one agent that answers people: OAI-SearchBot, ChatGPT-User, Perplexity's bots or Claude-User.
Of the 109 sites that block GPTBot, 69 (63%) also block ChatGPT's search or browsing agent. Only 40 block training and keep ChatGPT search open.
9% turn away requests that carry an AI bot's name at the server, even though their robots.txt allows those bots.
About 14% publish an llms.txt; 45% have structured data on the home page; 38% have no sitemap I could find.

1. robots.txt: training bots are blocked more than answer bots

Share of the 740 websites whose robots.txt keeps each agent off the home page (in practice, off the site):

Agent	What it does	Blocked
CCBot	Common Crawl, an open dataset many AI labs train on	16.8%
Bytespider	ByteDance training crawler	15.7%
GPTBot	OpenAI training crawler	14.7%
ClaudeBot	Anthropic training crawler	13.6%
Google-Extended	Gemini training (a robots.txt token, not a separate crawler)	13.2%
meta-externalagent	Meta training crawler	12.6%
PerplexityBot	Perplexity search index	10.9%
ChatGPT-User	ChatGPT opening a page for a person	9.7%
Applebot-Extended	Apple Intelligence training	9.3%
Perplexity-User	Perplexity opening a page for a person	8.4%
Claude-User	Claude opening a page for a person	8.0%
OAI-SearchBot	ChatGPT search index	7.8%

Most of these are deliberate: 92 of the 109 GPTBot blocks name GPTBot explicitly, the rest come from a rule for all bots.

Each agent has a page in this AI bots directory with its user agent, robots.txt rules and published IP ranges.

2. Blocking GPTBot usually blocks ChatGPT search too

GPTBot only collects training data. Whether a site appears in ChatGPT search depends on OAI-SearchBot, and ChatGPT-User is the agent that opens a page while ChatGPT answers someone. OpenAI documents them as separate agents. One nuance: OpenAI says robots.txt rules may not apply to ChatGPT-User, because a person starts those requests, so a robots.txt block of it is a stated preference more than a guarantee.

Of the 109 websites that block GPTBot:

69 also block OAI-SearchBot or ChatGPT-User. Among them are cnn.com, bbc.com, reddit.com, imdb.com, espn.com, nature.com and Amazon's stores. They opt out of ChatGPT as a whole.
40 block only training and leave ChatGPT search and browsing open, including forbes.com, bloomberg.com, airbnb.com, figma.com, ted.com and nintendo.com.

Both are legitimate choices. The second one is what most businesses want: stay out of training, stay in the answers. If you copied a "block AI bots" robots.txt from a blog post, check which group you're in.

3. robots.txt isn't the whole story

For each site I also requested the home page with each bot's name in the user agent (GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot) and compared the answer with a normal browser request. On 130 sites, at least one bot-named request was turned away (a 403, a challenge page or no answer) although robots.txt allows that bot.

I then re-tested those 130 sites with a fourth request: a generic, made-up crawler name. That splits them into groups:

69 (9.3% of all sites) let the browser and the generic crawler through, but turn away requests named after AI bots. 37 of them turn away ChatGPT-User, the agent that opens pages to answer people.
37 (5%) turn away any bot-like request, AI or not.
19 did not repeat on the second test, and 5 blocked the browser too.

One caution: these requests don't come from OpenAI's or Anthropic's servers. Some CDNs block requests that claim to be GPTBot but don't come from the IP ranges OpenAI publishes, and let the real GPTBot in. From the outside, "blocks AI bots" and "blocks impostors" look the same, which is why I don't name sites in this group. If you're one of them, your CDN's bot settings (in Cloudflare: Security → Bots) will tell you which it is.

The useful lesson for a site owner: robots.txt can say "allowed" while the server says no, and nothing in robots.txt will show it.

4. Can AI read the page once it gets it?

AI crawlers don't run JavaScript. On 22% of home pages, fewer than 120 words were readable in the HTML before scripts run. This is an upper bound: some sites send bots a short interstitial page instead of the real one. But for a single-page app, an empty HTML shell is what every AI crawler sees.

Signal	Share of the 740 sites
Has a robots.txt	76%
Has a sitemap (in robots.txt or at /sitemap.xml)	62%
Has structured data (JSON-LD) on the home page	45%
Missing a title or meta description	21%
Publishes an llms.txt (Markdown with a title)	14%

llms.txt is still a minority practice even among the largest sites: about one in seven. Several more sites answered /llms.txt with something that isn't one (an IP address, a bare "OK", an error page), which a naive check would count.

What to do on your own site

Decide training and answers separately. Blocking GPTBot and ClaudeBot keeps you out of training; to stay in ChatGPT and Claude answers, allow OAI-SearchBot, ChatGPT-User, Claude-SearchBot and Claude-User.
Check the server, not only the file. CDN bot settings and firewalls override robots.txt.
Make sure the text is in the HTML. Server rendering or prerendering for the pages you want quoted.
Then measure what AI actually does. Server logs show which pages AI crawlers read and which ones assistants open to answer people.

If you want to run the same check on your site, I built it as a free tool (no sign-up): pagelens.io/tools/ai-crawler-checker.

Method and limits

Domains: top 1,500 of Tranco list 4672X. A domain counts as a website if a normal browser request to its home page got a page (2xx or 3xx, no challenge). 740 did.
robots.txt rules were evaluated for the home page path with the most specific matching user-agent group, as crawlers do.
Bot requests used each bot's name and said they came from the checker. Run from Central Asia on October 12, 2026; results can differ by location and change at any time.
These are shares, not judgements: blocking AI bots is a valid choice.

Happy to answer questions about the data or the method in the comments.