DEV Community: Gabriele

The AI crawlers in your logs, and what blocking each one actually costs

Gabriele — Thu, 03 Sep 2026 21:31:31 +0000

Sooner or later someone forwards you a screenshot of the access log and asks whether to block "the AI bots". It is a reasonable question with an unreasonable premise: there is no single population there. There are at least three, they want different things, and blocking them costs you different things.

Here is the map I wish I had the first time I was asked.

Three jobs, not one

Training crawlers collect pages to train a future model. Nothing you do today changes a model that already shipped, and nothing about your search ranking depends on them.

Live-fetch bots fetch one page because a user asked something right now. Block these and you disappear from the answer that was about to cite you.

Answer-engine indexers build an index used to answer questions with citations. Closest thing to search, and the one where blocking most directly costs you referral traffic.

The same company usually runs one of each, under different names. That is the part people miss.

User-agent	Operator	Job	What blocking it costs you
`GPTBot`	OpenAI	Training	Nothing in the short term
`OAI-SearchBot`	OpenAI	Search index for ChatGPT	Visibility in ChatGPT search
`ChatGPT-User`	OpenAI	Live fetch on user request	Your page cannot be opened when a user asks about it
`ClaudeBot`	Anthropic	Training	Nothing in the short term
`Claude-Web`	Anthropic	Live fetch	Same as `ChatGPT-User`
`PerplexityBot`	Perplexity	Index for cited answers	Citations, and the referral traffic with them
`Google-Extended`	Google	Gemini training only	Nothing in Search — see below
`CCBot`	Common Crawl	Open dataset many models train on	Nothing directly; wide indirect reach
`Bytespider`	ByteDance	Training	Nothing in the short term
`Applebot-Extended`	Apple	Apple Intelligence training	Nothing in Search — `Applebot` is separate
`Amazonbot`	Amazon	Assistant and search	Alexa-surface visibility
`meta-externalagent`	Meta	Training	Nothing in the short term

The one that causes the most damage by accident

Google-Extended is not Googlebot.

It controls whether your content is used to train Gemini. It has no effect on crawling for Search, on indexing, or on ranking. Blocking it does not deindex you and does not cost you a position.

Googlebot is what crawls for Search. If you block that one, you leave the index.

I have seen the two conflated in both directions: people who avoid blocking training because they fear losing rankings they were never going to lose, and — worse — people who block Googlebot believing they are opting out of AI. The names are similar; the consequences are not.

Same pattern at Apple: Applebot crawls for Siri and Spotlight, Applebot-Extended is the training opt-out. Different bots, different consequences.

The trade-off nobody can settle for you

For scale, we crawled the robots.txt of a large sample of sites for a study on this exact question: 17.8% block GPTBot against 2.5% for Googlebot, and 12.3% block PerplexityBot, a crawler that does not train models at all. That last figure is the accidental-damage bucket, measured in the wild.

The argument for blocking training crawlers is simple: your content is an asset, training on it is a use, and you get nothing back.

The argument against is subtler. Assistants that cite sources send traffic, and increasingly they are the search interface. A site that is invisible to them is invisible in a place where people now ask questions. And the distinction between "training" and "answering" is blurrier in practice than the bot names suggest.

The position most people land on, once it is laid out this way:

Block training — GPTBot, ClaudeBot, CCBot, Google-Extended, Bytespider, Applebot-Extended, meta-externalagent.
Allow live-fetch and answer indexes — ChatGPT-User, Claude-Web, OAI-SearchBot, PerplexityBot.
Never touch Googlebot, Bingbot.

Which comes out as:

User-agent: *
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: meta-externalagent
Disallow: /

Sitemap: https://example.com/sitemap.xml

That is a position, not a default. A documentation site that wants to be the answer everywhere might block nothing. A publisher whose archive is the product might block everything. Both are coherent; drifting into one by accident is not.

Three things to know before you edit the file

A specific group replaces the wildcard group — it does not extend it. If you write a group for GPTBot, that group is the whole ruleset for GPTBot. Rules under User-agent: * are not inherited. This is the most common way a hand-written file ends up doing the opposite of what was intended.

robots.txt is a request, not a fence. Well-behaved crawlers obey it. Nothing enforces it. If a bot must be stopped rather than asked, that is a job for authentication or blocking at the edge, and the operator list above is the wrong tool.

Test the file before you ship it. The precedence rules are unintuitive: the longest matching rule wins, not the first, and on a tie Allow beats Disallow. A file that reads correctly to a human can behave differently to a parser.

I maintain two small browser-based tools for exactly this: a robots.txt generator with the AI-crawler presets above, and a tester that shows which line decided each verdict. Neither uploads anything — both run on the page.

The robots.txt rule almost every tester gets wrong

Gabriele — Thu, 03 Sep 2026 21:30:39 +0000

Here is a robots.txt. Is /private/report.pdf crawlable?

User-agent: *
Disallow: /private/
Allow: /private/report.pdf

If you answered "no, it's under /private/", you have company — including, in my experience, a few online testers. The correct answer is yes, and the reason is a rule that is easy to state and easy to implement backwards.

I ran into this while writing a robots.txt tester, which is a good way to discover that you did not understand a spec as well as you thought.

The three rules that decide everything

RFC 9309 standardised what Google's parser had been doing for years. Three rules, applied in order:

1. One group wins, and only one.

A crawler scans the file for the group whose User-agent names it most specifically. User-agent: * is a fallback, not a base class. If a group names the bot directly, the wildcard group is ignored entirely — its rules are not merged in, not inherited, not consulted.

User-agent: *
Disallow: /

User-agent: GPTBot
Allow: /

GPTBot may crawl everything. It does not "inherit" the Disallow: /. This surprises people who read the file top-down like a firewall.

2. The longest matching rule wins — not the first.

This is the one in the example above. /private/ is 9 characters, /private/report.pdf is 19. The longer pattern matches, so the Allow decides, and that single file stays crawlable while the rest of the folder does not.

Order in the file is irrelevant. You can put the Allow first, last, or between two unrelated lines; the outcome is the same. An implementation that returns the first match will disagree with Google on every file that uses this pattern — and it is a common pattern, because it is the only way to expose one file from a blocked directory.

3. On a tie, Allow wins.

User-agent: *
Disallow: /a
Allow: /a

Same length, both match. /a is crawlable. This is the case I would bet most hand-rolled parsers get wrong, because it only shows up if you deliberately test for it.

A test file for your tester

Paste this into whatever tool or library you rely on, then check the verdicts against the right-hand column. It takes a minute and it is worth doing before you trust a tool with a migration.

User-agent: *
Disallow: /private/
Allow: /private/report.pdf
Disallow: /a
Allow: /a
Disallow: /*.pdf$

User-agent: GPTBot
Allow: /

URL	Correct verdict	Why
`/private/notes.txt`	Blocked	Only `Disallow: /private/` matches
`/private/report.pdf`	Allowed	Longer `Allow` beats shorter `Disallow`
`/a`	Allowed	Equal length, `Allow` wins the tie
`/docs/manual.pdf`	Blocked	`/*.pdf$` matches, `$` anchors the end
`/docs/manual.pdf.html`	Allowed	`$` anchors — the path does not end in `.pdf`
`/private/notes.txt` as GPTBot	Allowed	Specific group replaces the wildcard group entirely

If your tool disagrees on rows 2, 3 or 6, it is not implementing RFC 9309, and its verdicts will drift from reality exactly where it matters.

Two more things that bite

An empty Disallow: restricts nothing. It is the standard way of saying "everything is open". More than once I have seen it read as "block everything" — the opposite of the truth. Disallow: / with the slash is what blocks.

* and $ are the only wildcards. No regex, no character classes. * matches any run of characters, $ anchors the end of the path. Everything else is a literal, including ? and ..

Where this matters more than it used to

The rules above have been stable for years. What changed is who is reading the file.

A robots.txt now governs at least three different populations, and they are separate user-agents doing separate jobs:

Search crawlers — Googlebot, Bingbot. They decide whether you appear in search results.
AI training crawlers — GPTBot, ClaudeBot, CCBot, Google-Extended, Bytespider. They collect pages to train models.
Live-fetch bots — ChatGPT-User, Claude-Web, PerplexityBot. They fetch a page because a user asked something right now.

The distinction that trips people up most: Google-Extended controls Gemini training and has no effect whatsoever on Google Search ranking. Blocking it does not deindex you. Googlebot is what crawls for Search. They are different bots with different rules, and a group naming one says nothing about the other.

Which means a file like this is coherent, and increasingly common:

User-agent: *
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Search engines crawl everything. Training crawlers get nothing. Nothing about your rankings changes.

What robots.txt still cannot do

It is a crawling instruction. It is not an access control and not an indexing control.

A disallowed URL can still appear in search results if other sites link to it — the crawler never fetches the page, so it never sees your noindex, and it can list the URL from the link alone. If you want a page out of the index, allow the crawl and serve noindex. If you want it private, require authentication. And robots.txt is a public file: writing Disallow: /secret-admin/ publishes the existence of /secret-admin/ to anyone who looks.

I built a robots.txt tester that implements the precedence above and shows you the exact line that decided each verdict, including the AI crawlers. It runs entirely in the browser, so you can paste a file that is not published yet. The six rows in the table are its test suite.

Debugging a Python requests proxy issue: works in curl but fails in script

Gabriele — Wed, 11 Mar 2026 10:00:02 +0000

While debugging a Python script that sends HTTP requests through a proxy, I ran into a strange issue.

The proxy works perfectly when tested with curl:

curl -x http://user:pass@proxy:port https://quantumproxies.io

This returns the correct IP, so the proxy itself is clearly working.

However the equivalent Python code fails:


python
import requests

proxy = "http://user:pass@proxy:port"

proxies = {
    "http": proxy,
    "https": proxy
}

url = "https://httpbin.org/ip"

response = requests.get(
    url,
    proxies=proxies
    timeout=10
)

print(response.json())