DEV Community: Nate

Cloudflare's AI block names eight crawlers. None is ChatGPT's search bot

Nate — Tue, 18 Aug 2026 15:35:36 +0000

Eight user agents, and the one that decides whether ChatGPT cites you is not among them.

An r/SEO post from April, 53 points and 40 comments, says Cloudflare quietly cut the author's site off from ChatGPT, from Perplexity and from Google's AI Overviews. I went and read the block. It names ChatGPT's training crawler and not its search crawler, and it never mentions Perplexity at all. The Google agent it does name is one Google says does not affect inclusion or ranking in Search.

Take just two of the eight agents. GPTBot governs inclusion in OpenAI's training data, and Google-Extended governs grounding in Gemini Apps. Those are real things to give up. They are not the three things the warning names.

What is actually in the file

Cloudflare prints the whole block on its managed robots.txt page:

# BEGIN Cloudflare Managed content
User-Agent: *
Content-signal: search=yes, ai-train=no, use=reference
Allow: /

User-agent: Amazonbot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: meta-externalagent
Disallow: /
# END Cloudflare Managed Content

Read that against Cloudflare's own crawler reference table and a pattern falls out. GPTBot is in, OAI-SearchBot is out. ClaudeBot is in, Claude-SearchBot is out. For both of those pairs the table calls the blocked agent an AI Crawler and the one left alone AI Search. Applebot-Extended is in and plain Applebot is not, the same split again. The table has no Applebot-Extended row, so it cannot tell you what Cloudflare calls that one. The block runs along the training and search seam, and that looks deliberate to me.

OpenAI's side of it is one line: "Each setting is independent of the others". I worked through the three OpenAI bots and which one governs search visibility in an earlier post, so I will point there rather than do it twice.

Google-Extended is the entry that causes the trouble, because it sounds like it governs AI answers in Search. Google's crawler documentation scopes it to training and grounding future Gemini models, then says flatly: "Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search."

So what does control AI Overviews? Nothing, in the sense people mean. From Google's AI features page: "AI is built into Search and integral to how Search functions, which is why robots.txt directives for Googlebot is the control for site owners to manage access to how their sites are crawled for Search. To limit the information shown from your pages in Search, use nosnippet, data-nosnippet, max-snippet, or noindex controls." Those are the ordinary snippet controls, so opting out of the AI answer means opting out of the snippet. Google states the trade in one paragraph and almost nobody repeats it.

One more from the crawler page, which kills a piece of advice I keep seeing: "Google-Extended doesn't have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings..." Anyone telling you to check your access logs for Google-Extended is sending you after a string that does not exist.

Did anyone turn this on, or did it turn itself on

Cloudflare's documentation describes the managed file as a switch, in the same grammatical mood every time. "When you turn on the managed robots.txt setting, Cloudflare generates and maintains a robots.txt file that instructs known AI crawlers to stay away from your content." I searched the docs, the changelog, four announcement posts and the July 2025 press release for a case where it got enabled on an existing domain without the owner acting. I did not find one. That is a failure to find rather than a proof of absence.

What did change is adjacent and gets mistaken for it. The press release from 1 July 2025 says "every new domain will now be asked if they want to allow AI crawlers..." A question at onboarding, for new domains, governing the traffic block rather than the file.

One Cloudflare default does appear in a file you never touched, and it is not a block: "Domains on the Free plan that do not have their own robots.txt file and do not use the managed robots.txt feature will display the Content Signals Policy when a crawler requests the robots.txt file for your domain." Cloudflare adds that the policy "does not express any specific preferences about your content".

Check it over the network, not on disk

The managed file is assembled at the edge. Cloudflare says it "will prepend our managed robots.txt before your existing robots.txt, combining both into a single response", so the file in your web root and the file the world receives are two different objects. Your hosting file manager, FTP and your SEO plugin's robots editor can none of them see the difference, because none of them sits downstream of the edge.

curl -s https://example.com/robots.txt

Opening the same URL in a browser tab does the same job.

If the block is on, you get the # BEGIN Cloudflare Managed content marker above whatever your own site serves, the eight Disallow groups, then # END Cloudflare Managed Content. Note the capitalisation: lowercase c on the opening marker, capital C on the closing one. Either marker is conclusive on its own, and so is User-agent: GPTBot sitting in a file you never edited.

The false positive to know about is the Content Signals preamble. It runs to a screenful, opens with "As a condition of accessing this website, you agree to abide by the following content signals:", and defines a vocabulary in lettered clauses. Every line of it starts with a #, and a robots.txt comment does nothing. So: if every line starts with a hash, nothing is blocked. A block is a Disallow: line with no hash in front of it.

If you find AI crawler rules and no Cloudflare markers, the source is inside your own stack. WordPress core's complete default output, from the do_robots() source, is three lines:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Core adds no AI rules of any kind. The function ends by passing that output through a robots_txt filter, which is the hook a plugin would use to append its own. So the rules came from a physical file in the web root, a plugin on that filter, an SEO plugin's own robots editor, or the edge. Those are the only four places I can find that they could come from, so checking in that order settles it.

If you do change something, OpenAI is the only operator I found that publishes a lag: "it can take ~24 hours from a site's robots.txt update for our systems to adjust." Nobody else gives a figure, so do not read that one as the industry's.

The date is 15 September 2026, and it is a different toggle again

Most of the argument is about a default Cloudflare changed in 2025. There is a second change dated 15 September 2026, and that one has an action attached. From the Block AI bots documentation:

"On September 15, 2026, Cloudflare will set updated defaults for new domains: bots classified as Training or as Agent will be blocked on pages that display ads, and Search will remain allowed. Mixed-purpose crawlers that combine Search and Training will also be blocked by all configurations to block AI training, including the legacy 'Block AI bots' option. Before September 15, all customers can opt out of these new defaults."

Two things happen there and they need keeping apart. The ads-page default is for new domains only, so if your site is already on Cloudflare that half does not reach you at all.

The other half does. From Cloudflare's announcement on 1 July 2026: "multi-purpose crawlers such as Googlebot, Applebot, and BingBot will be blocked by customers who have selected to block Training (either through the new options to manage AI traffic, or through the legacy Block AI bots service)." The legacy service there is the "AI Scrapers and Crawlers" toggle from July 2024.

My reading, and Cloudflare does not put it in these words: if you clicked that toggle in 2024 and do nothing before 15 September 2026, you are opting into blocking a crawler Cloudflare itself names as Googlebot. I am not going to model what that does to any particular site. The opt-out sits in Security Settings and it closes on 15 September 2026.

The category is not marginal either. Cloudflare Radar's AI insights page splits crawl traffic by declared purpose, and for the seven days to 18 August 2026 it put crawlers declaring both training and search at 33.3%, training alone at 40.2% and search alone at 18.5%.

Bot Fight Mode is not the same feature

In the same thread, somebody undid all of Cloudflare's bot blocking and found Claude, Grok and Perplexity still blocked. Another commenter named Bot Fight Mode as the cause, on the grounds that those three are not verified bots. The mechanism is right. The list is not.

Bot Fight Mode is a separate product with a separate toggle. It does not produce a file, it issues challenges, and it is rigid by design: "You cannot bypass or skip Bot Fight Mode using WAF custom rules or Page Rules. This is because Bot Fight Mode does not run on the Ruleset Engine..." Cloudflare's Free-plan table gives the type of bots it detects as "Simple bots from cloud hosting providers and headless browsers" and the control as "Applied to all traffic across a domain". A crawler on cloud IPs with no exemption and no rule you can write around it is the shape of a block that survives every AI setting you turn off. That is my reading of four documented properties, not a sentence Cloudflare writes.

The list is where it comes apart. I went through all 693 entries in Cloudflare's own bots directory on 18 August 2026. ClaudeBot is in there, labelled Verified, operator Anthropic, and so is Claude-SearchBot. Perplexity has no entry, and Cloudflare has published why: on 4 August 2025 it wrote that it had "de-listed them as a verified bot and added heuristics to our managed rules that block this stealth crawling". Perplexity disputed that publicly and I am not going to adjudicate it. Grok and xAI have no entry, and I found nothing Cloudflare has published about them either way. Absence carries no single meaning: Bytespider and CCBot are missing too, and both sit on Cloudflare's own block list.

One thing I would like to state and cannot. Cloudflare's verified bots page says "Historically, Verified bots have been excluded in default bot configurations across all plans." The Bot Fight Mode page says nothing about verified bots either way, and I read all of it. Configurable allowlisting is a Super Bot Fight Mode feature, Pro and above.

The Content-signal line does not do what its wording implies

Content-signal: search=yes, ai-train=no, use=reference is the first non-comment line of the managed block, and it is the part people read as the actual instruction to AI companies.

It is Cloudflare's convention. contentsignals.org is Cloudflare's own site, calls itself "Cloudflare's implementation", and now sits alongside the IETF's AI Preferences working group. That group is real and chartered, neither of its two standards-track documents is a published RFC, and its charter puts "Technical enforcement of preferences" out of scope in writing.

The vocabulary has a sharp edge in it. Cloudflare's policy text defines search as "building a search index and providing search results" and then adds: "Search does not include providing AI-generated search summaries." So search=yes is not permission to be summarised in an AI answer. The signal for that is ai-input, the managed block omits it entirely, and under the policy's own terms omitting a signal expresses no preference either way. A quick reader gets this backwards.

use=reference is newer. Cloudflare calls content-use a test, and added the field to existing customers' files on 1 July 2026, so anyone who turned the setting on before that got a fourth field without doing anything.

Whether anybody acts on any of it, I could not establish. OpenAI's crawler documentation does not mention content signals. Google is worse than silent: its robots.txt specification publishes a closed list, "Google supports the following fields (other fields such as crawl-delay aren't supported)", and that list is user-agent, allow, disallow and sitemap. The same page says "Google ignores invalid lines in robots.txt files". Anthropic, Perplexity, Meta, Apple, Amazon and ByteDance say nothing either way, and Cloudflare makes no adoption claim of its own.

Nobody else is doing this to you quietly

I went looking for a second host with a silent AI default and did not find one. WP Engine documents two and neither is AI related: a block on sites still running on a .wpengine.com subdomain, and a platform-wide crawl delay of 10 seconds nobody asked for. Pressable's live-site default allows crawling. For Kinsta, SiteGround and cPanel I found no primary documentation either way.

The one that surprised me sits lower down. hosting.com documents that on shared and reseller servers, "if a robots.txt file does not exist in the document root (public_html) directory, the server automatically creates a new robots.txt file at midnight". The article never says what goes in that file, so I cannot tell you whether it mentions AI crawlers.

So should you block them

I do not have an answer for you, and I want to be exact about why.

Nothing I read measures the outcome. No study in any of it says what blocking training crawlers does to your citations or your traffic. The argument on both sides is people reasoning from first principles about a system nobody has measured. Fine as reasoning, not evidence.

What I can tell you is narrower. The seam in that block list runs between training and search rather than across AI as a category, so what you give up by leaving it on is not what the loudest warning says you give up. And if you clicked an AI toggle in 2024 and have not opened the screen since, Cloudflare has put a date on it in writing. The opt-out window closes on 15 September 2026, and the screen it closes on is Cloudflare's Security Settings.

Amazon PA-API v5 deprecates April 30, 2026 and retires May 15. Here is what changes at the auth layer.

Nate — Tue, 14 Apr 2026 10:12:10 +0000

If you maintain anything that calls Amazon's Product Advertising API v5 (PA-API), this affects you. Amazon's published timeline has two dates: April 30, 2026 is the deprecation date and Amazon's recommended migration cutoff, and May 15, 2026 is when the endpoint itself is retired. Amazon explicitly recommends completing migration by April 30 to avoid service disruption. The replacement is the Creators API, and the migration is not a config change. It is a different authentication scheme, a different request shape, and a different token lifecycle.

This post is a quick technical orientation: what changes at the auth layer, what changes in the request payload, and how to test your migration before the cutoff. Useful whether you are maintaining a WordPress plugin, a custom Node integration, or a one-off affiliate scraper.

What changes at the auth layer

PA-API v5 uses AWS Signature Version 4. You sign every request with your AWS Access Key ID and Secret Access Key, derive a signing key from your secret + date + region + service, and attach an Authorization header that looks like this:

Authorization: AWS4-HMAC-SHA256 Credential=AKIA.../20260413/us-east-1/ProductAdvertisingAPI/aws4_request,
               SignedHeaders=content-encoding;content-type;host;x-amz-date;x-amz-target,
               Signature=...

You also send X-Amz-Date, X-Amz-Target, and Content-Encoding: amz-1.0. The signing process is finicky: every header has to be canonicalized, the body has to be hashed, and the timestamp window is tight (15 minutes). If you have ever debugged Sig V4, you know.

The Creators API replaces all of that with OAuth 2.0 client credentials. You register your application in the Amazon Associates portal, get a Client ID and Client Secret, and exchange them for a bearer access token:

# The URL depends on your credential Version (Amazon shows this on your credential page).

# v2.x credentials (Cognito-fronted OAuth):
POST https://creatorsapi.auth.us-east-1.amazoncognito.com/oauth2/token   # v2.1 (NA)
POST https://creatorsapi.auth.eu-south-2.amazoncognito.com/oauth2/token  # v2.2 (EU)
POST https://creatorsapi.auth.us-west-2.amazoncognito.com/oauth2/token   # v2.3 (FE)

# v3.x credentials (Login with Amazon path):
POST https://api.amazon.com/auth/o2/token      # v3.1 (NA)
POST https://api.amazon.co.uk/auth/o2/token    # v3.2 (EU)
POST https://api.amazon.co.jp/auth/o2/token    # v3.3 (FE)

# Common headers and body for v2.x (Cognito):
Content-Type: application/x-www-form-urlencoded
Authorization: Basic <base64(client_id:client_secret)>

grant_type=client_credentials&scope=creatorsapi/default

Amazon issues credentials with a Version (2.1, 2.2, 2.3 for the Cognito path; 3.1, 3.2, 3.3 for the LwA path). The token endpoint, scope format, and Authorization header shape all depend on which Version you hold. For v3.x credentials, the scope is creatorsapi::default (double colon) and the product Authorization header is Bearer <access_token> without the Version suffix.

Amazon documents both forms as valid for v2.x credentials: you can send client_id and client_secret in the body, or use the Authorization: Basic <base64(id:secret)> header shown above. The Basic header form is what Amazon shows in the migration guide example, so that is the one I would default to.

You get back a JSON envelope:

{
  "access_token": "<opaque-string>",
  "token_type": "bearer",
  "expires_in": <seconds>
}

For v2.x credentials, every product request carries Authorization: Bearer <access_token>, Version <credential_version> plus an x-marketplace header naming your storefront (for example www.amazon.com). The Version suffix is how Amazon ties the token to your Associates credential version - do not drop it. For v3.x credentials (Login with Amazon path), the header is just Authorization: Bearer <access_token> without the Version suffix.

Practical implications:

No more derived signing keys, no more canonical request strings, no more 15-minute timestamp window.
The token is opaque to you. You do not roll your own crypto.
You do need a token cache layer. In WordPress, any shared persistent store works: set_transient with a TTL under expires_in, or update_option with a companion expiry timestamp that your code checks before each request. The must-have is "shared across PHP-FPM workers," not "auto-expiring" - you will check expiry yourself before every request anyway.
Pre-emptive refresh works well here: keep an expires_at timestamp next to your cached token and refresh before you hit it (a 60-second buffer is plenty). Refresh-on-401 also works as a fallback if a request races with expiry, but pre-emptive refresh keeps the common path to a single HTTP call.

What changes in the request payload

The request body shape stays largely the same, but every parameter name has been converted from PascalCase to lowerCamelCase. SearchItems now takes keywords, searchIndex, itemCount, resources[]. GetItems takes itemIds[]. Both also take partnerTag (lowercase p) and marketplace. If you are porting existing PA-API v5 code, a find-and-replace across parameter names is mandatory - Amazon will reject the PascalCase forms. The big differences beyond the casing:

Endpoint host changes. PA-API v5 is webservices.amazon.com (regional). Creators API uses a single global endpoint: https://creatorsapi.amazon/catalog/v1/. Unlike PA-API v5 (which had regional hosts like webservices.amazon.com baked into Sig V4 signing), Creators API is region-agnostic at the host level - regional routing is driven by your x-marketplace header and credential Version. Note the .amazon TLD - that is Amazon's own top-level domain, not a typo.
No regional signing. Sig V4 had us-east-1, eu-west-1, etc. baked into the signing key. The Creators API decouples auth from marketplace - you specify the marketplace in the request body and the x-marketplace header, not in the auth itself.
Per-request item caps you will hit. SearchItems returns up to 10 items per call (itemCount max 10). GetItems accepts up to 10 itemIds per call. If you were batching more than 10 ASINs per GetItems request on PA-API v5, you will need to split. The resources[] field-selection array length is not numerically capped in Amazon's public docs - add what you need.
Throttling behavior may shift. Check the Creators API rate-limit docs before assuming PA-API v5 limits carry over one-for-one.

How to test your migration before the cutoff

Three steps that do not require burning your production setup:

Apply for Creators API credentials now. Prerequisites you may hit: you need final-accepted Amazon Associates status, and Amazon requires at least 10 qualifying referred sales in the last 30 days to access Creators API. Only the primary account owner can register. If you clear those gates, go to Associates Central -> Tools -> Creators API, apply, and wait for approval. This unblocks everything else.
Run both auth methods side-by-side on a staging site. Keep your existing PA-API v5 wiring intact. Add a feature flag or a config toggle that routes a copy of each search to the Creators API endpoint with the bearer token. Compare the response shapes for the same query. You will catch any payload differences (mostly minor field naming) without touching production.
Migrate one site, then the rest. Pick your lowest-traffic affiliate site, flip it to Creators API, run for a week, watch error rates and CTR. If clean, roll the rest. Do not flip everything on April 29.

Common gotchas

A few things that catch people:

Token caching across processes. If you are running PHP-FPM with multiple workers, each worker can request its own token if you cache in process memory. Use a shared cache (WordPress options, transients, Redis, memcached) instead.
Clock skew on the staging server. OAuth tokens are timestamped. If your server clock is off by more than a few minutes, token validation can fail. ntpdate is your friend.
Marketplace defaults. Make sure your migration code passes the same marketplace as your old code, both in the request body AND the x-marketplace header. Default endpoint behavior is not always identical.
Throttling under load. The new API may enforce different rate-limit shapes. If you were already cache-walking PA-API v5 (search 30 min, product 1 hr is a common pattern), keep that behavior - the cache layer insulates you from whatever the new limits turn out to be.

After April 30

April 30 is Amazon's recommended migration deadline and the official deprecation date - documentation stops being maintained and Amazon's guidance is to be off v5 by then. The endpoint itself keeps accepting requests until it retires on May 15, so you have a two-week tail, but there is no documented grace period on the retirement itself. Plugins and integrations that still send AWS Sig V4 requests after May 15 will start getting auth failures. The path forward is straightforward, but the window is fixed and the application-and-test cycle takes more than zero days. (These dates come from Amazon's April 2026 notification to existing PA-API customers. Amazon's public Creators API migration guide does not currently publish an endpoint retirement date, so if a reader checks that and finds nothing, this is why.)

If you do nothing else this week: apply for Creators API credentials. Everything downstream depends on having them in hand.

Footnote: I maintain Product Search for Amazon, a WordPress plugin that supports both PA-API v5 and the Creators API during the migration window. If you are running WordPress and want a reference implementation of the dual-auth pattern, the source is on the WordPress.org plugin directory. Not the point of this post - just disclosing relevant context.