DEV Community: Mahiro Hirakawa

日本のサイト248件中23件、robots.txtは許可でもAIクローラーには403

Mahiro Hirakawa — Wed, 07 Oct 2026 00:20:09 +0000

日本のサイトで、robots.txtはAIクローラーを許可しているのに、サーバーの手前(CDNやWAF)ではそのクローラーに403を返している、という食い違いがどれくらいあるのかを数えました。2026-10-07の朝、Trancoのリストに載っている日本のサイト300件を1件ずつ確認したところ、ブラウザとして普通にページが返ってきた248件のうち23件で、robots.txtでは許可されているAIクローラーのUser-Agentに403が返りました。

この記事は、その数字と内訳、層ごとの例、そしてこの測り方でわからないことです。

先に限界(2026-10-07時点)

User-Agentを名乗っただけです。 送ったのは各社が公開しているUser-Agentの文字列で、送り元は私の普通の回線です。本物のクローラーは各社のIPアドレスから来ます。CDNのボット対策には、IPで本物と確認できたクローラーと、名前を真似ただけのリクエストを別に扱うものがあります。ここでの403は「このUser-Agentを名乗ると断られる」という意味で、「本物のGPTBotが断られている」の証明ではありません。
場所は1か所です。 日本の家庭用回線1本から送りました。国や回線によって応答を変えるサイトもあります。
1回のスナップショットです。 2026-10-07 08:19〜09:12(日本時間)に1回ずつ取得し、403になった23件だけを09:13〜09:15にもう一度確認しました。設定は日々変わります。
トップページだけです。 下層ページやAPIは見ていません。
意図はわかりません。 学習用クローラーを止めるのは正当な選択です。ここで数えているのは、robots.txtとサーバーの手前の層が別のことを言っている件数で、どちらが意図どおりかはサイトの方にしかわかりません。

対象と測り方

リスト: Trancoのリスト56WKN(2026-10-05生成、100万件)。これまでの記事と同じリストです。
対象300件: リストの中の.jpドメインを順位の高いほうから288件(最下位は15,883位)。それに、日本向けのサイトを.comで運営していると確認できた12件(nikkei.com、asahi.com、sankei.com、jiji.com、kakaku.com、tabelog.com、cookpad.com、mercari.com、dmm.com、fc2.com、livedoor.com、yodobashi.com)。日本企業でも海外向けの.comは入れていません。
リクエスト: 1件ずつ順番に、間を0.3秒あけ、タイムアウト10秒、リダイレクトは追いました。robots.txtは自分の名前を名乗るUser-Agent(glovrex-measure/1.0)で取得。トップページは、ブラウザ(Chrome)→ GPTBot → OAI-SearchBot → ClaudeBot → PerplexityBot → もう一度ブラウザ、の順に6回取得しました。apex(example.jp)が応答しなかった97件はwww.付きで取り直しています。
Google-Extendedはrobots.txtの中の名前だけで、それを名乗るクローラーは存在しません(取得はGooglebotが行います)。なのでrobots.txtの集計にだけ入れ、サーバーの応答は測っていません。
「サーバーの手前で断る」の定義: robots.txtがそのクローラーにトップページ(/)を許可している、ブラウザの2回がどちらも200、そのクローラーのUser-Agentには401か403。robots.txtが404などの4xxのときは、RFC 9309にしたがって「制限なし」として扱いました。
判定に使ったスクリプトは依存なしの.mjs1本で、robots.txtの判定はRFC 9309の最長一致です。

数字

300件のうち、ブラウザの2回がどちらも200だったのは248件です。残り52件は、どちらの名前でも応答がない、ブラウザにもチャレンジページ(202など)や403を返す、といったサイトで、比較ができないので外しています。

robots.txtの取れ方は次のとおりです。

robots.txt	件数
テキストのファイルが返った	183
200でHTMLページが返った	29
4xx(ファイルなし扱い)	47
応答なし	40
5xx	1

robots.txtでトップページを拒否している件数(読めた259件中)は、GPTBot 25、OAI-SearchBot 17、ClaudeBot 21、PerplexityBot 18、Google-Extended 24でした。

そして本題です。robots.txtでは許可されているのに、そのUser-Agentに403が返ったサイトは248件中23件でした。クローラーごとの件数はこうです。

クローラー	robots.txtは許可、サーバーは403
GPTBot	22
ClaudeBot	19
PerplexityBot	16
OAI-SearchBot	15

23件のうち14件は、4つのUser-Agentすべてに403を返しました。23件は09:13〜09:15の再確認でも、最初に403だったUser-Agentにすべて403のままでした。

23件のrobots.txtの状態

「robots.txtが許可」の中身は均一ではありません。

23件のrobots.txt	件数
ファイルがあり、そのクローラーを許可している	13
robots.txt自体が私の計測用User-Agentに403	7
robots.txtが404	2
robots.txtの場所でHTMLが返った	1

13件は、robots.txtのファイルに「来ていい」と読める内容が書かれていて、その手前の層が断っている、いちばんはっきりした食い違いです。7件は、robots.txt自体が名乗った計測用のUser-Agentに403でした。規格上は「制限なし」扱いですが、実際にはボットらしいリクエスト全般を手前で止めている可能性が高い形です。

断っている対象で3つに分かれる

23件に、AIではないUser-Agentを3つ(名乗った計測用、curl、Googlebotの文字列)追加で送りました。

形	件数	何が言えるか
A. 計測用もGooglebotの文字列も200、AIのUser-Agentだけ403	9	AIクローラーの名前を指定したルールがある
B. 計測用は200、Googlebotの文字列も403	8	有名なボットの名前を、確認できないIPから名乗るリクエストを断っている可能性がある。この場合、本物のクローラーは通っているかもしれない
C. 計測用のUser-Agentも403	6	ブラウザ以外のアクセス全般を断っている

Aの9件では、403だったのはGPTBotが9件、ClaudeBotが6件、PerplexityBotとOAI-SearchBotが2件ずつでした。学習用のクローラー(GPTBot、ClaudeBot)を名指しし、検索用のOAI-SearchBotは通している形が多いということです。robots.txtで学習用だけ止めたい場合とやりたいことは同じですが、robots.txtには書かれていません。

ドメインの種類別

種類	普通に読めた件数	うち食い違い
`.co.jp`	70	9
`.jp`(汎用)	130	9
`.go.jp`	17	4
`.ne.jp`	9	1
日本向けの`.com` 12件	10	0
`.ac.jp`、`.or.jp`、`.ad.jp`	12	0

レスポンスヘッダーから見たサーバーの手前の層は、Akamaiが8件、CloudFrontが6件、Cloudflareが1件、ヘッダーからは判別できないもの(nginxやAWSのロードバランサーなど)が8件でした。特定のCDNに偏った現象ではありません。

分野ごとの例

以下は測定時点で観測した応答の記録で、設定の良し悪しの評価ではありません。どれも意図した設定である可能性があります。

交通・旅行: jorudan.co.jpは、robots.txtのファイルがあり、GPTBot、ClaudeBot、PerplexityBotに403、OAI-SearchBotには200でした(形A)。jtb.co.jpはGPTBotだけが403で、ほかの3つは200(形A)。jreast.co.jpは、robots.txtも計測用のUser-Agentに403で、ブラウザ以外を広く断っている形(形C)。
メディア・出版: yanmaga.jpはGPTBotだけが403(形A)。trafficnews.jpはGPTBotとClaudeBotが403で、検索用の2つは200(形A、Cloudflare)。kabutan.jpは4つとも403、Googlebotの文字列は200(形A)。
EC・小売: creema.jpとtower.jpは、GPTBotとClaudeBotが403、OAI-SearchBotとPerplexityBotが200(形A)。melonbooks.co.jpはrobots.txtが404で、PerplexityBot以外の3つが403(形A)。buyee.jpは4つとも403で、Googlebotの文字列も403(形B)。
行政: moj.go.jpはGPTBotだけが403(形A)。e-gov.go.jpとmeti.go.jpは4つとも403で、Googlebotの文字列も403(形B)。mofa.go.jpは計測用のUser-Agentも403(形C)。
金融ほか大手企業: mizuhobank.co.jpとsuntory.co.jpは、どちらも計測用のUser-Agentも403(形C、Akamai)。smbcnikko.co.jpは4つとも403で、Googlebotの文字列も403(形B)。

自分のサイトを確かめる

bashで次のように確かめられます(WindowsならGit BashかWSL)。URLをご自身のサイトに置き換えてください。

URL=https://example.com/
while IFS='|' read -r NAME UA; do
  curl -s -o /dev/null -L -w "%{http_code}  $NAME\n" -A "$UA" "$URL"
done <<'EOF'
browser|Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0 Safari/537.36
GPTBot|Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.1; +https://openai.com/gptbot)
OAI-SearchBot|Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot
ClaudeBot|Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)
PerplexityBot|Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)
EOF
curl -s "${URL}robots.txt" | head -40

ブラウザが200でボットが403なら、robots.txtの内容と見比べてください。食い違っていれば、勝つのはサーバーの手前の層です。ただし上の形Bのように、CDNが本物のクローラーをIPで見分けている場合は、curlの結果と本物の扱いが違うことがあります。いちばん確かなのはサーバーやCDNのアクセスログで、GPTBotやClaudeBotで絞り込んでステータスを見てください。層ごとの見分け方は以前の記事に書きました。

403が返っていないことは、AIの回答にサイトが出ることの約束にはなりません。この確認でわかるのは、読みに来たクローラーが入口で止まっていないかどうかだけです。

1サイトを人の手で見てほしい場合は、GEO Mini Auditで、この確認を含むミニ診断を、そのまま貼れる修正案付きでお渡ししています(AIエンジンへの問い合わせや、引用・順位の測定はしません)。

15 merged IME fixes: the recurring ways Japanese typing breaks web apps

Mahiro Hirakawa — Tue, 06 Oct 2026 22:32:23 +0000

Between late June and early July 2026 I had 15 pull requests merged into open-source projects, all fixing the same family of bug: a keyboard handler that does not know an input method editor (IME) is in the middle of composing text. This post groups them into the bug classes that keep recurring, with a minimal repro and the fix pattern for each. Every fix links to the merged PR so you can read the real diff.

Background in one paragraph

Japanese, Chinese and Korean are typed through an IME. In Japanese you type the reading (toukyou), the IME shows candidates (東京), and you press Enter to confirm the candidate. That Enter is a real keydown event. While composition is active, KeyboardEvent.isComposing is true (MDN), and keydowns handled by the IME carry the special keyCode 229 (MDN keydown, which shows the guard if (event.isComposing || event.keyCode === 229) return;). Code that reads only event.key === "Enter" cannot tell "confirm this kanji" from "send this message".

Class 1: Enter sends the message mid-composition

What the user sees: they type a reading, press Enter to pick the kanji, and the chat sends half a sentence in romaji or unconverted kana.

Minimal repro:

textarea.addEventListener("keydown", (e) => {
  if (e.key === "Enter" && !e.shiftKey) { e.preventDefault(); send(); }
});

Fix pattern:

textarea.addEventListener("keydown", (e) => {
  if (e.isComposing || e.keyCode === 229) return;
  if (e.key === "Enter" && !e.shiftKey) { e.preventDefault(); send(); }
});

Merged evidence:

CopilotKit/CopilotKit #5764: Angular chat input submitted during composition.
misskey-dev/misskey #17646: chat sent while the IME was composing; the guard also checks e.key === "Process".
twentyhq/twenty #22270: chat-thread input and attachment rename input.
enricoros/big-AGI #1145: custom instruction field in the Diagrams view.

Class 2: Enter commits a partial value (tags, list items, renames, validated fields)

What the user sees: a tag chip, list item, card title or file name is saved as the unconverted reading, and the actual kanji they picked ends up as a second stray entry, or is lost. In a validated numeric or formatted field, the half-typed value is committed and validated.

Minimal repro:

input.addEventListener("keydown", (e) => {
  if (e.key === "Enter" && input.value.trim()) addTag(input.value);
});

Fix pattern: the same guard, placed before any commit or validation runs.

input.addEventListener("keydown", (e) => {
  if (e.isComposing || e.keyCode === 229) return;
  if (e.key === "Enter" && input.value.trim()) addTag(input.value);
});

In React, several of these PRs (onyx, payload, rsuite, LibreChat) read e.nativeEvent.isComposing, because the handler receives React's synthetic event rather than the DOM KeyboardEvent.

Merged evidence:

tusen-ai/naive-ui #8115: n-dynamic-tags committed a tag on the confirm Enter; the PR includes a test that triggers compositionstart and then Enter.
onyx-dot-app/onyx #12511: ListFieldInput added an item on the confirm Enter.
LibreChat-AI/LibreChat #13996: prompt name, labels and dynamic tag inputs.
TriliumNext/Trilium #10315: board card and column titles.
siyuan-note/siyuan #17988: agent session rename input.
chakra-ui/zag #3198: the color picker channel input committed a partial value on the confirm Enter.

Class 3: Enter selects or searches on unconfirmed text

What the user sees: in an autocomplete, picker or search box, the confirm Enter picks whatever option is highlighted, or fires the search with the reading instead of the converted word.

Minimal repro:

combo.addEventListener("keydown", (e) => {
  if (e.key === "Enter") selectHighlighted();
});

Fix pattern: guard first, then select. In component libraries the guard belongs in the shared keydown handler so every consumer gets it.

Merged evidence:

payloadcms/payload #17138: SearchInput searched on the confirm Enter.
rsuite/rsuite #4585: InputPicker selected an item; the PR adds a test that onChange is not called for Enter with isComposing: true.
vuetifyjs/vuetify #22974: VAutocomplete keydown listener ran during composition.

Related, no PR of mine behind it: search-as-you-type that fires on every input event also queries the unconfirmed reading while the user is still choosing candidates. InputEvent.isComposing is true for those events (MDN), so a common pattern is to skip them and run the query from compositionend instead. Whether that is what your users want is a product choice; I have not tested this pattern across browsers for this post.

Class 4: keyCode 229 where `isComposing` is false

What the user sees: the fix from Class 1 works in Chrome, but the bug is still there for some Safari users.

Why: the plate PR below documents the Safari plus Japanese IME case where the confirm Enter arrives with keyCode === 229 and isComposing === false. A guard that reads only isComposing lets it through. I did not reproduce this in Safari myself; the claim rests on that PR and on MDN's recommended guard checking both.

Fix pattern: always check both, e.isComposing || e.keyCode === 229. keyCode is deprecated, but this is the one place it still carries information.

Merged evidence:

udecode/plate #5048: the floating link editor submitted on the Safari confirm Enter.
misskey-dev/misskey #17646 (also in Class 1) checks isComposing, key === "Process" and keyCode === 229.

Class 5: composition flags that depend on event order

What the user sees: the same as Class 1 or 2, but only in one browser.

Minimal repro:

let composing = false;
input.addEventListener("compositionstart", () => (composing = true));
input.addEventListener("compositionend", () => (composing = false));
input.addEventListener("keydown", (e) => {
  if (!composing && e.key === "Enter") submit();
});

Why it breaks: the flag is only correct if the confirm keydown arrives before compositionend. The UI Events spec (section 3.6.5, "Key Events During Composition") says keydown events sent during the composition session must have isComposing set to true. The plate PR in Class 4 shows Safari delivering the confirm Enter with isComposing false, which is consistent with that keydown landing outside the session, after compositionend has already cleared your flag. I have not measured the event order in Safari or any other browser for this post; treat browser-specific ordering as untested.

Fix pattern: read the state off the event itself (e.isComposing || e.keyCode === 229) instead of, or in addition to, a flag you maintain. Where a library keeps an instance flag, as naive-ui #8115 does with its input's composition state, add a test that fires Enter during an active composition.

A bonus class: Escape during composition

Pressing Escape to cancel the IME candidate list also fires keydown with isComposing: true. If a modal or drawer closes on Escape, it closes while the user only meant to cancel the candidates.

Tencent/tdesign-vue-next #6756: the drawer closed on the composition-cancel Escape; the fix is if (e.key === "Escape" && e.isComposing) return; plus a test.

Test your own app in 5 minutes

You do not need to read Japanese. Add the Japanese IME in your OS settings (Windows: Settings, Time & language, Language, add Japanese; macOS: System Settings, Keyboard, Input Sources, add Japanese - Romaji).

Switch to Japanese input and focus your chat box. Type toukyou, press Space to see candidates, press Enter once to confirm. Expected: 東京 stays in the box and nothing is sent. Press Enter again; now it sends.
Repeat in every field where Enter does something: tag inputs, list editors, rename fields, inline title editing, search boxes, autocompletes, comboboxes.
With candidates open, press Escape. Expected: the candidate list closes and your modal or drawer stays open.
In a search-as-you-type box, type a reading slowly and watch the network tab. Decide whether queries for the unconverted reading are acceptable.
Grep your code for key === "Enter", key === 'Enter', keyCode === 13 and "Escape", and check each handler has isComposing || keyCode === 229 before it acts. If your handlers set flags in compositionstart and compositionend, check them against step 1 in every browser you support.

If you can only test on Windows, steps 1 to 4 do not cover the Safari cases in Classes 4 and 5; step 5 does, by reading the code.

This article and the fixes it links were drafted with Claude Code and reviewed by the author.

If you would rather have someone else run this check on your app, I offer it at ime-check.vercel.app.

154 of 529 top homepages have no canonical tag. 15 point to a URL that redirects straight back.

Mahiro Hirakawa — Tue, 06 Oct 2026 22:06:00 +0000

I fetched the homepages of the Tranco top 1,000 once each and read two tags that tell crawlers which URL a page is: <link rel="canonical"> and the hreflang alternates. 529 homepages came back as real pages. 154 of the 529 have no canonical tag at all. Of the 375 that do, 317 point at the URL they were served from and 58 point somewhere else. I then fetched those 58 targets: 20 redirect, 15 of them straight back to the page that named them, one is a 404 and one is a 403.

The hreflang side is in better shape than its reputation. 197 homepages declare language versions. I fetched up to three alternates for each and checked whether they link back. 479 of the 522 alternates I fetched do; 15 of the rest didn't answer 200.

Below: the limits first, the numbers, the mistakes that repeat, and a short checklist.

Limits first (measured 2026-10-06/07)

One fetch per domain, from Japan. Homepages on 2026-10-06 between 21:30 and 21:54 UTC, then the canonical targets and alternates between 21:55 and 22:03 UTC, one request at a time, 10 second timeout, redirects followed, no JavaScript. Several sites serve a Japanese page to a Japanese IP, and their canonical changes with it. From another country you'd see other URLs for some of these.
A bot user agent. I identified myself (glovrex-measure/1.0). 96 domains answered 403 to that, which is fewer pages than my earlier runs with a browser user agent. Those sites are unknown here, not "no canonical".
List and set: Tranco list 56WKN, top 1,000. Many entries are CDN, API or tracking domains with no homepage: 314 gave no HTTP answer (DNS, TLS, timeout), 148 answered with something other than 200, 9 answered 200 with no HTML head or an "Access Denied" page. n = 529 below.
Homepage only, <head> only. I read canonical and hreflang from the HTML head and from the HTTP Link header. hreflang in an XML sitemap isn't covered, so a site can be fine there and still show up as "no hreflang" here.
Return links are sampled. At most three alternates per site, skipping x-default and the page itself.
I don't know how AI crawlers use these tags. Google and Bing document canonical and hreflang. OpenAI, Anthropic and Perplexity don't say whether their crawlers read them. Answer engines that build on a search index inherit whatever the index decided.

The canonical numbers

n = 529 homepages.

	Homepages
No canonical (head or `Link` header)	154
Canonical = the URL served	317
Canonical points elsewhere	58
More than one canonical tag	4
Two canonicals that disagree	1
Relative canonical (`/`, `/us/`)	2
`og:url` present	291
`og:url` disagrees with canonical	19

The 58 that point elsewhere, by what differs:

Difference	Homepages
Only the query string	18
A different host	15
A different path	14
Only a trailing slash	8
Only `www.`	3

The query-string group is mostly fine. bing.com, ya.ru and youtu.be redirect with a tracking or redirect parameter and the canonical drops it, which is what a canonical is for.

What the 58 targets answered

Canonical target
200, and declares itself canonical	32
Redirects to another URL	20
No answer	2
404	1
403	1
200, no canonical of its own	1
200, canonical points on again	1

The 20 redirects are the interesting group. 15 of them redirect to exactly the URL that declared them, 3 more come back to it with a fresh query parameter (ya.ru twice, salesforce.com), and 2 go somewhere new (amp.dev, hihonorcloud.com). The loops look like this:

Trailing slash, both ways. huawei.com serves /en/ and names /en as canonical, and /en redirects to /en/. agora.io does the same. ubisoft.com does it in the other direction.
www. both ways. go.com is served at www.go.com with canonical go.com, which redirects to www.go.com. bugsnag.com does the same.
Mobile and edition hosts. vk.com sends my user agent to m.vk.com, whose canonical is vk.com, which redirects back to m.vk.com. cnn.com lands on edition.cnn.com with canonical www.cnn.com.
A path the server won't serve. epa.gov names /home, which redirects to /.

A canonical that redirects isn't fatal. Google calls rel=canonical and redirects both signals and picks one. But here the two signals point in opposite directions, so the site has handed the choice to the crawler.

The two hard failures are a stanford.edu homepage whose canonical is https://www.stanford.edu/home, which answers 404 (rechecked with curl, same result), and synology.com, whose canonical is https://www.synology.com// with a double slash, which answers 403.

The canonical depends on where you are

Six of the "different host" and "different path" cases, from four companies, point at a regional page when seen from Japan:

facebook.com (and fb.com, fbcdn.net): canonical https://ja-jp.facebook.com/
android.com: canonical https://www.android.com/intl/ja_jp/
forbes.com: canonical https://www.forbes.com/home_asia/
intel.com: served from www.intel.co.jp, canonical on www.intel.com

This can be intentional. It also means a crawler in the US and a crawler in Japan are told different things about the same URL, and a homepage can quietly be declared a copy of a regional page. If you serve by location, check the canonical from more than one country.

`og:url` versus canonical

19 of 291 disagree. Most are small (instagram.com has canonical www.instagram.com and og:url instagram.com), but two are real: roblox.com's homepage says og:url is /CreateAccount, and gandi.net's canonical is /en-US while og:url is /en. Link previews and some crawlers read og:url, so it's worth keeping it in sync.

The hreflang numbers

n = 197 homepages with hreflang (37% of 529). Median 13 alternates, maximum 270.

	Homepages
Includes `x-default`	137
Lists itself among its alternates	183
Doesn't list itself	14
hreflang on a page whose canonical points elsewhere	19
A code that isn't a valid language or region	7
A 3-letter language code (`fil`, `ceb`, `skr`)	5
A withdrawn code (`iw`, `in`)	4
Same code, two different URLs	4
Relative or non-HTTP URL	2

The 7 invalid ones: weebly.com and amp.dev use underscores (en_GB, pt_BR); viber.com uses ua and gr, which are country codes, not languages; aliyun.com uses tc; hp.com writes country first (py-es, us-en) plus internal names like lamerica_nsc_carib-en; ezviz uses region eu; shein.com uses hr-eur and zh-tw-sg. Google's localized versions guide asks for an ISO 639-1 language with an optional ISO 3166-1 region. I count fil or iw separately because they are valid language tags, but fil has no ISO 639-1 code and iw was replaced by he in 1989.

"hreflang on a page whose canonical points elsewhere" is worth a look on your own site: the page says it is a copy of another URL and, at the same time, offers its own set of language versions. On the top 1,000 most of these are the harmless query-string cases above.

Return links

191 sites had at least one alternate I could fetch. 522 alternates in total.

Alternate
200 and links back	479
200, has hreflang, but not back to the homepage	20
200, no hreflang at all	8
Not 200 (403, 404, 401, no answer)	15

174 sites passed on every alternate I checked, 13 had at least one missing return link, and 4 couldn't be checked.

Most of the 13 have the same cause: the homepage isn't part of its own set. atlassian.com, cisco.com, ea.com and name.com list /ja/, /de-de/ or /site/us/en/index.html from the root URL but not the root URL itself, so the alternates have nothing to link back to. That's the "doesn't list itself" row above. The 8 alternates with no hreflang at all were on aliexpress.com, squarespace.com, ikea.com and blackberry.com, whose localised pages carried no hreflang in the HTML I received.

7 of the 15 non-200 alternates are on cloudflare.com: workers.dev and pages.dev land on product pages whose /es-es/, /fr-fr/ and /de-de/ versions answered 404 to me, and the /de-de/ homepage answered 403 to my user agent.

A checklist that catches all of the above

For a homepage, and any page you care about:

One canonical, absolute, in the head. Not two, not relative.
Fetch the canonical URL with redirects off. It should answer 200, and its own canonical should be itself. This one check finds the trailing-slash and www. loops, the 404 and the 403.
If you serve by location, fetch from two countries and compare the canonical.
hreflang: list the page itself, add x-default, use language or language-REGION with a hyphen, and point only at canonical URLs.
Fetch every alternate and check it answers 200 and lists the others, including this page.
Make og:url the same as the canonical.

In shell, step 2 is one line per page:

curl -s -o /dev/null -w '%{http_code} %{redirect_url}\n' "$(curl -s https://example.com/ | grep -o '<link[^>]*rel="canonical"[^>]*>' | grep -o 'href="[^"]*"' | cut -d'"' -f2)"

200 and an empty redirect is what you want.

The probe, follow and analysis scripts, the raw JSON lines per domain, and the summary file that every number above comes from are kept with the date above.

If you'd like one site checked by hand, the paid GEO Mini Audit covers crawler access and machine-readability for one page you give (robots.txt per AI crawler, what your server answers each one, canonical, hreflang, meta robots and structured data) and adds paste-ready fixes (a corrected robots.txt, CDN/WAF steps for bot 403s, meta and canonical fixes, JSON-LD and llms.txt skeletons), all marked review before deploying. It queries no AI engine and measures no citations or rankings.

Your IMAP script stopped logging in: XOAUTH2 for Gmail and Microsoft 365

Mahiro Hirakawa — Tue, 06 Oct 2026 16:35:06 +0000

If you have a small Python script that logs in to Gmail or Microsoft 365 over IMAP with a username and password, there's a good chance it already gets AUTHENTICATE failed or Invalid credentials back, even with the right password. Both providers have moved IMAP to OAuth 2.0, and the IMAP side of that is a SASL mechanism called XOAUTH2.

This post covers why the password path closed, what XOAUTH2 actually sends over the wire, where the access token comes from, and how long it lives. The worked example uses mail-rule-digest, a stdlib-only Python CLI I wrote that filters a mailbox with a TOML rule file and writes a Markdown digest. Every command and output below comes from a real run on 2026-10-07 against a fake IMAP server on localhost, with a fake token. No real mailbox or credential was involved.

Why the password stopped working

Microsoft 365 / Exchange Online. Microsoft turned off Basic authentication for IMAP and POP in Exchange Online. The deprecation page says "All other cloud environments were subject to the October 1, 2022, date" and "Basic authentication is now disabled in all tenants" (learn.microsoft.com). For personal Outlook.com accounts the date was "September 16th, 2024 - Basic Authentication no longer available to access any Outlook account" (support.microsoft.com).

Google. Google wound down "less secure apps", meaning third-party apps that sign in with only a username and password. The Workspace Updates post first announced September 30, 2024, and its latest update reads: "Less Secure Apps will no longer be supported as of May 1, 2025" (workspaceupdates.googleblog.com). Google's pages don't all agree on the final date; the admin help page that the old support link redirects to still says March 14, 2025. Either way, it's in the past.

One exception remains for personal Google accounts: app passwords. Google's help page says "App passwords can only be used with accounts that have 2-Step Verification turned on", calls them not recommended, and excludes work and school accounts (support.google.com). If you control a personal account and just need a script working tonight, that's the short path. For Workspace or Microsoft 365 mailboxes, OAuth is the path.

What XOAUTH2 sends

XOAUTH2 is a SASL mechanism (SASL is RFC 4422; IMAP carries it with the AUTHENTICATE command, RFC 9051 section 6.2.2). The client sends one string, base64-encoded. Google documents it as:

base64("user=" {User} "^Aauth=Bearer " {Access Token} "^A^A")

where ^A is the byte 0x01 (developers.google.com). Microsoft's page has the same format: base64("user=" + userName + "^Aauth=Bearer " + accessToken + "^A^A") (learn.microsoft.com).

That's the whole credential. There is no signature and no nonce: the bearer token is the secret, so anything that logs this line logs the token. In Python it's one line, and imaplib does the base64 step for you:

def xoauth2_string(user: str, token: str) -> bytes:
    return f"user={user}\x01auth=Bearer {token}\x01\x01".encode()

conn.authenticate("XOAUTH2", lambda _challenge: xoauth2_string(user, token))

Here is what my fake server received during the demo below, decoded with ^A for 0x01:

server: C: JGLC1 AUTHENTICATE XOAUTH2
server: C: dXNlcj1kZW1vQGV4YW1wbGUuY29tAWF1dGg9QmVhcmVyIHlhMjkuRkFLRS1MT0NBTC1ERU1PLVRPS0VOAQE=
server:    decoded: user=demo@example.com^Aauth=Bearer ya29.FAKE-LOCAL-DEMO-TOKEN^A^A

The failure path has one quirk worth knowing. Gmail doesn't answer a bad token with NO straight away. It sends another + continuation carrying a base64 JSON error, and "The client sends an empty response ("\r\n") to the challenge containing the error message" (Google's XOAUTH2 page). Only then does the server send the tagged NO. If your callback blindly returns the token again, you resend the secret into an error path. The callback I use returns the token on the first challenge and b"" on any later one. Microsoft's page shows only a plain A01 NO AUTHENTICATE failed. for IMAP, so handling both shapes is the safe choice.

Where the token comes from, and how long it lives

The tool never runs a consent flow. You register an OAuth client with the provider, do the consent once with whatever library or CLI you like, and hand the script an access token.

Gmail. The scope "for IMAP, POP, and SMTP access is https://mail.google.com/" (Google's XOAUTH2 page). On the Gmail API scopes page it sits under "Restricted scopes", which require Google's restricted-scope app verification before a public app can use them. Two expiry details matter for a scheduled job:

Access tokens: Google's OAuth overview says "Access tokens have limited lifetimes" and the token response carries expires_in, the remaining lifetime in seconds. Read that field; don't hard-code an hour (developers.google.com).
Refresh tokens: a project with an external user type and "a publishing status of 'Testing' is issued a refresh token expiring in 7 days" (same page). If your cron job dies every week, this is usually why.

Microsoft 365 / Outlook.com. The delegated scope string on Microsoft's IMAP OAuth page is https://outlook.office.com/IMAP.AccessAsUser.All, and you add offline_access to get a refresh token. The IMAP host is outlook.office365.com, port 993 (learn.microsoft.com). For an unattended service there's an app-only route instead: the IMAP.AccessAsApp permission, a token requested with https://outlook.office365.com/.default, a service principal registered in Exchange, and an explicit mailbox permission grant. That one needs tenant admin consent.

Access tokens from the Microsoft identity platform get "a random value ranging between 60-90 minutes (75 minutes on average) as the default lifetime" (learn.microsoft.com). Also check IMAP itself: it's on by default in Exchange Online, but if security defaults are enabled, "POP3 and IMAP4 are automatically disabled" (the POP3/IMAP4 page above).

The practical consequence for both providers: a token pasted into an environment variable works for about an hour. A daily job needs something that refreshes. mail-rule-digest takes either IMAP_ACCESS_TOKEN (the token itself) or IMAP_TOKEN_COMMAND, a command whose stdout is a fresh access token, run at the start of every run.

The demo: a local fake server, a fake token

To get real output without touching a real mailbox, I wrote a ~100-line IMAP-over-TLS server in Python that only speaks XOAUTH2, accepts exactly one fake token, serves the five .eml test fixtures from the repo, and logs every command it receives. It refuses LOGIN the way a provider that disabled Basic auth would. Its certificate is signed by a throwaway local CA. All of this ran on Ubuntu under WSL2, Python 3.12.3, with the package installed from the repo at commit 698f960 (version 0.2.0).

The core of the fake server's AUTHENTICATE handling:

self.send("+ ")
decoded = base64.b64decode(self.rfile.readline().strip())
if decoded == f"user={USER}\x01auth=Bearer {TOKEN}\x01\x01".encode():
    self.send(f"{tag} OK XOAUTH2 authentication successful")
else:
    err = b'{"status":"400","schemes":"Bearer","scope":"https://mail.google.com/"}'
    self.send("+ " + base64.b64encode(err).decode())   # Gmail-style error challenge
    self.rfile.readline()                               # the client's empty response
    self.send(f"{tag} NO [AUTHENTICATIONFAILED] Invalid credentials (Failure)")

Config: a rule file and an env file

The rules are the repo's example file, examples/rules.toml, unchanged:

[[rule]]
name = "Invoices over $100"
from = ["billing@acme.example", "*@invoices.example"]   # glob, case-insensitive
subject = '(?i)\binvoice\b'                           # Python regex, searched
body_any = ["due", "overdue"]                          # case-insensitive substring

  [rule.numbers.amount]
  # The named group must have the same name as the field.
  pattern = 'Total:\s*\$(?P<amount>[\d,]+(?:\.\d+)?)'
  min = 100

[[rule]]
name = "Disk alerts at 90% or more"
from = "alerts@monitor.example"

  [rule.numbers.disk]
  pattern = '(?i)disk usage:?\s*(?P<disk>\d+)%'
  min = 90
  max = 100

[[rule]]
name = "Parcels out for delivery"
body_all = ["tracking number", "out for delivery"]

Connection settings live in the environment, never in that file. fake-token-cli is a tiny shell script standing in for a real OAuth helper that refreshes and prints a token:

$ cat mail.env
export IMAP_HOST=localhost IMAP_PORT=9993 IMAP_USER=demo@example.com
export IMAP_TOKEN_COMMAND=./fake-token-cli
$ ./fake-token-cli
ya29.FAKE-LOCAL-DEMO-TOKEN
$ . ./mail.env

First run: the certificate is checked

Before trusting the local CA, the run fails, which is what you want:

$ mail-rule-digest --rules rules.toml --days 3 --dry-run
error: cannot connect to localhost:9993: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1000)

Exit code 2, and the server log stays empty: the token was never sent. For the rest of the demo I point Python at the throwaway CA with export SSL_CERT_FILE=$PWD/ca.pem. Don't do that against real servers; it replaces the system trust store for that process.

Dry run

--days 3 because the fixtures are dated two days before the run, and IMAP SINCE works in whole days. --dry-run prints the digest and writes nothing. The [dry-run] line goes to stderr, which is why it shows up first here:

$ mail-rule-digest --rules rules.toml --days 3 --dry-run

[dry-run] 5 scanned, 3 matched; nothing written, nothing posted
# Mail digest 2026-10-07

3 matching message(s) across 3 rule(s).

## Invoices over $100 (1)

- **Invoice INV-1042 is ready** - billing@acme.example, 2026-10-05 09:15 - amount: 1,240.50

## Disk alerts at 90% or more (1)

- **\[ALERT\] disk usage 93% on web-1 @everyone** - alerts@monitor.example, 2026-10-05 12:05 - disk: 93

## Parcels out for delivery (1)

- **Your parcel is out for delivery 📦** - noreply@ship.example, 2026-10-05 11:45

The other two fixtures are an invoice under $100 and a newsletter, and neither matched. The server's view of the same run is the more interesting part:

server: C: JGLC0 CAPABILITY
server: C: JGLC1 AUTHENTICATE XOAUTH2
server: C: dXNlcj1kZW1vQGV4YW1wbGUuY29tAWF1dGg9QmVhcmVyIHlhMjkuRkFLRS1MT0NBTC1ERU1PLVRPS0VOAQE=
server:    decoded: user=demo@example.com^Aauth=Bearer ya29.FAKE-LOCAL-DEMO-TOKEN^A^A
server: C: JGLC2 EXAMINE "INBOX"
server: C: JGLC3 SEARCH SINCE 05-Oct-2026
server: C: JGLC4 FETCH 1 (BODY.PEEK[])
server: C: JGLC5 FETCH 2 (BODY.PEEK[])
server: C: JGLC6 FETCH 3 (BODY.PEEK[])
server: C: JGLC7 FETCH 4 (BODY.PEEK[])
server: C: JGLC8 FETCH 5 (BODY.PEEK[])
server: C: JGLC9 LOGOUT

EXAMINE opens the folder read-only, and BODY.PEEK[] fetches a message without setting \Seen. No STORE, COPY, MOVE or EXPUNGE appears, because the code has none.

A bad token

An expired or wrong token goes through the error-challenge path described earlier:

$ IMAP_ACCESS_TOKEN=ya29.EXPIRED-TOKEN mail-rule-digest --rules rules.toml --days 3 --dry-run
error: IMAP error: [AUTHENTICATIONFAILED] Invalid credentials (Failure)

server: C: EAIP1 AUTHENTICATE XOAUTH2
server: C: dXNlcj1kZW1vQGV4YW1wbGUuY29tAWF1dGg9QmVhcmVyIHlhMjkuRVhQSVJFRC1UT0tFTgEB
server:    decoded: user=demo@example.com^Aauth=Bearer ya29.EXPIRED-TOKEN^A^A
server: C: '' (empty response to the error challenge)
server: C: EAIP2 LOGOUT

The token went out once, the client answered the JSON challenge with an empty line, and the error message the user sees doesn't contain the token. When IMAP_ACCESS_TOKEN is set it wins over IMAP_TOKEN_COMMAND, which is why this run used the bad token.

The old password path, for comparison

$ env -u IMAP_TOKEN_COMMAND IMAP_PASSWORD=hunter2 mail-rule-digest --rules rules.toml --days 3 --dry-run
error: IMAP error: b'[AUTHENTICATIONFAILED] password login disabled, use XOAUTH2'

The server log shows LOGIN demo@example.com "hunter2". Plain LOGIN puts the password on the wire inside TLS, and a server that has switched Basic auth off just says no. (The b'...' in that message is imaplib handing back bytes on the LOGIN path; cosmetic, but it's a real difference between the two paths.)

Digest file and webhook

Without --dry-run the tool writes digest-YYYY-MM-DD.md, and --webhook also posts it to $WEBHOOK_URL. Here the webhook is a second local HTTPS server that prints what it receives:

$ export WEBHOOK_URL=https://localhost:8443/hook
$ mail-rule-digest --rules rules.toml --days 3 --webhook
5 scanned, 3 matched -> digest-2026-10-07.md
webhook: HTTP 200
$ ls digest-*.md
digest-2026-10-07.md

The receiver got this (the text value is truncated here; it's the same Markdown as above):

hook: POST /hook application/json
{
  "text": "# Mail digest 2026-10-07\n\n3 matching message(s) across 3 rule(s).\n\n## Invoices over $100 (1)\n\n- **Invoice INV-1042 is ready** - billing@acme.example, 2026-10-05 09:15 - amount: 1,240.50\n\n..."
}

{"text": ...} is the Slack incoming-webhook shape. For a discord.com URL the tool sends {"content": ..., "allowed_mentions": {"parse": []}} instead, cut to Discord's 2,000-character limit. Mentions are disabled on purpose: look at the disk alert subject above. It contains @everyone, and without that setting a forwarded email subject could ping a whole Discord channel. I didn't run the Discord branch here because it needs the real host; the repo's tests cover it.

A plain-HTTP webhook URL is refused. The digest file is still written first, so the exit code is 1, not 2:

$ WEBHOOK_URL=http://localhost:8443/hook mail-rule-digest --rules rules.toml --days 3 --webhook
5 scanned, 3 matched -> digest-2026-10-07.md
error: webhook failed: webhook URL must use https

Safety notes

Never put tokens in the config file. The rule file holds rules only. Credentials come from IMAP_PASSWORD, IMAP_ACCESS_TOKEN or IMAP_TOKEN_COMMAND, and there are no command-line flags for them, since flags end up in shell history and in ps output. The token command runs without a shell (split like a POSIX command line), with a 30-second timeout. If it fails, the error names the exit status only; neither its stdout nor its stderr is echoed, because either could contain the token. The refresh token itself should live wherever your OAuth helper keeps it, ideally the OS keychain, not next to the cron entry.

Read-only fetch. EXAMINE plus BODY.PEEK[], as the server log showed. The tool never sends, deletes, moves or flags mail. The scope is another matter: Gmail's https://mail.google.com/ and Microsoft's IMAP.AccessAsUser.All both grant full mailbox access. Read-only is a property of this code, not of the token, so treat the token as a full-access credential.

Certificate verification. This one surprised me when I first checked it. imaplib.IMAP4_SSL(host, port) with no ssl_context does not verify the server certificate. On the same Python 3.12.3, against the same fake server and with no trust for its CA, the bare call connected, while ssl_context=ssl.create_default_context() failed with CERTIFICATE_VERIFY_FAILED. An unverified connection hands your bearer token to whoever answers on port 993. Always pass a verifying context, as the first demo run showed the tool doing.

The digest is as private as the mailbox. It contains subjects and sender addresses. A webhook URL is also a secret, since anyone holding it can post to your channel.

What the tool does not do. It doesn't run the OAuth consent flow, store or refresh tokens, or ship a client ID. It only speaks IMAP over TLS on port 993: no STARTTLS, no POP3, no Gmail API. It keeps no state between runs, so running twice in a day produces the same digest twice. And it has not been run against live Gmail or Microsoft servers by its own test suite; the 38 tests use fake servers and fixtures, as this post does. If a provider rejects your token, you'll see the server's IMAP error: ... text, without the token in it.

Sources

All fetched 2026-10-07.

Google, "OAuth 2.0 Mechanism" (XOAUTH2 format, error challenge, scope): https://developers.google.com/gmail/imap/xoauth2-protocol
Google, "Using OAuth 2.0 to Access Google APIs" (token lifetime, 7-day Testing refresh tokens): https://developers.google.com/identity/protocols/oauth2
Google, Gmail API scopes (restricted scopes): https://developers.google.com/workspace/gmail/api/auth/scopes
Google Workspace Updates, "Winding down Google Sync and Less Secure Apps support": https://workspaceupdates.googleblog.com/2023/09/winding-down-google-sync-and-less-secure-apps-support.html
Google, "Sign in with app passwords": https://support.google.com/accounts/answer/185833
Microsoft, "Authenticate an IMAP, POP or SMTP connection using OAuth": https://learn.microsoft.com/en-us/exchange/client-developer/legacy-protocols/how-to-authenticate-an-imap-pop-smtp-application-by-using-oauth
Microsoft, "Deprecation of Basic authentication in Exchange Online": https://learn.microsoft.com/en-us/exchange/clients-and-mobile-in-exchange-online/deprecation-of-basic-authentication-exchange-online
Microsoft, "POP3 and IMAP4 in Exchange Online": https://learn.microsoft.com/en-us/exchange/clients-and-mobile-in-exchange-online/pop3-and-imap4/pop3-and-imap4
Microsoft, "Access tokens in the Microsoft identity platform": https://learn.microsoft.com/en-us/entra/identity-platform/access-tokens
Microsoft Support, "Modern authentication methods now needed to continue syncing Outlook Email in non-Microsoft email apps": https://support.microsoft.com/en-us/office/modern-authentication-methods-now-needed-to-continue-syncing-outlook-email-in-non-microsoft-email-apps-c5d65390-9676-4763-b41f-d7986499a90d
RFC 4422 (SASL): https://www.rfc-editor.org/rfc/rfc4422 and RFC 9051 (IMAP4rev2): https://www.rfc-editor.org/rfc/rfc9051

The tool, its tests and the example rules: https://github.com/mahirhir/mail-rule-digest

0 of 550 top homepages send noai. 21 send Bing's noarchive.

Mahiro Hirakawa — Tue, 06 Oct 2026 16:06:59 +0000

I fetched the homepages of the Tranco top 1,000 and read every meta robots tag and every X-Robots-Tag header they sent. 550 homepages came back as real pages. None of the 550 sends noai or noimageai. None sends nosnippet either, and the 47 that send max-snippet all set it to -1, which means "no limit". The page-level AI rule that big sites actually use is Bing's: 21 of the 550 send noarchive or nocache, the two values Microsoft says keep a page out of Copilot answers or cut it down to a link and a snippet.

So the AI policy of the top 1,000 lives almost entirely in robots.txt. That's fine, as long as you know what each lever reaches. Most of the confusion I see comes from mixing them up: a noai tag nobody major reads, a noindex behind a robots.txt block that stops the crawler from ever seeing it, or a header and a meta tag that say different things.

Below: what each lever reaches, the numbers, three real sites with three different setups, and how to write one policy that says the same thing everywhere.

Limits first

noai and noimageai are not a standard. They aren't in Google's list of supported robots rules, and the crawler pages from OpenAI, Anthropic and Perplexity describe robots.txt tokens as the opt-out, not page tags. Some tools honour them. img2dataset, which builds image-text training sets, skips images whose own HTTP response carries X-Robots-Tag: noai or noimageai. It reads the header on the image, not a meta tag on the page around it.
Homepage only. One URL per site. A site can put directives on articles, images or PDFs and leave the homepage clean, and I wouldn't see it. "0 of 550" is about homepages, not about whole sites.
One network location, one user agent. One GET per domain on 2026-10-06, 15:59 to 16:01 UTC, from one connection in Japan, with a desktop Chrome user agent, redirects followed. A server can send different headers to GPTBot than to Chrome. I didn't test that. I also didn't run JavaScript, so a meta tag injected client-side is invisible here (and to most crawlers).
List and set: Tranco list 56WKN, top 1,000, the same 742 domains that answered HTTPS in my llms.txt count and my robots.txt count.
192 of the 742 didn't give me a homepage. 114 answered 401 or 403 to a browser user agent, 8 answered 200 with a bot wall or an error page (microsoft.com's said "Your request has been blocked"), and 70 answered 404, 400, 5xx, 429, something that wasn't HTML, or nothing. Those 192 are unknown, not "no directives". n = 550 below.

What each lever reaches

Lever	Who reads it	What it does
robots.txt `User-agent: GPTBot` etc.	Crawlers that name that token	Whether the bot may fetch the URL at all. Details per token in the robots.txt post
`nosnippet`, `max-snippet:N` (meta or header)	Google	Google: `nosnippet` "will also prevent the content from being used as a direct input for AI Overviews and AI Mode". `max-snippet` limits how much can be used
`noindex`, `none`	Search engines	Out of the index, and so out of everything built on it
`noarchive`	Bing	Microsoft: "will not be included in Bing Chat answers, not be linked to in the answers", and not used for training. Google no longer lists it
`nocache`	Bing	Microsoft: only "URL/Snippet/Title" in answers; only those used for training
`noai`, `noimageai`	Some dataset tools	Non-standard. See the limits above

Sources: Google's robots meta tag reference and AI features page, Microsoft's Bing webmaster blog, 2023-09-22, img2dataset. All read 2026-10-07.

Two rules hold across all of these.

A crawler that robots.txt keeps out never reads your page-level rules. Google says it directly: if a page is disallowed in robots.txt, "any information about indexing or serving rules will not be found and will therefore be ignored." A noindex on a page Googlebot can't fetch does nothing.

Meta tag and header are the same instruction in two places. Google treats them as one set, and "the more restrictive rule applies". The header is the only option for PDFs and images. The meta tag is easier to change in a CMS.

The numbers

n = 550 homepages, 2026-10-06.

	Homepages
Any meta robots tag	192
Any `X-Robots-Tag` header	15
`noai` or `noimageai` (meta or header)	0
`nosnippet`	0
`max-snippet`	47, all `-1` (no limit)
`noindex` for every crawler	7
`noarchive` or `nocache`	21
Directive aimed at a named AI crawler (`GPTBot`, `ClaudeBot`, ...)	0

The 7 noindex homepages aren't AI policy. They're login apps (outlook.com and live.com land on Outlook's mail login), two domain registrars, a cloud login, an ad-tech company and a parked domain.

The 21 Bing signals split in two:

10 aim only at Bing: <meta name="bingbot" content="noarchive"> or X-Robots-Tag: bingbot: noarchive. instagram.com, threads.com, theguardian.com, bbc.com, bbc.co.uk, elpais.com, cnet.com and nikkei.com send noarchive; tumblr.com and theverge.com send nocache.
11 send noarchive to every crawler, among them linkedin.com, npr.org, lemonde.fr, dailymail.co.uk, usatoday.com and zoho.com.

max-snippet:-1 on 47 homepages is mostly SEO plugin boilerplate (it comes with max-image-preview:large and max-video-preview:-1). It says "use as much as you like". Nobody in the top 1,000 uses max-snippet to limit AI Overviews on the homepage.

Meta versus header

3 sites send rules in both places for the same crawler, and 2 of them send different sets. usatoday.com sends noarchive, nocache in the header and only old noodp, noydir in the meta tag. repubblica.it sends noarchive in the header and not in the meta tag. Neither is a contradiction, since the stricter value wins. But anyone reading only the HTML would get the policy wrong, and that includes the person who maintains the template.

10 sites have more than one meta robots tag on the homepage. I found no page that says index in one place and noindex in another.

Against robots.txt

I compared every homepage with the same site's robots.txt from the earlier run (545 files, RFC 9309 matching).

noai against robots.txt that lets training crawlers in: 0, because there's no noai to compare.
A directive aimed at an AI crawler that robots.txt already keeps out: 0. The only named targets were bingbot (10) and googlebot-news (1).
Page-wide noindex that some AI crawlers can't see: 1. sharethrough.com puts noindex on its homepage and blocks all nine AI tokens in robots.txt. Both say no, so nothing breaks. The tag just never reaches those bots.

Of the 21 Bing-signal sites, 15 served a robots.txt file. 8 of the 15 fully block GPTBot and 10 fully block Google-Extended. 2 fully block none of the six AI tokens I compared: rubiconproject.com and zoho.com.

Three sites, three setups

bbc.com: every layer says the same thing. robots.txt fully blocks GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended and CCBot. The homepage adds X-Robots-Tag: bingbot: noarchive. Bingbot can't be blocked in robots.txt without leaving Bing search, so for Copilot the page-level switch is the only lever. That's the pattern I'd copy.

rubiconproject.com: page-level only. The meta tag says noarchive to every crawler. robots.txt allows all six AI tokens. If the intent is "no generative use", it has been said only to Microsoft. If the intent was only "no cached copy", then it's a leftover that now also keeps them out of Copilot answers. I can't tell which from the outside.

theverge.com: mixed, and maybe on purpose. <meta name="bingbot" content="nocache"> (Copilot may show a link and a snippet), robots.txt blocks ClaudeBot, PerplexityBot, Google-Extended and CCBot fully, and doesn't block GPTBot at all. That reads like a policy decided per company. It could also be a deal. The file doesn't say, and that's the problem with a policy spread over three places: the next person to edit it can't either.

Write one policy

Decide per purpose first, then pick the lever the target actually reads.

Training. Use robots.txt tokens: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider. For Bing, noarchive. If you also want noai, send it as a header on images, where the tools that honour it look.
AI answers and search. OAI-SearchBot, Claude-SearchBot and PerplexityBot in robots.txt. For Google's AI Overviews, robots.txt can't help without leaving Google Search; nosnippet or a max-snippet value is the lever. For Copilot, nocache (link and snippet only) or noarchive (out).
Gone completely. noindex, and then let crawlers fetch the page so they can read it.

"No training, answers with a link only" for a news-style site, written out:

# robots.txt
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Bytespider
Disallow: /

User-agent: *
Disallow: /search

<!-- every HTML page -->
<meta name="bingbot" content="nocache">

# images and PDFs, where a meta tag can't go
location ~* \.(jpg|jpeg|png|webp|pdf)$ {
  add_header X-Robots-Tag "noai, noimageai" always;
}

Then check for the four contradictions I looked for above:

a page-level rule for a bot that robots.txt keeps out (it's never read),
a meta tag and a header for the same crawler with different values,
two meta robots tags that disagree,
a page-wide noarchive or noindex left over from an old SEO setup that now also decides your AI policy.

Write the policy down in a comment at the top of robots.txt, even if it's only one line. robots.txt is the one place every crawler reads, and the comment is the only part a person reads.

The raw data (one JSON line per homepage with the raw X-Robots-Tag values and meta tags, the saved <head> of each page, the probe and analysis scripts, and a control file that plants each directive to check the counts) is kept with the date above.

The free checker shows, for GPTBot, ClaudeBot and PerplexityBot, whether robots.txt allows them and what your server actually returns, with a short preview of what else it found on the page. It flags noai, noimageai, nosnippet, low max-snippet, noindex and none; it doesn't read Bing's noarchive or nocache yet.

If you want one site gone through end to end, the paid audit on the same page covers crawler access and machine-readability: robots.txt per bot, what your server answers each crawler, page-level directives, structured data, sitemap and llms.txt. It doesn't query any AI engine, so it can't tell you whether you appear in their answers. It tells you whether they can read you, and whether your rules say what you meant.

111 of the top 1,000 sites block GPTBot. 35 never mention the bot behind ChatGPT search.

Mahiro Hirakawa — Tue, 06 Oct 2026 10:06:03 +0000

111 of the top 1,000 sites fully block GPTBot in robots.txt. 51 of those 111 don't fully block OAI-SearchBot, and 35 of the 51 never mention OAI-SearchBot at all. GPTBot is OpenAI's training crawler. OAI-SearchBot is the one that decides whether a page can show up in ChatGPT search. So a third of the sites that said "no" to OpenAI said it only to training, and most of them probably didn't decide that on purpose. I can't tell from the file. Nobody can, which is the point of this post.

Below: what each AI token actually controls, the numbers from the top 1,000 sites as of today, the mistakes I found in real files, and how to write your own policy so it says what you mean.

Limits first

robots.txt is a request. It doesn't block anything. A crawler that ignores it gets the page. And it says nothing about what your server or CDN does: in an earlier measurement, 19 of 79 small-business sites refused at least one AI crawler at the server, mostly against their own robots.txt. Everything below is about the file, not about what a bot can actually fetch.
List: Tranco list 56WKN, generated 2026-10-05, top 1,000. It's the same list and the same reachable set as my llms.txt count: 742 of the 1,000 domains answered HTTPS. The rest are mostly infrastructure hostnames with no website.
When and how: one GET of https://<domain>/robots.txt per domain on 2026-10-06, 09:59 to 10:00 UTC, from one connection in Japan, with a desktop Chrome user agent, redirects followed. A server can hand a different robots.txt to a different user agent. I didn't test that.
One snapshot. These files change, and AI sections change more often than the rest.
Matching: RFC 9309 rules. A bot uses the group that names it; only if no group names it does it fall back to User-agent: *. Blank lines are ignored. "Fully blocked" below means disallowed at both / and a deep path. "Partially" means allowed at / but at least one Disallow applies to it.

Of the 742, 545 served a robots.txt file. 144 had none (76 answered 404 or similar, 68 answered 200 with an HTML page, which a parser reads as a file with no rules). 53 I couldn't read: 45 answered 401 or 403, 8 returned 5xx, 202 or 429. RFC 9309 says a 4xx means "no restrictions" and a 5xx means "assume everything is disallowed". I count all 53 as unknown.

What each token controls

This is the part most robots.txt files get wrong, so here it is from each vendor's own page, read today.

Token	Owner	What disallowing it does
`GPTBot`	OpenAI	Opts out of crawling for model training
`OAI-SearchBot`	OpenAI	Opts out of ChatGPT search results. OpenAI: "Each setting is independent of the others"
`ChatGPT-User`	OpenAI	User-triggered visits. OpenAI says robots.txt rules "may not apply"
`ClaudeBot`	Anthropic	Opts out of crawling for training
`Claude-SearchBot`, `Claude-User`	Anthropic	Search quality, and visits on a user's request
`anthropic-ai`	(Anthropic, older)	Not on Anthropic's current crawler page, which lists only the three above
`PerplexityBot`	Perplexity	Opts out of Perplexity's search index. Perplexity says it is not used for model training
`Perplexity-User`	Perplexity	User-triggered fetches. Perplexity says it "generally ignores robots.txt rules"
`Google-Extended`	Google	Not a crawler. Controls whether content Google already crawls can train Gemini. Google: it "does not impact a site's inclusion in Google Search"
`Applebot-Extended`	Apple	Not a crawler either. Controls training use of what Applebot fetched; such pages "can still be included in search results"
`CCBot`	Common Crawl	Opts out of the open crawl that many training sets are built from
`Bytespider`	ByteDance	Training crawler

Sources: OpenAI, Anthropic, Perplexity, Google, Apple, Common Crawl, RFC 9309.

The split that matters: training tokens (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider) versus answer and search tokens (OAI-SearchBot, Claude-SearchBot, PerplexityBot). Blocking a training token doesn't take you out of that company's answers. Blocking a search token does, or at least asks to. And the user-triggered agents mostly don't read robots.txt at all.

The numbers

n = 545 robots.txt files, 2026-10-06.

Token	Fully blocked	of which by name	of which via `*`	Partially	Allowed
Bytespider	123	88	35	350	72
CCBot	121	88	33	348	76
GPTBot	111	83	28	351	83
ClaudeBot	109	78	31	353	83
Google-Extended	97	68	29	365	83
PerplexityBot	90	61	29	373	82
Applebot-Extended	90	57	33	380	75
anthropic-ai	84	47	37	385	76
OAI-SearchBot	61	35	26	400	84

A few things in there:

"Partially" is mostly noise. For GPTBot, 322 of the 351 come from the * group: ordinary rules like Disallow: /search or /cart that apply to every crawler. Those sites have no AI policy at all. They just have a robots.txt.
"Via *" means no AI policy either. 49 files have Disallow: / in their * group. Some are ad and CDN hostnames that block everyone. Others are allowlists from big platforms (facebook.com, x.com, reddit.com, netflix.com, yelp.com): block everyone, then name who gets in. For a bot those files don't name, "via *" is still a decision, just not one about AI.
169 of 545 files (31%) name at least one of the nine tokens. That's the real count of sites with any written AI policy.
OAI-SearchBot is blocked about half as often as GPTBot (61 vs 111). Some of that is a choice. Some of it isn't, as the next section shows.
In the top 100, 55 domains served a file and 13 of those block GPTBot fully, among them amazon.com, x.com, pinterest.com, tiktok.com, yahoo.com and baidu.com.
40 files block all nine tokens fully. 13 block all four training tokens I grouped (GPTBot, Google-Extended, Applebot-Extended, CCBot) and leave both OAI-SearchBot and PerplexityBot open. That second group is the clean "train on nothing, but answer with my pages" policy. It's rarer than I expected.

Mistakes in real files

Blocking training and, maybe by accident, leaving answers open

Of the 111 sites that fully block GPTBot, 51 don't fully block OAI-SearchBot.

16 name OAI-SearchBot and leave it open. yahoo.com, ebay.com, reuters.com, bloomberg.com, airbnb.com, forbes.com, tripadvisor.com, figma.com and others. That's a decision, and a coherent one.
35 never mention it. medium.com, nature.com, apnews.com, techcrunch.com, latimes.com, webmd.com, goodreads.com, pixiv.net and more. OAI-SearchBot falls through to their * group, which doesn't block it. Maybe that's what they want. But GPTBot and OAI-SearchBot are both from OpenAI and are often listed together, so a site that blocks one by name and doesn't mention the other most likely never decided about the second.

The reverse happened once. facebook.com ends its file with User-agent: * / Disallow: / and names the bots it allows. GPTBot is named, with path rules. OAI-SearchBot isn't, so it falls into the block-everything group. Result: training crawler partly in, search crawler out. When your file is an allowlist, every token you forget is blocked.

Blocking a token nobody sends anymore

4 files block anthropic-ai fully and never mention ClaudeBot: chatgpt.com, espn.com, repubblica.it and elmundo.es. ClaudeBot falls back to their * group and is only partly restricted. If the intent was "no Anthropic training", the file doesn't say so for the crawler Anthropic currently lists.

Naming a bot replaces the `*` group

This one surprised me most. Once a group names GPTBot, GPTBot ignores the * group completely. Whatever you disallowed for everyone no longer applies to it.

23 sites give GPTBot its own group without a full block, and that group lacks Disallow rules their * group has. hp.com has User-agent: GPTBot / Allow: /, and with it 170 paths from its * group (order status pages, site search, login forms, /api/) no longer apply to GPTBot. vimeo.com loses 325, kleinanzeigen.de 215, cisco.com 175, wordpress.org 4 (including /wp-admin/). Same pattern for OAI-SearchBot (28 sites) and PerplexityBot (25).

Maybe some of them want exactly that. Most * rules exist to keep crawlers out of cart and search pages, though, and I doubt anyone meant to let AI crawlers in there and nobody else. If you add a named group, copy your * rules into it.

Tokens that match no crawler

6 files had a user-agent value that is a near-miss of one of the nine:

lemonde.fr: User-agent: CCBo (it also has a correct CCBot group earlier, so no harm)
usatoday.com: User-agent: Claude, which no Anthropic crawler sends as its token
linktr.ee: User-agent: Google with a Content-Signal line. Google's tokens are Googlebot and Google-Extended
bestbuy.com: GPTBot-Web; nbcnews.com: ByteSpider Desktop; bloomberg.com: Bytespider-Image

Matching is on the whole product token, case-insensitive. Claude doesn't match ClaudeBot. These lines do nothing.

Blank lines inside a group

58 files put a blank line between a User-agent line and its rules. chatgpt.com does it for *, hp.com for every AI token it names. RFC 9309 ignores blank lines, so a compliant parser reads these files the way the authors meant. The old 1994 convention ended a record at a blank line, and a parser written to it would see those rules as belonging to no one. I didn't test which crawlers still parse that way. Deleting the blank lines costs nothing.

A rule above every group

sophos.com commented out the User-agent: * line at the top of its file and left Allow: / under it, above the first real group. A rule before any User-agent line belongs to no group and applies to nobody. Harmless here, since it was an Allow, but the same edit on a Disallow would silently drop it.

Decide your own policy

Two questions, answered separately.

Can my content train models? Then decide on GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot and Bytespider. Blocking Google-Extended doesn't remove you from Google Search, and blocking Applebot-Extended doesn't remove you from Apple's.
Can AI search products cite and link me? Then decide on OAI-SearchBot, Claude-SearchBot and PerplexityBot. Blocking these takes you out of those answers, or asks to.

User-triggered agents (ChatGPT-User, Claude-User, Perplexity-User) are a third thing, and by their own vendors' account robots.txt mostly doesn't govern them. If you need them stopped, that's a server rule, not a robots.txt line.

"No training, yes to answers" written out, with the * rules repeated where a named group would otherwise drop them:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Bytespider
Disallow: /

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Disallow: /cart
Disallow: /search

User-agent: *
Disallow: /cart
Disallow: /search

Several User-agent lines in a row form one group. Name every token you made a decision about, even when the decision is "allow", so the next person to edit the file can see it was decided.

Then check what your server actually answers each crawler. robots.txt is only the request; a CDN rule can refuse a bot your file allows, or let in one it disallows. The free checker shows, for GPTBot, ClaudeBot and PerplexityBot, whether your robots.txt allows them and what status your server actually returns to each.

The raw data (one JSON line per domain, every robots.txt I fetched, the probe and analysis scripts) is kept with the date above. Ask in the comments if you want the per-domain table.

If you want someone to go through one site end to end, the paid audit on the same page covers crawler access and machine-readability (robots.txt per bot, server answers, structured data, sitemap and llms.txt) and includes paste-ready fixes (a corrected robots.txt, CDN/WAF steps for bot 403s, meta and canonical fixes, JSON-LD and llms.txt skeletons), all marked review before deploying. It queries no AI engine and measures no citations or rankings. It tells you whether they can read you.

246 of the top 1,000 sites answer /llms.txt with 200. Only 114 serve the file.

Mahiro Hirakawa — Tue, 06 Oct 2026 06:42:03 +0000

If you count llms.txt adoption by asking which sites return 200 for /llms.txt, you'll more than double the real number. I checked the top 1,000 domains today. 246 answered 200. 114 of them actually served an llms.txt file. The other 132 sent a login page, a "not found" page with a 200 status, their home page, or a single-page app shell.

This post is the measurement, the three ways a naive check goes wrong, and a 20-line script that doesn't.

What I measured, and the limits first

List: Tranco list 56WKN, generated 2026-10-05, top 1,000 entries. Tranco ranks registrable domains by combined traffic signals, so it includes infrastructure domains (CDN hostnames, API domains, ad servers) that have no website at all.
When and from where: 2026-10-06, 06:30 to 06:37 UTC for the files, with the dead-link re-check in the following hour, from one ordinary connection in Japan. Some sites answer differently by country (a few redirected me to /jp/ or /ja-jp/ pages), so a run from elsewhere will differ at the edges.
Requests: one GET each to https://<domain>/, /llms.txt and /llms-full.txt, with a desktop Chrome user agent and Accept: text/plain,text/markdown,*/*. If https://<domain> didn't answer, I tried https://www.<domain>. Redirects were followed.
Not counted as "no file": 70 domains answered /llms.txt with 401 or 403. Some of those may have a file that a bot-protection layer kept from me. I count them as unknown, not as absent.
One snapshot. No second pass on the presence check. Status codes for a static file are much steadier than for home pages, but a single run is a single run.

742 of the 1,000 domains answered HTTPS at all. That's the denominator for everything below unless I say otherwise.

The headline numbers

What `/llms.txt` returned	Domains
An llms.txt-shaped text file	114
200, but an HTML page	122
200, but a redirect to some other page served as text (my `Accept` header asked for text, and a few servers converted the page to markdown)	8
200, but not a file at all (an IP echo service, a JSON error)	2
404 or 410	388
401 or 403 (unknown)	70
Anything else (400, 5xx, 202, timeouts)	38

So 114 of 742 reachable domains (15%) serve an llms.txt. In the top 100 it's 8 of the 75 that answered: cloudflare.com, github.com, fastly.net, wordpress.org, digicert.com, adobe.com, opera.com and sentry.io. 16 of the 114 get there by redirecting to a sibling domain's file, for example unity3d.com to unity.com and macromedia.com to adobe.com.

17 domains also serve llms-full.txt, and every one of them has an llms.txt too. Worth knowing: llms-full.txt is not in the llms.txt proposal at all. It's a convention that documentation platforms popularised.

Three ways a naive check lies

1. A 200 that is an HTML page

122 domains answered with an HTML page and status 200. 37 of them serve that HTML at /llms.txt itself, which is what a single-page app does when it routes every unknown path to the same shell. instagram.com, linkedin.com, pinterest.com and chatgpt.com are in this group. The other 85 redirected first: 38 to a home page, the rest to login screens or to a "not found" page that returns 200 (netflix.com lands on /NotFound, tiktok.com on /404).

Any tool that reports "llms.txt found" on status alone counts all 122. Check the Content-Type, check that the first non-blank byte isn't <, and check where you ended up after redirects.

2. A text file that isn't an llms.txt

The proposal requires exactly one thing: the file starts with an H1, the name of the site or project. Everything else (a blockquote summary, H2 sections holding lists of [name](url) links) is the recommended shape.

108 of the 114 start with an H1.
97 have the full shape: H1 first, at least one H2 section, at least one list item that is a link.
The 6 that don't start with an H1 open with a "Generated by ..." line (playstation.com, moloco.com), a bare URL (cisco.com), an H2 (agora.io, trendyol.com), or straight into a link list (salesforce.com). A generator tool wrote the first two, and nobody read the first line afterwards.
11 have the H1 but no linked list sections. porkbun.com has 14 H2 sections and no links at all; it's a prose summary of the site. That's allowed by the letter of the proposal, but an agent reading it has nowhere to go next.
9 have more than one H1.

Size is the other surprise. The median file is 11,378 bytes, which is fine. 12 are over 100 KB and the largest, salesforce.com's, is 1,021,703 bytes. At the rough rate of four bytes per token that's about a quarter of a million tokens. Version 2 of the proposal, published in August 2026, says the file "stays small enough to fit in context" and that the detail belongs behind the links. A 1 MB index is a sitemap with extra steps.

The same version says links should point to LLM-friendly content, such as a markdown copy of each page at page.md or page.html.md. Of the 107 files that contain links, 74 link to no .md or .txt URL at all and 8 point mostly to them. I only looked at the extension, so a site that serves markdown by content negotiation on a normal URL is undercounted here. Still, most files today are a list of ordinary HTML pages.

3. Dead links, and HEAD requests that lie about them

For every file with links I took up to 20 of them, spread evenly through the file, 2,043 links from 107 files in total. I sent HEAD first and fell back to GET only on errors.

HEAD said 81 of them were 404 or 410. When I re-fetched those 81 with GET, only 42 were actually gone. dynatrace.com answers HEAD with 404 on pages that GET returns with 200, which accounts for 15 of the 39 false alarms on its own. If your link checker uses HEAD to save bandwidth, it will report rot that isn't there.

The 42 confirmed dead links (2.1% of the sample) sit in 13 files, and they bunch up: 10 of 20 sampled links on moloco.com and 9 of 20 on weather.com are gone. 40 of the 42 point at the site's own host. These files were written once and the site moved on underneath them.

I wanted a staleness number from Last-Modified too, and it doesn't work. 74 of the 114 files send the header. 24 of those are dated the same day I fetched them, because the file is generated on each request, and one says 1 February 1980. Dead internal links are a better staleness signal than any date the server gives you.

Check your own

This prints one verdict per host. It needs bash and curl.

check_llms() {
  local host=$1 tmp
  tmp=$(mktemp)
  IFS=$'\t' read -r code ctype final < <(curl -s -L -o "$tmp" -A 'Mozilla/5.0' \
    -w '%{http_code}\t%{content_type}\t%{url_effective}\n' "https://$host/llms.txt")
  final=${final%$'\r'}
  local first
  first=$(sed -e '1s/^\xEF\xBB\xBF//' "$tmp" | grep -m1 -v '^[[:space:]]*$')
  printf '%-22s %s %-28s %s\n' "$host" "$code" "${ctype%%;*}" "$final"
  if [ "$code" != 200 ]; then echo "  -> no file ($code)"
  elif [[ "$ctype" == *html* || "$first" == \<* ]]; then echo "  -> HTML page answering 200, not an llms.txt"
  elif [[ "${final%%\?*}" != */llms.txt ]]; then echo "  -> redirected to $final, not an llms.txt"
  elif [[ "$first" != "# "* ]]; then echo "  -> text, but the first line is not an H1: ${first:0:60}"
  else echo "  -> llms.txt, $(wc -c < "$tmp") bytes, $(grep -c '^## ' "$tmp") H2 sections, $(grep -cE '^\s*[-*] \[' "$tmp") list links"
  fi
  rm -f "$tmp"
}
check_llms vercel.com; check_llms netflix.com; check_llms moloco.com; check_llms pages.dev

What it printed for me today:

vercel.com             200 text/plain                   https://vercel.com/llms.txt
  -> llms.txt, 4868 bytes, 9 H2 sections, 25 list links
netflix.com            200 text/html                    https://www.netflix.com/NotFound?prev=https%3A%2F%2Fwww.netflix.com%2Fllms.txt
  -> HTML page answering 200, not an llms.txt
moloco.com             200 text/plain                   https://www.moloco.com/llms.txt
  -> text, but the first line is not an H1: Generated by Moloco, this is an llms.txt file designed to he
pages.dev              200 text/html                    https://www.cloudflare.com/products/pages/
  -> HTML page answering 200, not an llms.txt

The script doesn't check links. For that, take the URLs out of the file and GET them; don't trust a HEAD 404.

What I'd change on a site today

If you don't have an llms.txt, the first thing is to make sure /llms.txt returns a real 404. A 200 HTML page there is worse than nothing, because a careless checker will tell you you're done. If you do have one: start it with # Your Site, keep it under a few tens of kilobytes, and re-run a link check whenever you restructure URLs, since that's what killed the dead links above.

And an llms.txt only helps if crawlers can fetch your pages in the first place. In my earlier measurement, 19 of 79 Amsterdam bike-shop sites refused at least one AI crawler at the server, mostly against their own robots.txt. The free checker tests that half: robots.txt against what your server actually answers GPTBot, ClaudeBot and PerplexityBot. It doesn't look at llms.txt.

The raw data (one JSON line per domain, the files I fetched, the probe and analysis scripts) is kept with the date above; ask in the comments if you want the per-domain table.

If you'd rather have a person go through one site end to end, the paid mini audit is on the same page as the checker. It checks AI-crawler access and machine-readability and includes paste-ready fixes (a corrected robots.txt, CDN/WAF steps for bot 403s, meta and canonical fixes, JSON-LD and llms.txt skeletons), all marked review before deploying. It queries no AI engine and measures no citations or rankings.

Correction 2026-10-06: an earlier version said the paid audit measures what AI engines answer; it checks crawler access and machine-readability only.

Your CDN can overrule robots.txt: finding the layer that refuses AI crawlers

Mahiro Hirakawa — Sun, 04 Oct 2026 13:22:33 +0000

In the first post I found Japanese tourism sites whose robots.txt welcomes GPTBot while the server answers it with 403. This one is about the next question: which layer is saying no, did anyone mean it, and what do you change.

I ran the same kind of check on a different set of sites first, to see whether the pattern was a Japanese hosting quirk. It isn't.

Amsterdam bike shops, measured on 2026-10-04

Source: every OpenStreetMap object tagged shop=bicycle with a website tag inside the Amsterdam municipality boundary, pulled from the Overpass API on 2026-10-04. That gave 108 objects and 92 unique hostnames. It is mostly independent repair shops and rental places, with a few chain stores and a couple of oddly tagged entries mixed in. I didn't hand-pick anything.

Each home page got one request with a browser user agent and one each with the published GPTBot, ClaudeBot and PerplexityBot user agents, from a single ordinary connection. Every site that refused a crawler got a second pass a few minutes later, and I only count a refusal if the same crawler got 403 both times. That rule threw out a handful of 429s, 503s and one timeout that looked like rate limiting or flaky hosting rather than a policy.

13 of the 92 hosts didn't return 200 to the browser at all (dead domains, timeouts, server errors), which leaves 79. Of those 79, 19 returned 403 to at least one AI crawler user agent on both passes while the browser got 200. That's 24%, close to the quarter I saw in the Japanese sample (6 of 23).

On 15 of the 19, robots.txt allows the crawler that was refused. 1 disallows it, so robots.txt and the server agree there, and 3 have no robots.txt.
ClaudeBot was refused on all 19, GPTBot on 14, PerplexityBot on 5. Five sites refuse ClaudeBot and nothing else. 12 of the 19 also refuse /robots.txt itself when asked as GPTBot.
A made-up user agent (ExampleFetchBot/1.0) got 200 on 18 of the 19. These rules name AI crawlers; they aren't generic "block anything with bot in it" filters.
5 refusals come from Cloudflare's edge (out of 21 sites served through Cloudflare). The other 14 come from the origin: 8 answer as nginx, 3 as Apache, 1 as LiteSpeed, 1 as OpenResty and 1 sends no Server header.

To rule out my own connection as the cause, I ran 6 of the 19 through a second checker hosted on Vercel, so a different network. All 6 returned the same status codes.

n is 79 reachable sites in one city and one trade, so don't read 24% as a rate for small businesses in general. The useful part is that two unrelated samples on two continents show the same shape.

Find the layer before you change anything

A request passes through up to four places that can refuse it: a CDN or WAF at the edge, a reverse proxy, the web server, and the application (a CMS security plugin, for example). The response usually tells you which one answered. Run this against your own site:

URL=https://example.com/
probe() {
  printf '%-9s ' "$1"
  curl -s -L -o /dev/null -D - -A "$2" "$URL" | tr -d '\r' | grep -i -E '^(HTTP/|server:|cf-mitigated:)' | paste -sd' ' -
}
probe browser   'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128 Safari/537.36'
probe GPTBot    'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.1; +https://openai.com/gptbot)'
probe ClaudeBot 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)'
probe made-up   'Mozilla/5.0 (compatible; ExampleFetchBot/1.0; +https://example.com/bot)'
URL=${URL}robots.txt probe robots-as-GPTBot 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.1; +https://openai.com/gptbot)'

One of the nginx sites from the sample prints this:

browser   HTTP/1.1 200 OK Server: nginx
GPTBot    HTTP/1.1 403 Forbidden Server: nginx
ClaudeBot HTTP/1.1 403 Forbidden Server: nginx
made-up   HTTP/1.1 200 OK Server: nginx
robots-as-GPTBot HTTP/1.1 403 Forbidden Server: nginx

How to read it:

What you see on the 403	Who refused
`Server: cloudflare`, page titled "Attention Required! \| Cloudflare" with "Sorry, you have been blocked"	A Cloudflare block: a WAF custom rule, a managed rule, or the AI bot setting
`Server: cloudflare` plus `cf-mitigated: challenge`	A Cloudflare challenge (Bot Fight Mode or a rule with a challenge action). A crawler can't solve it, so it's a refusal in practice
`Server: nginx`, `Apache` or `LiteSpeed`, small plain error page	The web server or something behind it. `/robots.txt` refused too means the rule runs before any per-path logic, typically a server-wide user-agent match or an `.htaccess` rule
200 from the server, but a CMS error or login page in the body	An application plugin

The Server header names the outermost server that answered, not necessarily the one that made the decision. One of the "nginx" sites returns Apache's stock "You don't have permission to access this resource" page. That's the usual shared-hosting setup: nginx in front, Apache behind it, and the rule sits in Apache config or .htaccess.

Intentional, or two layers disagreeing?

Blocking AI crawlers is a reasonable choice and plenty of owners make it on purpose. What you're checking for is whether the site says one thing and does another. Three questions settle most cases:

What does robots.txt say about this crawler? If it disallows the bot and the server refuses it, the layers agree. That was 1 of the 19. If robots.txt allows it and the server refuses, someone made one decision and something else made the opposite one.
Does a made-up bot get through? If ExampleFetchBot gets 200 and GPTBot gets 403, the rule names AI crawlers. Someone or something wrote that list. If the made-up bot also gets 403, you're looking at a generic bot filter that probably never had AI crawlers in mind.
Who controls the layer that answered? Shared hosts and site builders ship their own user-agent blocklists, and security plugins add them on update. The owner of the site often never saw the rule. Ask the host, or search the config: grep -rn -i 'gptbot\|claudebot' /etc/nginx/ .htaccess.

If you run the server, the access log answers the question for the real crawler, not just my imitation of it:

grep -E 'GPTBot|ClaudeBot|PerplexityBot' /var/log/nginx/access.log \
  | awk '{ match($0, /GPTBot|ClaudeBot|PerplexityBot/); print substr($0, RSTART, RLENGTH), $9 }' \
  | sort | uniq -c

That prints a count per crawler and status code. It assumes the default combined log format, where field 9 is the status.

Fixes, layer by layer

Decide what you want first, then write it in robots.txt, because that's the layer the crawlers read and the one you can show to anyone who asks. To let the three in:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: PerplexityBot
Allow: /

User-agent: *
Disallow: /wp-admin/

To keep training crawlers out but stay visible to Perplexity's search crawler, swap the first group for User-agent: GPTBot, User-agent: ClaudeBot, Disallow: /. Then make every other layer agree with that file.

Cloudflare

Three settings can produce these refusals, and they behave differently.

Bot Fight Mode (Free plan, Security settings). Cloudflare's docs say it "cannot be bypassed or skipped using WAF custom rules or Page Rules". On the Free plan you either leave it on and accept the challenges, or turn it off.
Super Bot Fight Mode (Pro, Business and Enterprise) can be skipped. Add a WAF custom rule with the Skip action, pick Super Bot Fight Mode as the thing to skip, and use an expression that only matches crawlers Cloudflare has verified:

(cf.verified_bot_category in {"AI Crawler" "AI Search" "AI Assistant"})

Put it above any rule that blocks. Cloudflare verifies these bots by where the request comes from, not by the user agent string, so this doesn't open the door to anyone who copies GPTBot's user agent.

AI bot policies (Security settings, "Configure AI bot policies", which replaced the older "Block AI bots" toggle). Per Cloudflare's docs, from 2026-09-15 new domains block Training and Agent bots on pages that show ads by default, and "mixed-purpose crawlers that combine Search and Training" are blocked by every configuration that blocks training. If you added a domain recently and never opened this page, check it.

If you'd rather write your own block for impostors, this expression matches requests that claim to be GPTBot but don't come from a verified bot:

(http.user_agent contains "GPTBot" and not cf.client.bot)

I checked both expressions with Cloudflare's open-source rules parser (wirefilter). Two deliberately broken versions, one with a missing closing brace and one with a typo in and, fail with a parse error, so the check does reject bad input. The parser checks syntax against field types I declared. The dashboard's expression editor is still the final word.

nginx

The accidental version usually looks like this, copied from a "block bad bots" list:

if ($http_user_agent ~* "(bot|crawl|spider)") { return 403; }

At server level it refuses every crawler, /robots.txt included. I ran it locally and GPTBot, ClaudeBot and a made-up scraper all got 403 on both / and /robots.txt. This replacement lets the named AI crawlers through, keeps the generic filter for everything else, and always serves robots.txt:

map $http_user_agent $block_ua {
    default                                            0;
    ~*(GPTBot|ClaudeBot|PerplexityBot|OAI-SearchBot)   0;
    ~*(bot|crawl|spider|scrape)                        1;
}

server {
    listen 80;
    root /var/www/html;

    location = /robots.txt { }

    location / {
        if ($block_ua) { return 403; }
    }
}

map goes in the http block. Regex entries are checked in order and the first match wins, so the allow line has to come before the generic one. I tested this with nginx 1.28.0: nginx -t passes, and against a running instance the browser, GPTBot and ClaudeBot got 200 while SomeScraperBot/2.0 got 403 on / and 200 on /robots.txt. Deleting the semicolon on the default line makes nginx -t fail, which is how I know the test isn't just saying yes. If you want the crawlers blocked instead, move them to the 1 side and keep the robots.txt location, so they can still read the file that tells them so.

Shared hosting, Apache and plugins

When the 403 comes from a host you don't control, look for the crawler names in .htaccess (a RewriteCond %{HTTP_USER_AGENT} line followed by a rule with [F]), then in your security plugin's bot or firewall settings. If you find nothing, ask the host's support whether they block AI crawlers by user agent at the server level. On 12 of the 19 sites above even robots.txt is refused, which suggests a rule that runs before the site's own files are consulted.

What this check can't tell you

My probes send the crawler's user agent from an ordinary connection. The real crawlers come from their operators' published IP ranges. Origin rules that match on the user agent (the 14 nginx, Apache and LiteSpeed cases) treat both the same way, so those results should hold for the real crawler. Cloudflare can tell the two apart. Its AI bot setting covers verified crawlers "as well as a number of unverified bots that behave similarly", so a 403 to my imitation may be Cloudflare stopping an impostor while the real GPTBot gets through. For the 5 Cloudflare cases, Security Events in the dashboard or the origin access log is the only reliable answer.

A 200 also doesn't promise anything shows up in AI answers. This only removes one reason a site might be missing.

If you want to test a site without the shell, the free AI crawler check runs the same robots.txt-versus-server comparison for GPTBot, ClaudeBot and PerplexityBot and stores nothing. If you'd like a person to go through one site end to end, the paid mini audit is on that same page. It checks AI-crawler access and machine-readability and includes paste-ready fixes (a corrected robots.txt, CDN/WAF steps for bot 403s, meta and canonical fixes, JSON-LD and llms.txt skeletons), all marked review before deploying. It queries no AI engine and measures no citations or rankings.

Correction 2026-10-06: an earlier version said the paid audit measures what AI engines answer; it checks crawler access and machine-readability only.

Your robots.txt allows GPTBot. Your server may still answer 403.

Mahiro Hirakawa — Sun, 04 Oct 2026 06:32:35 +0000

I was checking small business websites for how readable they are to AI answer engines, and kept running into the same thing: robots.txt allows GPTBot and ClaudeBot, and the server answers them with 403 anyway.

Nobody reading robots.txt would notice. A robots.txt checker would report "AI crawlers allowed". The block sits one layer lower, in the web server or a security plugin, and it matches on the user-agent string.

What I measured

On 2026-10-04 I probed 26 websites of small Japanese tourism businesses (guesthouses, cooking classes, kimono rental, pottery and tea-ceremony studios). I found them through search and AI-answer queries for those categories, so this is a convenience sample, not a random one. Each site got one request per user agent, from one residential connection in Japan.

3 of the 26 did not answer at all (timeout or connection error), which leaves 23.
5 of those 23 returned 403 to both the GPTBot and the ClaudeBot user agent while a browser user agent got 200. All 5 have a robots.txt that allows both crawlers.
1 more returned 403 to ClaudeBot only. That site has no readable robots.txt.
The same 5 sites returned 200 to OAI-SearchBot and PerplexityBot. The rule targets the training crawlers by name, not every bot.
4 of the 5 send Server: nginx, and on those 4 even /robots.txt returns 403 when requested as GPTBot. The crawler is refused the file that says it is welcome.

So in this sample, 6 of 23 reachable sites (about a quarter) refuse at least one of the two training crawlers at the server, and none of them say so in robots.txt. I would not stretch that ratio to "small business sites" in general. n is 23 and the sample leans toward Japanese tourism. The pattern is the useful part, and you can check your own site in under a minute.

Test it yourself with curl

These are the user-agent strings the crawlers publish. Replace https://example.com/ with your own site.

URL=https://example.com/
BROWSER='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128 Safari/537.36'
GPTBOT='Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.1; +https://openai.com/gptbot)'
CLAUDEBOT='Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)'

curl -s -o /dev/null -w 'browser   %{http_code}\n' -A "$BROWSER"   "$URL"
curl -s -o /dev/null -w 'GPTBot    %{http_code}\n' -A "$GPTBOT"    "$URL"
curl -s -o /dev/null -w 'ClaudeBot %{http_code}\n' -A "$CLAUDEBOT" "$URL"
curl -s -o /dev/null -w 'robots.txt as GPTBot %{http_code}\n' -A "$GPTBOT" "${URL}robots.txt"

On glovrex.com, a site I run (hosted on Vercel, robots.txt allows everyone, no user-agent rules on the server), all four lines print 200:

browser   200
GPTBot    200
ClaudeBot 200
robots.txt as GPTBot 200

On the four nginx sites from the sample the output was:

browser   200
GPTBot    403
ClaudeBot 403
robots.txt as GPTBot 403

If you see that second shape, also run curl -s "${URL}robots.txt" to read what your robots.txt actually says. If it allows the crawler and the server refuses it, the two layers disagree, and the server wins. A 404 on the robots.txt line is fine, it only means you have no robots.txt. Sending just -A GPTBot gave the same 403 on all 4 nginx sites, which points to a plain substring match on the user agent.

Why it matters for AI answers

There are two ways an AI assistant ends up saying something about a business. It can search the web at question time, using crawlers like OAI-SearchBot or Claude-SearchBot. Or it can answer from what the model already absorbed in training, which is where GPTBot and ClaudeBot come in.

The sites above still let the search-time crawlers in, so a browsing assistant can read them. What they lose is the second path. When a model answers without searching, it knows the business only through other pages that mention it, usually booking platforms, listicles and review sites. In one sample check I ran on a guesthouse from this group, an AI answer about the business cited only booking platforms, and the prices it quoted were the platforms' prices, not the ones on the official site. That is one answer from one engine, not proof of cause, but it is what you would expect if the official site is missing from training data.

Caveats before you change anything

Blocking training crawlers is a legitimate choice. Plenty of site owners do it on purpose, and the crawler operators document how to opt out for exactly that reason. The problem I am pointing at is only the case where nobody chose it: robots.txt was written to allow the crawlers, and a hosting default or a security plugin overrode it without anyone noticing.

My probe sends the published user-agent string from an ordinary IP. The real crawlers come from their operators' IP ranges, and some setups (Cloudflare's bot settings, for example) treat a verified crawler differently from someone who just copied its user agent. One of the 5 sites in the sample is behind Cloudflare, so for that one my result may not match what the real crawler sees. The way to be sure is your server's access log: search it for GPTBot and ClaudeBot and look at the status codes.

A 200 from your server is also not a promise that anything will show up in AI answers. Off-site reputation still decides most of who gets named. This check only removes one reason you might be missing.

If you want to go further

geo-checker is a free MIT tool I wrote that scores a page for structured data, headings and robots.txt rules. It reads robots.txt only, so it would not have caught any of the 6 sites above. The repository is archived, so take it as it is. If you would rather have a person run the full check, I do a paid GEO mini audit for one site: AI-crawler access and machine-readability, plus paste-ready fixes (a corrected robots.txt, CDN/WAF steps for bot 403s, meta and canonical fixes, JSON-LD and llms.txt skeletons), all marked review before deploying. It queries no AI engine and measures no citations or rankings.

Follow-up: how to find which layer (CDN, web server or plugin) is refusing the crawler, and how to fix each one.

Correction 2026-10-06: an earlier version said the paid audit measures what AI engines answer; it checks crawler access and machine-readability only.

A six-line edit made fifteen units stale. The freshness check was working correctly.

Mahiro Hirakawa — Mon, 14 Sep 2026 19:32:21 +0000

Every unit of this project carries a record of its last verification, and each record pins the configuration it was checked against by hashing it. If the configuration changes, the pin no longer reproduces and the record goes stale, which is exactly what you want: a verification result that quietly outlives its inputs is worse than no result.

One afternoon I added a six-line block to that configuration file. It described a new component and touched nothing else.

Fifteen units went stale. Every one of them owed a full re-run of its checks and a fresh record appended afterwards, which is most of an hour of machine time and a boundary ceremony, for six lines that concerned one of them.

Nothing was broken. The check did precisely what it promised.

The pin was over the whole file

pin = sha256(entire configuration file)

Any byte anywhere changes that hash. So the pin does not mean "the configuration this unit depends on is unchanged". It means "nobody has edited this file at all", which is a far stronger statement and one that becomes false constantly in a file every component writes to.

The gap between what a pin means and what everyone reads it to mean is the whole bug, and it is invisible while the file is small.

what the pin says	what everyone reads it as
this file is byte-identical	my inputs are unchanged
any edit invalidates me	edits to my section invalidate me

The failure mode is people, not machines

An hour of machine time is affordable. What is not affordable is what it does to behaviour.

When correcting a six-line block costs fifteen re-runs, three things start happening. I watched all three:

edits get batched until there are enough to justify the ceremony, so the configuration is wrong for longer
the staleness line gets read past, because it is almost always red for reasons unrelated to what you are doing
someone proposes re-pinning without re-running, which is the point where the check stops meaning anything

A check whose cost is out of proportion to the change gets routed around. It does not get fixed, because it is not broken.

The repair is a declared partition

The pin became the hash of the unit's own block plus the sections genuinely shared between units, rather than the whole file:

pin = sha256(block for this unit) + sha256(shared sections)

Two properties matter more than the saving. The partition is declared, as rows in the file itself rather than inferred by a parser guessing at section boundaries, so it is readable and arguable. And it is recomputable by hand, since the recipe is printed next to the hash and anyone can rerun it with sha256sum.

Now an edit to one unit's block stales one unit. An edit to a shared section stales everything, and that is correct, because everything does depend on it.

What I would check earlier next time

There is a cheap diagnostic I did not run for months. Take the last N edits to the file and ask, for each, how many units the pin invalidated and how many actually depended on the change.

edits sampled          20
units invalidated      15 each time
units actually affected 1, 1, 1, 2, 1, 15, 1, ...

One row genuinely needed all fifteen. The rest were a partition problem visible in five minutes of arithmetic over the git log, and the ratio is the argument for fixing it.

Two things I keep

Hash granularity is an interface, not an implementation detail. Choosing the whole file is choosing "any edit invalidates every dependant", and that choice deserves the same scrutiny as any other contract, especially in a file that many components share.

"Correct but disproportionate" is a defect class of its own. It never shows up as a failing check, so nothing surfaces it. It shows up as people batching their edits, skimming past a red line, and eventually proposing to weaken the check, and by then the conversation is about the check rather than about its granularity.

The OOM killer stopped my acceptance check four times in two days. It was rebuilding the project to read a number.

Mahiro Hirakawa — Mon, 14 Sep 2026 18:58:29 +0000

There is a step at the end of each work session that decides whether each unit of the project is accepted. It gathers, for every unit, whether its test suite passed, whether its checks are current, whether its documentation was regenerated, and prints one line per unit.

It kept dying.

acceptance read: killed

Four times in two days. No error, no stack, just gone. The machine's memory limit, with nothing else running. On the fourth kill I stopped restarting it and read what it does.

It was compiling the project. To find out whether the tests passed, it ran them.

The word "read" was doing a lot of work

The step is called a read, and I had thought of it as one for months. It sounded like gathering. What it actually did, for each of fifteen units, was invoke the build and wait.

That is not a read. It is a second full build whose output nobody looks at except to extract a boolean.

The memory limit was not the problem. It was the only thing telling me anything. A build is the most expensive operation in the repository, and the acceptance step ran fifteen of them in sequence, on a machine that can comfortably run one.

Why re-running looks like the rigorous choice

Here is the argument I would have made for the old design, and it is not stupid:

A recorded result is a claim about the past. The tree may have changed since. Running it now is the only way to know it holds now.

That is correct in general and wrong here, because of what the recording already is. Every control run appends a block to a permanent record, and that block carries the commit it ran at plus its verbatim output:

control block   unit=projection  commit=26ef4cc  OK_ALL controls=13  run=2026-09-13

So the record is not someone's summary. It is the control's own print, pinned to a commit. Asking "did this pass at this commit" is answerable from it exactly, and re-running the build answers the same question by doing the same work again.

step	question	how it should answer
the control	does this hold right now	run it, append the print
the acceptance read	what do the controls say at this commit	read the record

Two steps, two questions. The acceptance step had been answering the first question while being named for the second, so it duplicated the control's work and added nothing except a way to run out of memory.

The staleness question, which is the real one

"What if the record is from an older commit?" is the objection that matters, and it is answerable without building anything:

ACCEPT_GATE  current=S  next=face  recorded=9/15  stale=0

The read compares each block's commit against the commit being accepted. If they differ, that unit is stale and says so. Staleness becomes a printed number rather than something the design tries to make impossible by brute force.

That is the useful move: the cost of re-deriving was buying a property that a comparison provides for free. Freshness is a question about two identifiers. It does not need a compiler.

A general form I keep coming back to

Any step whose name is a verb of observation, read, check, report, gather, audit, that takes minutes and consumes gigabytes, is probably not observing. Something inside it is re-deriving a fact that already exists somewhere.

The tell is resource usage that does not match the verb. An observation's cost should scale with how much there is to observe, not with how expensive the thing observed was to produce.

Two things I keep

A read that builds is not a read. Separating "produce the fact" from "collect the facts" is not tidiness; it is what makes the collecting step cheap enough to run at every boundary instead of once a day when someone is brave.

When a step keeps being killed, read it before restarting it. I restarted three times and reduced parallelism twice, both of which are treatments for a resource problem. The fourth kill was the first time I asked what the step was actually doing, and the answer took two minutes to find.

My coverage check did arithmetic instead of looking. All 23 pointers were wrong.

Mahiro Hirakawa — Mon, 14 Sep 2026 18:03:19 +0000

A table in one document points into another. Each of its 24 rows carries a source pointer saying where the thing it describes actually lives, so a reader can go and check.

The coverage check for that table reported complete, always. Here is what it did:

rows in the table        24
rows carrying a pointer  23
rows that are a vessel    1
24 - 23 - 1 = 0 uncovered  ->  complete

That is arithmetic over counts. It never opens the file the pointers point into. A pointer can name a line that does not exist, or the wrong line, or a line in a file that was deleted, and the subtraction still comes out to zero.

When something finally resolved them, all 23 were wrong.

Wrong by one or two lines, which is the informative part

Not wrong by hundreds. Every pointer was off by one or two lines against the target.

That number tells you which failure this is. A pointer that drifts as a file grows ends up off by however many lines were inserted above it, which after a year is a large and varied number. Off by one or two, uniformly, across every row, is not drift. It is an off-by-one in whatever produced them, present from the first commit.

symptom	drift	never worked
offsets	large, varied	small, uniform
when introduced	gradually	at creation
repair	re-point and add a freshness check	fix the generator, then re-point
how it reads in a log	"these have gone stale"	"these have gone stale"

The two look identical in the report. They want different repairs, and only the distribution of the offsets tells you which one you have.

The same defect in four tools in one week

While fixing it I found that four separate readers were joining two tables the same way: by row order.

# join by position: row 7 here describes row 7 there
for (i = 0; i < left.length; i++) check(left[i], right[i])

Position is a key that every insertion invalidates. Add a row to one side and every pair after it is silently wrong, which is exactly what had happened. Nothing errors, because both sides still have rows at those indices.

All four became keyed joins, matching on an identifier that travels with the row:

# join by key: an unmatched row on either side is a named finding
for (row of left) check(row, right.byId[row.id] ?? MISSING)

The difference that matters is not correctness in the happy case. It is that a keyed join can report MISSING, and a positional join has no way to express it. A positional join cannot fail; it can only be wrong.

What replaced the subtraction

Each row now carries the pointer plus a verbatim quote of what it points at. Coverage is the count of rows whose quote still matches the bytes at that location, which is a question about the world rather than about the table's own shape.

Retired rows moved out of the denominator entirely, as pointer lines that no counter reads.

The satisfying part: after the repair, no count moved. The table had said complete and, once every pointer was actually resolved, it was complete. Every one of those 23 pointers had been wrong, and the number the check reported was right. It was right for no reason.

Two things I keep

A coverage number computed from counts is a statement about your table, not about your code. If the check never opens the thing being pointed at, it cannot go red for the reason you built it, and the correct total it prints is a coincidence you should not spend.

Read the distribution of the errors, not just how many. "All 23 are stale" and "all 23 are off by one" are the same count and different bugs. One asks for a freshness check. The other asks you to fix the thing that wrote them, or you will re-point 23 rows and generate 23 more.

DEV Community: Mahiro Hirakawa

日本のサイト248件中23件、robots.txtは許可でもAIクローラーには403

先に限界(2026-10-07時点)

対象と測り方

数字

23件のrobots.txtの状態

断っている対象で3つに分かれる

ドメインの種類別

分野ごとの例

自分のサイトを確かめる

15 merged IME fixes: the recurring ways Japanese typing breaks web apps

Background in one paragraph

Class 1: Enter sends the message mid-composition

Class 2: Enter commits a partial value (tags, list items, renames, validated fields)

Class 3: Enter selects or searches on unconfirmed text

Class 4: keyCode 229 where isComposing is false

Class 5: composition flags that depend on event order

A bonus class: Escape during composition

Test your own app in 5 minutes

154 of 529 top homepages have no canonical tag. 15 point to a URL that redirects straight back.

Limits first (measured 2026-10-06/07)

The canonical numbers

What the 58 targets answered

The canonical depends on where you are

og:url versus canonical

The hreflang numbers

Return links

A checklist that catches all of the above

Your IMAP script stopped logging in: XOAUTH2 for Gmail and Microsoft 365

Why the password stopped working

What XOAUTH2 sends

Where the token comes from, and how long it lives

The demo: a local fake server, a fake token

Config: a rule file and an env file

First run: the certificate is checked

Dry run

A bad token

The old password path, for comparison

Digest file and webhook

Safety notes

Sources

0 of 550 top homepages send noai. 21 send Bing's noarchive.

Limits first

What each lever reaches

The numbers

Meta versus header

Against robots.txt

Three sites, three setups

Write one policy

111 of the top 1,000 sites block GPTBot. 35 never mention the bot behind ChatGPT search.

Limits first

What each token controls

The numbers

Mistakes in real files

Blocking training and, maybe by accident, leaving answers open

Blocking a token nobody sends anymore

Naming a bot replaces the * group

Tokens that match no crawler

Blank lines inside a group

A rule above every group

Decide your own policy

246 of the top 1,000 sites answer /llms.txt with 200. Only 114 serve the file.

What I measured, and the limits first

The headline numbers

Three ways a naive check lies

1. A 200 that is an HTML page

2. A text file that isn't an llms.txt

3. Dead links, and HEAD requests that lie about them

Check your own

What I'd change on a site today

Your CDN can overrule robots.txt: finding the layer that refuses AI crawlers

Amsterdam bike shops, measured on 2026-10-04

Find the layer before you change anything

Intentional, or two layers disagreeing?

Fixes, layer by layer

Cloudflare

nginx

Shared hosting, Apache and plugins

What this check can't tell you

Your robots.txt allows GPTBot. Your server may still answer 403.

Class 4: keyCode 229 where `isComposing` is false

`og:url` versus canonical

Naming a bot replaces the `*` group