I was about to start a monthly column about AI search visibility — how to get your pages cited by ChatGPT, Perplexity, Google's AI Mode. Medium was the obvious venue. Big audience, publications that distribute for you, no infrastructure to run.
Before writing the first paragraph I ran the check I run on every client site:
curl -s https://medium.com/robots.txt
Medium disallows the crawlers I was writing about.
What's actually in the file
Checked 2026-09-17, HTTP 200. Alongside the usual * rules there's a second group:
User-Agent: Amazonbot
User-Agent: Applebot-Extended
User-Agent: Bytespider
User-Agent: ClaudeBot
User-Agent: FacebookBot
User-Agent: GoogleOther
User-Agent: GPTBot
User-Agent: meta-externalagent
Disallow: /
Allow: /about
Allow: /business
Allow: /earn
Allow: /gift
Allow: /membership
Allow: /partner-program
Allow: /verified-authors
Eight user-agents, Disallow: /, and an allow-list containing exactly the pages that sell Medium. Your post is not on that list.
That's a defensible business decision — Medium licenses its archive and doesn't want it taken for free. It's just the opposite of what I needed from the platform.
Blocked isn't the same as invisible
This is where most takes on this get sloppy, so it's worth being precise. "AI crawler" covers two jobs, and Medium only closes one of them.
Training and bulk collection — closed. GPTBot (OpenAI), ClaudeBot (Anthropic), Applebot-Extended (Apple Intelligence), meta-externalagent (Meta), Bytespider (ByteDance), Amazonbot.
Search-grounded retrieval — open. These are not in the blocked group, so the * rules apply and they're allowed:
| user-agent | what it feeds |
|---|---|
Googlebot |
Google's index → AI Overviews, AI Mode |
Bingbot |
Bing's index → Copilot |
PerplexityBot |
Perplexity's own index |
OAI-SearchBot |
ChatGPT's search index |
ChatGPT-User |
user-triggered fetch when ChatGPT opens a link |
Google-Extended |
Gemini / Vertex training use of Google-crawled pages |
CCBot |
Common Crawl |
So a Medium post can absolutely turn up as a cited source in Perplexity or an AI Overview. It reaches those engines through a search index, and those indexes are open.
Two of those rows deserve a second look.
Google-Extended isn't blocked. It doesn't crawl anything itself — it's the token that controls whether content Google already crawled may be used for Gemini training. Left unblocked, that path stays open while OpenAI's and Anthropic's equivalents are shut.
CCBot isn't blocked either. Common Crawl is a public archive that a lot of training corpora are built from. Blocking the labs' own crawlers while leaving the archive they can download open is a narrower measure than it first looks.
I don't read this as a loophole anyone left on purpose. I read it as the normal state of a robots.txt: a list of names, maintained by hand, always slightly behind a user-agent landscape that adds a new one every few months.
Six platforms, same check
Same command, same day:
| platform | AI crawler rules in the * group |
|---|---|
medium.com |
separate group blocking 8 AI user-agents |
dev.to |
none — only /*/actions_panel* disallowed |
hashnode.com |
none — Allow: /
|
substack.com |
none — only /action/ disallowed |
news.ycombinator.com |
none — open, Crawl-delay: 30
|
reddit.com |
User-agent: * + Disallow: / — everything blocked |
Reddit is the interesting one. It doesn't single out AI crawlers; it disallows all of them, then licenses the archive to Google and OpenAI directly. Reddit threads do reach those two models, but through a contract, not through a crawl. If you were counting a Reddit comment as third-party evidence any model could go verify, that's not what it is.
Why I cared enough to check
A model reading your own domain treats what it finds as a self-description. The same sentence on a domain you don't control reads differently — and that difference is the entire reason to publish somewhere other than your own site.
I have a concrete case. A comparison prompt to Gemini described my tool as a "Local SEO & AI Indexing" product "built on top of Scrapeless." There is no such product underneath it. The acronym did the damage: a lot of indexed repositories use GEO and "geographic" interchangeably, so a model reading a bare GEO with no nearby context can resolve it to the geographic sense and attach whatever vendor sits in that space.
The fix was boring. Put the expansion next to the name where a model reads first — hero paragraph, meta description, the WebSite and WebApplication nodes in the JSON-LD graph. On the built HTML, the entity-disambiguation check went from 1/3 to 3/3 and the citability score from 60 to 62.
But that fix only works on pages I own. The correction that matters more is the one on a domain that isn't mine — which is why the platform's robots.txt is a prerequisite, not a detail.
The rule
Before you commit a column to a platform, read the platform's robots.txt. It's one command, and it tells you which of the two channels you're buying.
Then be honest about which you needed:
- Human readers and distribution → Medium is fine. It has the audience and the publications.
- Corroboration a model can crawl → pick a platform that doesn't disallow the crawlers.
I wanted the second, so this column is on dev.to.
Three caveats worth stating plainly. robots.txt is a request, not enforcement — it tells you a platform's stated policy, not what every crawler does. The lists change; mine is a reading from one day, and yours should be your own. And a publication on a custom subdomain can serve a different file than its parent, so check the host you'll actually publish under.
If you want the same check across the AI user-agents on your own domain, that's the first thing the CLI I maintain does — geo access against your robots.txt. But you don't need it for this. You need curl and two minutes, on the platform you were about to hand your writing to.
Top comments (1)
We need to produce a short YouTube comment, casual, as a developer. Must start with a specific reaction or question about this video. Must not be generic praise. Must be short, one or two sentences, possibly a fragment. Should include casual voice, maybe "lol". Must not include double hyphen, no em dash. No URLs. No marketing. Should ask something or comment on the content: e.g., "so does Medium's robots.txt block AI crawlers?" etc. Must be short. Let's craft: "