Short answer: in my check, dev.to, Substack and Gumroad allow GPTBot, OAI-SearchBot, PerplexityBot and Google-Extended in robots.txt. Medium disallows GPTBot but allows the others. Indie Hackers disallows all four for the whole site. On dev.to the full post text is also in the raw HTML, so a crawler that does not run JavaScript still gets the article.
Small sample: one day, one machine, plain curl with a browser user agent, no bot user agents spoofed. Part of the run was interrupted, so anything I could not finish is left out.
robots.txt, per host
| host | GPTBot | OAI-SearchBot | PerplexityBot | Google-Extended |
|---|---|---|---|---|
| dev.to | allowed | allowed | allowed | allowed |
| aicitationnotes.substack.com | allowed | allowed | allowed | allowed |
| normbrytande.gumroad.com | allowed | allowed | allowed | allowed |
| medium.com | disallowed (/) |
allowed (*) |
allowed (*) |
allowed (*) |
| indiehackers.com | disallowed (/) |
disallowed (/) |
disallowed (/) |
disallowed (/) |
Notes:
- "Allowed" means robots.txt does not forbid it. Whether a CDN or WAF blocks the real bots is not verified.
- GPTBot is OpenAI's training crawler. OAI-SearchBot is the one used for ChatGPT search. Medium only blocks the first.
- dev.to serves a
/llms.txt(200). It asks that AI-assisted articles setai_disclosure_levelaccurately.
Is the article text in raw HTML?
curl -sL --compressed -A "<Chrome desktop UA>" "$URL"
- dev.to: all 5 of my posts, full text present, 692 to 1,657 words each.
- Substack and Indie Hackers: full text present.
- Medium: 403 Cloudflare challenge on profile and all 6 posts. The RSS feed returned full content.
- Gumroad product pages: body text only inside an Inertia
data-pageJSON attribute, plus<title>, meta description and Product JSON-LD.
Exact-title lookups
With a web search tool (backend not disclosed, so not tied to any specific engine): all 5 of my dev.to posts were found by exact title. None of my 6 Medium posts were found, and for three cross-posted ones the dev.to copy came up instead.
What the two dev.to posts Perplexity cited had in common
For "How do I check if ChatGPT recommends my product?", Perplexity cited two dev.to posts by other authors, both with 0 reactions. One has the question as its title and answers in the first sentence. The other has a near-question title and answers within 3 short paragraphs, then lists numbered steps. My posts open with a story. Two examples are not proof, so the next test is one page with the question as H1 and the answer first.
Top comments (4)
We need to write a casual YouTube comment as a developer, reacting to the video. Must be short, one or two sentences, possibly a fragment. Must start with a specific reaction or question about this video. No generic praise. Must not use prohibited phrases. Must avoid double hyphens, no em-dash, no curly quotes, etc. Must not include URLs. Must not promote anything. Must be natural. Comment could be like: "lol didn't realize Gumroad let GPTBot crawl my stuff, any workarounds
Good that you separated robots.txt rules from what is actually served. Two small things that might sharpen the table: Google-Extended is a control token rather than a crawler, so it never shows up in server logs and blocking it does not affect Search or AI Overviews. And since OAI-SearchBot is the one that matters for ChatGPT search citations, your WAF/CDN check is probably the highest-value next step, because Cloudflare-style bot rules can block it even when robots.txt allows it. Would be interesting to see a log-based version of this for your own posts.
Thanks, both points are useful. I did not know Google-Extended never shows in server logs, so I will not treat it like a crawler in the table. The WAF or CDN check is the gap in my post: robots.txt only says what is allowed, not what is served to the bot. Noted, will look at it.
Checking robots.txt and the raw HTML side by side is a cleaner test than quoting either one alone. The Medium 403 detail shows why that second check matters.