Skip to content
Navigation menu
Search
Powered by Algolia
Search
Log in
Create account
DEV Community
Close
← All Trends
AI Crawler Blocking vs. Traffic Acquisition in SEO
20 posts in this trend in the last 7 days
•
Active 2 days ago
Can ChatGPT, Claude and Perplexity actually crawl your site? A 10-minute check
Віталік Бакута
Віталік Бакута
Віталік Бакута
Follow
Nov 13
Can ChatGPT, Claude and Perplexity actually crawl your site? A 10-minute check
#
seo
#
ai
#
webdev
#
tutorial
Comments
2
comments
4 min read
Moderation
I measured 528 sites' robots.txt: two thirds blocking GPTBot also block the crawler that cites them
orenhacham-lab
orenhacham-lab
orenhacham-lab
Follow
Nov 8
I measured 528 sites' robots.txt: two thirds blocking GPTBot also block the crawler that cites them
#
seo
#
ai
#
webdev
#
opensource
1
reaction
Comments
2
comments
5 min read
Moderation
Can ChatGPT access your website? A 2-minute check (robots.txt, firewall, HTML)
Ali Baizhanov
Ali Baizhanov
Ali Baizhanov
Follow
Nov 9
Can ChatGPT access your website? A 2-minute check (robots.txt, firewall, HTML)
#
seo
#
ai
#
webdev
#
beginners
Comments
1
comment
3 min read
Moderation
Two thirds of sites blocking GPTBot are also blocking the crawler that brings them traffic
orenhacham-lab
orenhacham-lab
orenhacham-lab
Follow
Nov 7
Two thirds of sites blocking GPTBot are also blocking the crawler that brings them traffic
#
seo
#
webdev
#
ai
Comments
Add Comment
3 min read
Moderation
Can AI Crawlers Read Your Website? robots.txt Is Only Half the Answer
Himanshu Patel
Himanshu Patel
Himanshu Patel
Follow
for
Codevibe Technology
Nov 13
Can AI Crawlers Read Your Website? robots.txt Is Only Half the Answer
#
seo
#
ai
#
webdev
#
opensource
Comments
2
comments
6 min read
Moderation
I checked 30 top websites for AI-crawler readiness. The ones ChatGPT cites most scored worst.
Ke Gao
Ke Gao
Ke Gao
Follow
Nov 12
I checked 30 top websites for AI-crawler readiness. The ones ChatGPT cites most scored worst.
#
ai
#
seo
#
webdev
#
marketing
Comments
1
comment
4 min read
Moderation
A robots.txt that allows GPTBot does not mean GPTBot can reach your site
SeenLens
SeenLens
SeenLens
Follow
Nov 7
A robots.txt that allows GPTBot does not mean GPTBot can reach your site
#
seo
#
webdev
#
ai
#
tutorial
Comments
Add Comment
5 min read
Moderation
How to Check Which AI Crawlers Your robots.txt Actually Blocks
Daniel Di Cerbo
Daniel Di Cerbo
Daniel Di Cerbo
Follow
Nov 13
How to Check Which AI Crawlers Your robots.txt Actually Blocks
#
seo
#
webdev
#
ai
#
tutorial
Comments
Add Comment
4 min read
Moderation
How to show up in ChatGPT: first check that it can read your site (robots.txt, in 5 minutes)
Luziq
Luziq
Luziq
Follow
for
Luziq
Nov 7
How to show up in ChatGPT: first check that it can read your site (robots.txt, in 5 minutes)
#
ai
#
chatgpt
#
seo
#
webdev
Comments
2
comments
2 min read
Moderation
Is your site invisible to ChatGPT Search? Five checks you can run in a minute
Dejun (Alice) Chen
Dejun (Alice) Chen
Dejun (Alice) Chen
Follow
Nov 10
Is your site invisible to ChatGPT Search? Five checks you can run in a minute
#
seo
#
ai
#
mcp
#
webdev
Comments
Add Comment
3 min read
Moderation
Three quick checks before you expect AI search to quote your page
S. Trent
S. Trent
S. Trent
Follow
Nov 13
Three quick checks before you expect AI search to quote your page
#
seo
#
webdev
#
ai
#
tutorial
Comments
1
comment
2 min read
Moderation
We checked robots.txt on 1,069 top websites: who blocks AI crawlers, and who still allows ChatGPT search
Vimal Kumar Jain
Vimal Kumar Jain
Vimal Kumar Jain
Follow
Nov 9
We checked robots.txt on 1,069 top websites: who blocks AI crawlers, and who still allows ChatGPT search
#
seo
#
ai
#
webdev
#
datascience
Comments
Add Comment
9 min read
Moderation
Auditing Your Website for AI Search Engines in 2026: TTFB, JSON-LD, robots.txt, and /llms.txt
TCO Audit
TCO Audit
TCO Audit
Follow
Nov 12
Auditing Your Website for AI Search Engines in 2026: TTFB, JSON-LD, robots.txt, and /llms.txt
#
webdev
#
seo
#
ai
#
performance
Comments
Add Comment
2 min read
Moderation
Do AI bots actually read llms.txt? 8 days of server logs
Ali Baizhanov
Ali Baizhanov
Ali Baizhanov
Follow
Nov 9
Do AI bots actually read llms.txt? 8 days of server logs
#
seo
#
ai
#
webdev
#
llm
Comments
Add Comment
2 min read
Moderation
¿Cómo saber si ChatGPT puede leer tu sitio web? Dos revisiones en 5 minutos
Atalaya AI
Atalaya AI
Atalaya AI
Follow
Nov 11
¿Cómo saber si ChatGPT puede leer tu sitio web? Dos revisiones en 5 minutos
#
spanish
#
seo
#
webdev
#
ai
Comments
1
comment
2 min read
Moderation
Make your website readable by AI search: robots.txt, llms.txt and schema
DiziGoo Digital
DiziGoo Digital
DiziGoo Digital
Follow
Nov 11
Make your website readable by AI search: robots.txt, llms.txt and schema
#
seo
#
ai
#
webdev
#
marketing
Comments
Add Comment
2 min read
Moderation
Our store's robots.txt says no to AI training and yes to AI search. Here's what that actually means
RAMO
RAMO
RAMO
Follow
Nov 12
Our store's robots.txt says no to AI training and yes to AI search. Here's what that actually means
#
seo
#
ai
#
webdev
#
ecommerce
Comments
1
comment
4 min read
Moderation
AI search can reach almost every site we audited. Most give it little worth quoting
Kaustav Basak
Kaustav Basak
Kaustav Basak
Follow
Nov 7
AI search can reach almost every site we audited. Most give it little worth quoting
#
seo
#
ai
#
webdev
#
llm
Comments
2
comments
3 min read
Moderation
1
2
Next ›
Last »
We're a place where coders share, stay up-to-date and grow their careers.
Log in
Create account