Skip to content
Navigation menu
Search
Powered by Algolia
Search
Log in
Create account
DEV Community
Close
← All Trends
AI Crawler Blocking vs. Traffic Acquisition in SEO
20 posts in this trend in the last 7 days
•
Active 3 days ago
Our robots.txt bans 31 crawlers. I tested 7: two got through, and Baidu got a 520.
tong zhang
tong zhang
tong zhang
Follow
Nov 10
Our robots.txt bans 31 crawlers. I tested 7: two got through, and Baidu got a 520.
#
webdev
#
seo
#
http
#
security
Comments
Add Comment
9 min read
Moderation
Can ChatGPT Read Your Website? How I Found a Block My robots.txt Didn't Show
Yunus Ertürk
Yunus Ertürk
Yunus Ertürk
Follow
Nov 9
Can ChatGPT Read Your Website? How I Found a Block My robots.txt Didn't Show
#
chatgpt
#
seo
#
cloudflare
#
webdev
1
reaction
Comments
6
comments
8 min read
Moderation
« First
‹ Prev
1
2
We're a place where coders share, stay up-to-date and grow their careers.
Log in
Create account