Writing robots.txt by hand means remembering the exact syntax and knowing which bot names to use. I built a free, browser-only Robots.txt Generator with visual bot selection and AI crawler presets.
Live Tool
👉 https://devnestio.pages.dev/robots-txt-gen/
What it does
- Bot chips — toggle Googlebot, Bingbot, GPTBot, CCBot, anthropic-ai, Applebot, or any custom user-agent
- Allow/Disallow rules — add paths with quick buttons
- Templates — Allow all, Block all, Block AI crawlers, SEO-friendly, WordPress standard
- Crawl-delay and Sitemap — optional directives
- Syntax highlighting — color-coded output
- Download — save as robots.txt directly
- 100% client-side
Robots.txt quick reference
User-agent: *
Disallow: /admin/
Sitemap: https://example.com/sitemap.xml
# Block AI training crawlers
User-agent: GPTBot
Disallow: /
User-agent: anthropic-ai
Disallow: /
AI crawler user-agents
| Bot | Company |
|---|---|
| GPTBot | OpenAI |
| CCBot | Common Crawl |
| anthropic-ai | Anthropic |
| Applebot | Apple |
Testing
84 tests, all passing
Tests cover all 5 templates, bot chip toggle, Disallow/Allow rules, download, and HTML escaping.
All tools at devnestio.pages.dev.
Top comments (0)