Good Bots vs Bad Bots
Not all bots are created equal.
| Bots You Want | Bots You Dont Want |
|---|---|
| Googlebot — indexes your site for search | Scrapers — steal your content |
| Bingbot — same as Google, different engine | Credential stuffers — try leaked passwords |
| Ahrefs / Semrush — SEO analysis (debatable) | Vulnerability scanners — probe for weak points |
| Uptime monitors — Pingdom, Better Uptime | AI training crawlers — GPTBot, CCBot |
| Feed readers — RSS readers, Feedly | Price scrapers — competitor intelligence |
The goal: block bad bots while letting good ones through. A WAF with bot detection makes this automatic.
How SafeLines Bot Protection Works
SafeLine uses three layers to identify bots:
Layer 1: Static analysis
- User-Agent pattern matching against known bots
- IP reputation database (known scanner IPs)
- Request header consistency checks
Layer 2: Behavioral analysis
- Request frequency and timing patterns
- Page navigation sequences
- Resource loading behavior (does it load CSS/JS/images?)
Layer 3: Active challenge
- JavaScript challenge (execute JS, prove youre a browser)
- CAPTCHA (image/text challenge)
- Cookie validation
Setting Up Bot Protection
In SafeLine Dashboard -> Bot Management:
1. Enable Bot Detection: ON
2. Action for suspicious bots: JavaScript Challenge
3. Action for confirmed bad bots: Block
4. Whitelist: Googlebot, Bingbot, Pingdom, Stripe webhooks
Thats four settings and youre protected. The WAF handles the rest.
What It Caught on My Server
After enabling bot protection for 7 days:
| Bot Type | Detected | Action |
|---|---|---|
| Credential stuffing attempts | 2,847 | Blocked |
| Content scrapers | 1,203 | JS challenged (all failed) |
| Vulnerability scanners | 892 | Blocked |
| AI crawlers (GPTBot, CCBot) | 456 | JS challenged (all failed) |
| SEO crawlers (Ahrefs, Semrush) | 234 | Allowed (didnt trigger detection) |
| Googlebot | 3,456 | Whitelisted (never challenged) |
Zero legitimate users reported CAPTCHA fatigue. Google continued indexing normally.
FAQ
Will bot protection break my analytics?
No. Google Analytics, Plausible, and similar tools use JavaScript execution. Legitimate analytics are unaffected.
What about API clients (Postman, curl, SDKs)?
API endpoints with proper authentication should be added to the bot protection whitelist. Bots that authenticate with valid tokens are treated differently from anonymous scrapers.
Does it affect page load speed?
The JS challenge adds about 200ms on the first request for suspicious visitors. Legitimate users see zero delay.
Try SafeLine Community Edition — free, self-hosted, and takes 5 minutes to deploy:
bash -c "$(curl -fsSLk https://waf.chaitin.com/release/latest/manager.sh)" -- --en
Dashboard: https://<your-server-ip>:9443 | Docs
How much of your current traffic do you think is bots?
Top comments (0)