DEV Community

Lia
Lia

Posted on

How to Detect and Block Malicious Bots (Without Blocking Real Users)

#cybersecurity #security #programming #devops

How to Detect and Block Malicious Bots (Without Blocking Real Users)

Not all bots are bad. Search engines crawl your site to index it; monitoring and webhooks call your APIs legitimately. The problem is the other kind — scrapers, credential stuffers, spam bots, and DDoS clients — that abuse your resources or probe for weaknesses. The trick isn't blocking bots wholesale; it's telling the good from the bad and acting on the difference.

Here's a practical approach, and where a self-hosted WAF like SafeLine fits.

First, recognize the traffic

Bots reveal themselves through patterns a human never produces:

  • Volume and rate — hundreds of requests from one source far faster than a person can click.
  • Repetitive paths — the same endpoint hammered in a loop (login, search, checkout).
  • Thin or spoofed headers — missing User-Agent, or a fingerprint shared across thousands of requests.
  • Behavioral tells — same sequence, same timing, no variation.

Legitimate bots (Googlebot, Bingbot, status checkers) usually identify themselves and behave predictably. The noisy, evasive ones are what you want to stop.

A layered response

  1. Allowlist the good bots. If you want search engines to index you, let known crawlers through by identity rather than treating all automation as hostile.
  2. Rate limit the rest. Cap requests per IP and per path so one client can't monopolize an endpoint.
  3. Challenge the suspicious. Force a CAPTCHA or temporary block on clients that look automated but you're not certain about.
  4. Block the known-bad. Drop traffic from sources with clear abuse signatures.

Where SafeLine fits

SafeLine is a self-hosted WAF that runs in reverse-proxy mode in front of your app, so every request passes through it first. For bot mitigation:

  • Bot management distinguishes automated clients from real users using behavioral and request analysis, so it can throttle or block the abusive ones while letting people through.
  • Rate limiting can be applied per path — ideal for login, search, and API endpoints that bots love to hammer.
  • Because it's self-hosted and free to run (Community Edition covers up to 10 apps at 800 QPS), you get bot protection without adding a per-request vendor bill.

The goal is surgical: stop the abuse, keep the legitimate traffic — including the good bots you actually want.

Putting it in front of your traffic

Deploy SafeLine as the proxy in front of your service:

bash -c "$(curl -fsSLk https://waf.chaitin.com/release/latest/manager.sh)" -- --en
Enter fullscreen mode Exit fullscreen mode

In the console at https://<your-server-ip>:9443, add your site and point its upstream at your app. Enable bot mitigation, set rate limits on your highest-traffic paths, and allowlist the crawlers you trust. From there, malicious bots get throttled or blocked at the edge while real users and legitimate bots flow through.

FAQ

Won't blocking bots hurt my SEO?

Only if you block the good ones. Allowlist search-engine crawlers by identity and they'll index normally; you're targeting abusive automation, not legitimate crawlers.

Can a WAF tell a bot from a human reliably?

It uses behavioral and request signals, not a perfect test. Rate limiting and challenges handle the uncertain cases; outright abuse is blocked outright.

Do I need to write bot-detection rules myself?

No. SafeLine's bot management works out of the box; you tune thresholds and allowlists rather than hand-authoring detection logic.

Is this free to self-host?

Yes — the Community Edition is free to run indefinitely and covers up to 10 apps at 800 QPS.

That's it — your traffic now has bot filtering and rate limiting at the edge.

Top comments (0)