DEV Community

Jay Sullivan
Jay Sullivan

Posted on

The same redirect and bot-blocking rules for Apache, Nginx, Caddy and Cloudflare

#nginx #webdev #devops

Most server rules you find online are written for one server. Copy an .htaccess snippet to Nginx and nothing happens, because Nginx never reads .htaccess. Write HTTPS and www as two separate rules and every visitor takes two redirects instead of one.

Here are two everyday jobs written for all four places they usually live, plus the one mistake that turns a redirect list into a slow mess.

1. Force HTTPS and drop www, in one redirect

The goal: http://www.example.com/page goes straight to https://example.com/page in a single hop, not two.

Apache / LiteSpeed (.htaccess)

RewriteEngine On
RewriteCond %{HTTPS} off [OR]
RewriteCond %{HTTP_HOST} ^www\. [NC]
RewriteCond %{HTTP_HOST} ^(?:www\.)?(.+)$ [NC]
RewriteRule ^ https://%1%{REQUEST_URI} [L,R=301]
Enter fullscreen mode Exit fullscreen mode

The third condition captures the domain without www., so one rule fixes both problems at once. Behind Cloudflare or another proxy, %{HTTPS} can read off even when the visitor is on HTTPS; use RewriteCond %{HTTP:X-Forwarded-Proto} !https [OR] instead.

Nginx

server {
    listen 80;
    listen [::]:80;
    server_name example.com www.example.com;
    return 301 https://example.com$request_uri;
}
Enter fullscreen mode Exit fullscreen mode

You also need a second server block for https://www.example.com that returns the same redirect, with your certificate on it. No if, no rewrite: a separate server block is the fastest and clearest way.

Caddy

www.example.com, http://www.example.com, http://example.com {
    redir https://example.com{uri} permanent
}
Enter fullscreen mode Exit fullscreen mode

Caddy redirects HTTP to HTTPS on its own, so you only list the addresses that should land on the main one.

Cloudflare

No rule needed: turn on SSL/TLS > Edge Certificates > Always Use HTTPS, and create a redirect rule from the Redirect from WWW to root template under Rules.

2. Block AI training crawlers

The crawlers that collect training data identify themselves in the User-Agent. (List shortened here; the full, current list is in the generator.)

Apache / LiteSpeed

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (GPTBot|ClaudeBot|CCBot|Bytespider) [NC]
RewriteRule ^ - [F,L]
Enter fullscreen mode Exit fullscreen mode

Nginx (the map goes once inside http {}, the if inside your server {})

map $http_user_agent $blocked_ua {
    default 0;
    "~*(GPTBot|ClaudeBot|CCBot|Bytespider)" 1;
}

server {
    # ...
    if ($blocked_ua) { return 403; }
}
Enter fullscreen mode Exit fullscreen mode

Caddy

example.com {
    @ai_train header_regexp User-Agent "(?i)(GPTBot|ClaudeBot|CCBot|Bytespider)"
    respond @ai_train 403
}
Enter fullscreen mode Exit fullscreen mode

Cloudflare (Security > Security rules > Create rule > Edit expression, action: Block)

(lower(http.user_agent) contains "gptbot")
or (lower(http.user_agent) contains "claudebot")
or (lower(http.user_agent) contains "ccbot")
or (lower(http.user_agent) contains "bytespider")
Enter fullscreen mode Exit fullscreen mode

Add the same names to robots.txt as well: well-behaved crawlers stop there, and the server rule catches the rest.

3. The redirect-list mistake: chains

After a few redesigns, a redirect list ends up like this:

/old-page   /new-page
/2019/sale  /sale
/sale       /offers
Enter fullscreen mode Exit fullscreen mode

A visitor to /2019/sale now takes two hops: to /sale, then to /offers. Search engines follow chains, but each hop is wasted time, and long chains get abandoned. Fix the list so every old URL points at its final address:

/2019/sale  /offers
/sale       /offers
Enter fullscreen mode Exit fullscreen mode

Worse is a loop (/a → /b, /b → /a), which redirects forever until the browser gives up. Check for both every time you add lines.

A generator that does all of this

I run PaidHosting.com, and we built a free generator that writes these rules, and more, for all four servers from the same switches: .htaccess, Nginx & Caddy Rule Generator.

It shortens redirect chains and removes loops for you, blocks AI crawlers by group (training vs search), handles domain moves, removing .php from URLs and password-protected folders, and gives you a share link to your exact setup. We tested the output on Apache, Nginx, Caddy, LiteSpeed and a real Cloudflare account before release.

The code is open source (MIT): https://github.com/PaidHost/server-rules-generator. Bug reports for your own server setup are very welcome.

Top comments (0)