DEV Community

Cover image for Locking Down WHMCS: Nginx, Fail2ban, and What Nobody Tells You
Mourad Ilyes Mlik
Mourad Ilyes Mlik

Posted on

Locking Down WHMCS: Nginx, Fail2ban, and What Nobody Tells You

#security #nginx #sysadmin #webdev

If you run a hosting business on WHMCS, your admin panel is one of the most targeted endpoints on your entire infrastructure. Credential stuffing, brute force attempts, script scanners probing for known path, it never stops.

I run MMITech, a hosting provider in Slovenia, and our WHMCS instance sits behind Nginx serving seven language variants across multiple European markets. After watching the access logs for a while, I decided to properly harden the setup. Here's what I did and the gotchas I ran into along the way.

Step 1: Move the Admin Path

WHMCS ships with /admin as the default admin directory. Every scanner on the internet knows this. The first step is renaming it to something non-obvious.

In WHMCS, rename the admin directory:

mv /var/www/whmcs/admin /var/www/whmcs/your-secret-path
Enter fullscreen mode Exit fullscreen mode

Then update configuration.php:

$customadminpath = 'your-secret-path';
Enter fullscreen mode Exit fullscreen mode

This doesn't make you secure it just reduces noise. Security through obscurity is not security, but it does cut down on 99% of automated scans hitting your admin login.

Step 2: Restrict Admin Access by IP in Nginx

The real protection is restricting your admin path to known IP addresses at the web server level. Even if someone knows the path and has valid credentials, they can't reach it from an unauthorized IP.

location /your-secret-path/ {
    allow 203.0.113.10;    # Your office IP
    allow 198.51.100.0/24; # VPN range
    deny all;

    try_files $uri $uri/ /your-secret-path/index.php?$query_string;

    location ~ \.php$ {
        fastcgi_pass unix:/run/php/php8.3-fpm.sock;
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        include fastcgi_params;
    }
}
Enter fullscreen mode Exit fullscreen mode

After adding this, test from a non-whitelisted IP. You should get a 403 immediately. If you get a 200, your location block isn't matching, more on that below.

Step 3: Lock Down configuration.php

WHMCS stores database credentials, API keys, and license info in configuration.php. This file should never be served by the web server under any circumstances.

location ~* configuration\.php$ {
    deny all;
    return 404;
}
Enter fullscreen mode Exit fullscreen mode

Put this before your general PHP location block. Nginx processes location blocks in a specific order, exact matches first, then regex. If your general \.php$ block catches it first, the deny never triggers.

Test it:

curl -I https://yourdomain.com/configuration.php
# Should return 404
Enter fullscreen mode Exit fullscreen mode

Step 4: Fail2ban Jails for WHMCS

This is where it gets interesting. WHMCS logs failed login attempts, but its log format doesn't match any of fail2ban's built-in filters. You need custom jails.

The Nginx Log Format Problem

If you use a non-standard Nginx log format (which many WHMCS setups do because of reverse proxies or load balancers), fail2ban's default nginx-http-auth filter won't work. You need to write failregex patterns that match your actual log format.

Here's what our log lines look like:

203.0.113.55 - - [15/Mar/2026:14:22:31 +0100] "POST /dologin.php HTTP/1.1" 302 0 "https://example.com/login.php" "Mozilla/5.0..."
Enter fullscreen mode Exit fullscreen mode

A failed WHMCS login returns a 302 redirect back to the login page. A successful login also returns a 302 but redirects to the client area. The differentiator is the referer and the POST target.

Jail: WHMCS Client Login Brute Force

Create /etc/fail2ban/filter.d/whmcs-login.conf:

[Definition]
failregex = ^<HOST> .* "POST /dologin\.php HTTP/.*" 302
ignoreregex =
Enter fullscreen mode Exit fullscreen mode

And the jail in /etc/fail2ban/jail.local:

[whmcs-login]
enabled  = true
port     = http,https
filter   = whmcs-login
logpath  = /var/log/nginx/access.log
maxretry = 5
findtime = 300
bantime  = 3600
Enter fullscreen mode Exit fullscreen mode

This bans any IP that makes 5 failed login attempts within 5 minutes for 1 hour.

Jail: Admin Login Brute Force

Same concept but for the admin path, with a lower threshold:

Create /etc/fail2ban/filter.d/whmcs-admin.conf:

[Definition]
failregex = ^<HOST> .* "POST /your-secret-path/dologin\.php HTTP/.*" 302
ignoreregex =
Enter fullscreen mode Exit fullscreen mode 
[whmcs-admin]
enabled  = true
port     = http,https
filter   = whmcs-admin
logpath  = /var/log/nginx/access.log
maxretry = 3
findtime = 300
bantime  = 86400
Enter fullscreen mode Exit fullscreen mode

3 failed attempts and you're banned for 24 hours. Aggressive, but nobody legitimate fails admin login 3 times in 5 minutes.

Jail: Script Scanners

Bots constantly probe for paths like /wp-login.php, /xmlrpc.php, /phpmyadmin, etc. These are instant giveaways that the request is malicious:

Create /etc/fail2ban/filter.d/nginx-script-unknown.conf:

[Definition]
failregex = ^<HOST> .* "(GET|POST) /(wp-login|wp-admin|xmlrpc|phpmyadmin|pma|myadmin|administrator|\.env|\.git).*" (404|403)
ignoreregex =
Enter fullscreen mode Exit fullscreen mode 
[nginx-script-unknown]
enabled  = true
port     = http,https
filter   = nginx-script-unknown
logpath  = /var/log/nginx/access.log
maxretry = 2
findtime = 300
bantime  = 86400
Enter fullscreen mode Exit fullscreen mode

Jail: Recidive (Repeat Offenders)

The recidive jail catches IPs that keep getting banned and gives them a longer ban:

[recidive]
enabled  = true
logpath  = /var/log/fail2ban.log
banaction = %(banaction_allports)s
bantime  = 604800
findtime = 86400
maxretry = 3
Enter fullscreen mode Exit fullscreen mode

If an IP gets banned 3 times in 24 hours across any jail, they're blocked for a week on all ports.

Test Your Filters

Always test before deploying:

fail2ban-regex /var/log/nginx/access.log /etc/fail2ban/filter.d/whmcs-login.conf
Enter fullscreen mode Exit fullscreen mode

This shows you exactly how many lines match. If it returns 0 matches on a log that you know has failed login attempts, your regex is wrong.

Step 5: Security Headers (Watch the Inheritance)

This is the gotcha that cost me the most time. Nginx has a counterintuitive behavior with add_header: if you add any add_header directive in a child block, it completely overrides all add_header directives from the parent block.

Example of what goes wrong:

server {
    add_header X-Frame-Options "SAMEORIGIN" always;
    add_header X-Content-Type-Options "nosniff" always;
    add_header X-XSS-Protection "1; mode=block" always;

    location /your-secret-path/ {
        add_header X-Robots-Tag "noindex, nofollow" always;
        # BUG: The three headers above are now GONE for this location
    }
}
Enter fullscreen mode Exit fullscreen mode

The fix is to repeat all headers in every location block that adds its own headers. Or use the ngx_http_headers_more_module which has more_set_headers that doesn't have this inheritance problem.

Our approach repeat the base headers using an include file:

# /etc/nginx/snippets/security-headers.conf
add_header X-Frame-Options "SAMEORIGIN" always;
add_header X-Content-Type-Options "nosniff" always;
add_header X-XSS-Protection "1; mode=block" always;
add_header Referrer-Policy "strict-origin-when-cross-origin" always;
add_header Permissions-Policy "camera=(), microphone=(), geolocation=()" always;
Enter fullscreen mode Exit fullscreen mode

Then include it everywhere:

server {
    include /etc/nginx/snippets/security-headers.conf;

    location /your-secret-path/ {
        include /etc/nginx/snippets/security-headers.conf;
        add_header X-Robots-Tag "noindex, nofollow" always;
    }
}
Enter fullscreen mode Exit fullscreen mode

Verify with:

curl -I https://yourdomain.com/
curl -I https://yourdomain.com/your-secret-path/
Enter fullscreen mode Exit fullscreen mode

Both should show the full set of security headers.

Step 6: Rate Limiting the Login Endpoint

On top of fail2ban, add Nginx-level rate limiting as a first line of defense:

# In the http block
limit_req_zone $binary_remote_addr zone=whmcs_login:10m rate=5r/m;

# In the server block
location = /dologin.php {
    limit_req zone=whmcs_login burst=3 nodelay;
    limit_req_status 429;

    fastcgi_pass unix:/run/php/php8.3-fpm.sock;
    fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
    include fastcgi_params;
}
Enter fullscreen mode Exit fullscreen mode

This limits login attempts to 5 per minute per IP, with a burst allowance of 3. Anything beyond that gets a 429 response before it even hits PHP, saving server resources.

The Result

After deploying all of this, our daily failed login noise dropped from hundreds of attempts to effectively zero reaching the application. The combination works in layers:

  1. Non-default admin path: eliminates automated scanners
  2. IP restriction on admin: blocks everyone except authorized IPs
  3. Rate limiting on login: throttles brute force at the Nginx level
  4. Fail2ban jails: bans persistent offenders at the firewall level
  5. Recidive jail: escalates bans for repeat offenders
  6. Security headers: prevents clickjacking and XSS
  7. configuration.php blocked: protects credentials from direct access

None of these individually is sufficient. Together, they create enough friction that attackers move on to easier targets.

I'm Mourad, founder of MMITech a hosting provider based in Kranj, Slovenia. We run Cloud VPS on Proxmox/Ceph, AMD VPS on Ryzen 9/NVMe, Nextcloud storage, and dedicated servers. If you have questions about this setup or want to compare notes, drop a comment or reach out at hello@mmitech.si.

Top comments (0)