DEV Community

Cover image for SECURING A DEBIAN SERVER WITH FAIL2BAN
Frank Wisniewski
Frank Wisniewski

Posted on

SECURING A DEBIAN SERVER WITH FAIL2BAN

#web #debian

As soon as a server becomes reachable from the Internet, it usually does not take long before the first unfriendly visitors arrive. This is no longer limited to occasional SSH login attempts. Automated scanners continuously check whether WordPress, phpMyAdmin, forgotten .env files, Git repositories, PHP debugging tools, known vulnerabilities or other exploitable services are available.

And when I say "check", I am talking about fairly impressive numbers.

On my small Raspberry Pi, I observed one scanner sending 272 HTTP requests within the same logged second. Another scanner generated 300 requests within roughly two seconds.

These were not requests for existing pages. The scanners systematically searched for files such as:

/.env
/.git/config
/.aws/credentials
/.ssh/id_rsa
/wp-config.php
/phpinfo.php
/docker-compose.yml
/composer.json
Enter fullscreen mode Exit fullscreen mode

The small Raspberry Pi handled the traffic surprisingly well. Still, there is no reason to allow remote systems to test hundreds or thousands of URLs without restriction.

This is where Fail2Ban comes in.

Fail2Ban monitors system log files and detects repeated suspicious activity. If an IP address exceeds predefined limits, it is automatically blocked at the firewall level for a specified period of time.

Fail2Ban does not replace proper server configuration, regular updates or secure passwords. It also cannot protect against a real volumetric DDoS attack.

What it does very effectively is prevent automated scanners from spending hours testing their complete dictionaries against a server.

The resource requirements are minimal, configuration is straightforward and it runs perfectly well even on a small Raspberry Pi. For publicly accessible Linux servers, Fail2Ban has therefore become part of my standard setup.

My server runs Debian with Apache. At the router, only TCP ports 80 and 443 are forwarded to the web server. SSH and other administrative services are not accessible from the Internet at all.

That remains the most important security measure:

If a service does not need to be accessible from the Internet, do not expose it to the Internet.

Before starting with the configuration, here is a simplified overview of what happens:

flowchart TB

    A([Internet])
    B["Router<br/>only TCP 80 and 443"]
    C["Apache Web Server"]
    D{"Type of request"}

    DN["normal"]
    DF["file missing"]
    DX["exploit"]

    E["Serve normal request"]

    F["File does not exist<br/>403 or 404"]
    G["apache 404 scan<br/>counts matches"]
    H{"Threshold<br/>reached?"}
    HN["not yet"]
    HY["reached"]
    I["IP remains allowed"]

    J["Exploit attempt"]
    K["Apache processes or<br/>rejects request"]
    L["apache hacker<br/>detects pattern"]
    M{"Threshold<br/>reached?"}
    MN["not yet"]
    MY["reached"]
    N["IP remains allowed"]

    O["Firewall<br/>blocks IP address"]
    P(["Further requests from this IP<br/>no longer reach Apache"])

    A --> B
    B --> C
    C --> D

    D --> DN
    DN --> E

    D --> DF
    DF --> F
    F --> G
    G --> H
    H --> HN
    H --> HY
    HN --> I
    HY --> O

    D --> DX
    DX --> J
    J --> K
    K --> L
    L --> M
    M --> MN
    M --> MY
    MN --> N
    MY --> O

    O --> P

    classDef internet fill:#f00,stroke:#3578e5,stroke-width:4px,color:#fff
    classDef router fill:#fff4d6,stroke:#d89000,stroke-width:2px,color:#111
    classDef apache fill:#e4f7e8,stroke:#209447,stroke-width:2px,color:#111
    classDef normal fill:#eef8ee,stroke:#45a049,stroke-width:1.5px,color:#111
    classDef warning fill:#fff0e6,stroke:#e07020,stroke-width:2px,color:#111
    classDef fail2ban fill:#f0eaff,stroke:#7257c8,stroke-width:2px,color:#111
    classDef decision fill:#f8f4ff,stroke:#7257c8,stroke-width:2px,color:#111
    classDef label fill:#ffffff,stroke:#888,stroke-width:1px,color:#111
    classDef blocked fill:#ffe5e5,stroke:#b42318,stroke-width:3px,color:#111
    classDef endnode fill:#eeeeee,stroke:#555,stroke-width:2px,color:#111

    class A internet
    class B router
    class C,D apache
    class E normal
    class F,J,K warning
    class G,L fail2ban
    class H,M decision
    class DN,DF,DX,HN,HY,MN,MY label
    class O blocked
    class P endnode

Installing Fail2Ban

On Debian or Raspberry Pi OS, installation is straightforward:

sudo apt update
sudo apt install fail2ban
sudo systemctl enable --now fail2ban
Enter fullscreen mode Exit fullscreen mode

A quick check confirms whether the service is running:

sudo systemctl status fail2ban
Enter fullscreen mode Exit fullscreen mode

Configured jails can be displayed with:

sudo fail2ban-client status
Enter fullscreen mode Exit fullscreen mode

Fail2Ban already includes several filters for Apache.

However, for my web server I also use a few simple custom rules. This allows me to define exactly what I consider suspicious behaviour on this particular server.

Detecting scanners using 403 and 404 responses

A single 404 response is perfectly normal.

Someone may mistype an address, a browser may request an outdated favicon, or an old link may point to a page that no longer exists.

If the same IP address requests ten, twenty or one hundred different nonexistent files within a few minutes, the situation looks rather different.

For this purpose I create a custom filter:

sudo nano /etc/fail2ban/filter.d/apache-404-scan.conf
Enter fullscreen mode Exit fullscreen mode

Contents:

[Definition]

failregex = ^<HOST> .* "(?:GET|POST|HEAD|OPTIONS) .*" (?:403|404) (?:\d+|-)
ignoreregex =
Enter fullscreen mode Exit fullscreen mode

Then create the corresponding jail:

sudo nano /etc/fail2ban/jail.d/apache-404-scan.local
Enter fullscreen mode Exit fullscreen mode

Contents:

[apache-404-scan]

enabled = true
port = http,https
filter = apache-404-scan
logpath = /var/log/apache2/*access.log

findtime = 300
maxretry = 10
bantime = 86400
Enter fullscreen mode Exit fullscreen mode

With this configuration, an IP address may generate ten matching errors within five minutes.

Once the threshold is exceeded, the address is blocked for 24 hours.

These values are not universally correct for every website.

A large website with many outdated links may need more generous limits. On my small server, however, it is highly unlikely that a legitimate visitor will generate ten different 403 or 404 responses within five minutes.

Handling obvious attack patterns separately

Some requests are much less ambiguous.

There is no legitimate reason for an external visitor to search my server for files such as:

/.env
/.git/config
/.aws/credentials
/.ssh/id_rsa
/wp-config.php
/phpinfo.php
/cgi-bin/../../../../bin/sh
Enter fullscreen mode Exit fullscreen mode

Someone requesting such paths is probably not trying to read my homepage.

For these patterns I use an additional filter:

sudo nano /etc/fail2ban/filter.d/apache-hacker.conf
Enter fullscreen mode Exit fullscreen mode

Example configuration:

[Definition]

failregex = ^<HOST> .* "(?:GET|POST|HEAD|OPTIONS) .*?(?:\.env|\.git/|\.aws/|\.ssh/|wp-config|phpinfo|/cgi-bin/).*" (?:200|301|400|403|404) .*
ignoreregex =
Enter fullscreen mode Exit fullscreen mode

The corresponding jail can react much more aggressively:

sudo nano /etc/fail2ban/jail.d/apache-hacker.local
Enter fullscreen mode Exit fullscreen mode 
[apache-hacker]

enabled = true
port = http,https
filter = apache-hacker
logpath = /var/log/apache2/*access.log

findtime = 600
maxretry = 3
bantime = 604800
Enter fullscreen mode Exit fullscreen mode

In this example, three matching requests within ten minutes result in a seven day ban.

For a general 404 filter, that would be too aggressive for my taste.

But if someone repeatedly requests .aws/credentials, .ssh/id_rsa or wp-config.php, they can spend the next week looking for something else to do.

Testing filters before enabling them

Custom filters should always be tested against the existing Apache log before they are allowed to block addresses.

For example:

sudo fail2ban-regex /var/log/apache2/access.log /etc/fail2ban/filter.d/apache-404-scan.conf
Enter fullscreen mode Exit fullscreen mode

And for the second filter:

sudo fail2ban-regex /var/log/apache2/access.log /etc/fail2ban/filter.d/apache-hacker.conf
Enter fullscreen mode Exit fullscreen mode

Fail2Ban reports how many log entries were matched.

This makes it possible to check whether the regular expression is actually detecting the intended requests.

Afterwards, reload the configuration:

sudo fail2ban-client reload
Enter fullscreen mode Exit fullscreen mode

The status of an individual jail can be displayed with:

sudo fail2ban-client status apache-404-scan
Enter fullscreen mode Exit fullscreen mode

or:

sudo fail2ban-client status apache-hacker
Enter fullscreen mode Exit fullscreen mode

What happens when 272 requests arrive in one second?

This is where things become interesting.

Fail2Ban does not sit in front of Apache.

The basic process looks like this:

Request arrives
        ↓
Apache processes request
        ↓
Apache writes access log entry
        ↓
Fail2Ban detects the log entry
        ↓
Threshold is exceeded
        ↓
Fail2Ban creates firewall rule
        ↓
Further connections are blocked
Enter fullscreen mode Exit fullscreen mode

This inevitably introduces a small delay.

Modern scanners, however, can be extremely fast.

In my Apache log, one IP address appeared 272 times within the same logged second.

The scanner searched for Git files, WordPress configuration files, Composer files and other common configuration files.

That represents a burst in the range of:

272 requests per second.

Another scanner generated a total of approximately 300 requests within roughly two seconds.

This means something like the following can happen:

00.000 s   Scanner starts
00.050 s   Many requests are already in flight
00.200 s   Apache writes log entries
00.400 s   Fail2Ban detects matches
00.600 s   Threshold is exceeded
00.800 s   Firewall rule is installed
Enter fullscreen mode Exit fullscreen mode

These timings are only an illustration. The important part is the principle.

If the scanner has already sent hundreds of requests in parallel, many of them may still reach Apache even though maxretry is configured as only 10.

So:

maxretry = 10
Enter fullscreen mode Exit fullscreen mode

does not mean:

Apache will see no more than 10 requests.
Enter fullscreen mode Exit fullscreen mode

It means:

After enough matching requests have been detected,
the IP address will be blocked.
Enter fullscreen mode Exit fullscreen mode

With slow bots there may be almost no practical difference.

With scanners sending several hundred requests per second, the difference can be substantial.

Fail2Ban still performs its job.

Once the firewall rule has been installed, the next thousand or ten thousand requests from that IP address no longer reach Apache.

Results after only a few days

After only a few days of operation, the status on my server already looked like this:

Jail Detected matches Currently banned Total bans
apache-hacker 253 51 51
apache-php-scan 82 8 8
apache-404-scan 847 8 17

Fail2Ban had already triggered 76 ban actions.

Because some particularly active scanners were detected by several filters, these numbers cannot simply be added together to calculate the number of unique attackers.

Some IP addresses appeared simultaneously in the 404 filter, the PHP scanner and the hacker filter.

That makes sense.

A scanner requesting .git/config, followed by wp-config.php, followed by known PHP shell filenames, qualifies for several categories at once.

The change in Apache log file size was even more interesting.

Before enabling Fail2Ban, individual daily logs contained several thousand requests.

Afterwards, the numbers dropped considerably.

This does not mean the Internet suddenly became friendlier.

The bots are still there.

They simply do not get very far anymore.

The Raspberry Pi hardly notices

At first, 272 requests within a single second sounds like something that should make a small Raspberry Pi struggle.

For this type of scanning, however, most requests are extremely cheap to process.

Usually Apache only has to do something like:

GET /.env
File does not exist
Return 404
Done
Enter fullscreen mode Exit fullscreen mode

Or:

GET /wp-config.php
File does not exist
Return 404
Done
Enter fullscreen mode Exit fullscreen mode

There is no large database query, no computationally expensive application and no large document being generated.

A short burst of several hundred simple requests is therefore surprisingly undramatic for a Raspberry Pi.

Once Fail2Ban blocks the IP address, things become even easier.

Further connections are rejected at the firewall level.

Apache never sees them and therefore does not write additional access log entries.

This is also why Apache logs can become dramatically smaller after enabling Fail2Ban.

Fake bots and misleading user agents

Another interesting detail becomes obvious when looking through access logs.

One of the aggressive scanners identified itself as:

Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Enter fullscreen mode Exit fullscreen mode

This obviously does not mean that Google was trying to download my AWS credentials or .git/config.

The HTTP User Agent is simply text supplied by the remote client.

Anyone can write anything into it.

Other attack requests used the names of well known search engines, AI bots and normal browsers while simultaneously requesting paths such as:

../../../../etc/passwd
/@fs/proc/self/environ
/@fs/root/.aws/credentials
Enter fullscreen mode Exit fullscreen mode

Using the User Agent as a security criterion would therefore be a poor idea.

Behaviour is what matters.

Do not accidentally ban your own network

Trusted local networks can be excluded from banning.

For example:

sudo nano /etc/fail2ban/jail.local
Enter fullscreen mode Exit fullscreen mode

Then:

[DEFAULT]

ignoreip = 127.0.0.1/8 ::1 192.168.2.0/24
Enter fullscreen mode Exit fullscreen mode

Reload afterwards:

sudo fail2ban-client reload
Enter fullscreen mode Exit fullscreen mode

Naturally, only networks that are actually trusted should be added here.

The router firewall is still more important

Fail2Ban is only an additional layer.

On my server, the router exposes only:

80/tcp
443/tcp
Enter fullscreen mode Exit fullscreen mode

SSH is not reachable from the Internet.

The database is not reachable.

Samba is not reachable.

There is no externally accessible administration port.

This significantly reduces the attack surface.

A closed SSH port is better than a perfectly secured SSH service that I do not need remotely in the first place.

Fail2Ban then deals with the unavoidable noise hitting the two ports that actually need to remain publicly accessible because they provide the web service.

Conclusion

A publicly accessible web server will be scanned.

Not perhaps someday.

It starts almost immediately and continues permanently.

And we are not just talking about an occasional request for /wp-admin.

Modern scanners can test several hundred different URLs within a single second.

On my small Raspberry Pi, one real example produced:

272 requests within the same second.

Apache handled it without difficulty.

Fail2Ban detected the source.

The IP address was blocked.

From that point on, the firewall handled the rest.

Fail2Ban cannot turn an insecure server into a secure one.

But combined with an up to date Debian installation, a properly configured firewall, only the necessary port forwards and a clean Apache configuration, it provides a remarkably effective additional layer of protection.

And its resource requirements are so small that even a Raspberry Pi has plenty of time left to tell automated visitors from around the world:

404. Not here. And that is enough for today.

Top comments (0)