Jeremy Morgan

Posted on Feb 8, 2020 • Edited on Jul 25

Analyzing the Attacks on my Website

#webdev #linux #security #showdev

I was casually doing a security audit on my blog (JeremyMorgan.com) recently and decided to look a little deeper into my security logs. With a bit of Linux command line kung fu, some Golang, and Google sheets, I was able to get a pretty good idea of where the attacks are coming from.

To start, I'm using CentOS to host my site, so I checked out /var/log/secure. This log is where authentication logs are stored on my server.

This is what the log file looks like:

and with 301,327 lines it's not likely I'm going to manually look around much. Let's automate this a bit.

Getting the IP Address of attackers

I wanted to extract the IP address of attackers from this file. That way I can block them.

I started to mess around with Linux commands until I came up with this script.

What it does is pretty simple, it's going to look for these strings:



declare -a badstrings=("Failed password for invalid user"
                "input_userauth_request: invalid user"
                "pam_unix(sshd:auth): check pass; user unknown"
                "input_userauth_request: invalid user"
                "does not map back to the address"
                "pam_unix(sshd:auth): authentication failure"
                "input_userauth_request: invalid user"
                "reverse mapping checking getaddrinfo for"
                "input_userauth_request: invalid user"
                )

These are strings that identify logs of failed attacks. If they put in the wrong username or tried some other form of attack, it would have one of these strings.

So we loop through that list and search for these strings, then extract an IP address from the line the string exists in.



cat /var/log/secure | grep "$i" | grep -E -o "([0-9]{1,3}[\.]){3}[0-9]{1,3}" | awk '{print $0}' | sort | uniq >> "temp.txt"

It then dumps the IP into a temp.txt file. It will do this for all of the messages I have in my "badstrings" list.

That text file had a ton of duplicates in it, so I removed the duplicates and put only the unique IPs into a file:



# grab unique ips from temp and put them in a file
cat "temp.txt" | sort | uniq > "badguyips.txt"
# remove the temp file
rm "temp.txt"

Cool, now I have a list of IP addresses ready to go.

Yikes, I have 1,141 IP addresses here.

Blocking Them

Now I want to block these IP addresses. Since I'm running iptables, I can just drop them with this simple script:



#!/bin/bash
input="badguyips.txt"
while IFS= read -r line
do
  iptables -A INPUT -s $line -j DROP
done < "$input"

service iptables save

Cool. Now the attackers blocked from my server.

Then I got curious. Where the heck are these attacks coming from?

Getting their Location Data

Since I have a list of IP addresses, I thought I'd run them against a database like Maxmind to find some location information. So I did just that.

I wrote this Golang program called "find the bad guys" that would go through the text file of IP addresses, and look up their location information, then write it to a series of text files.

I wrote out locations based on:

Continent
Countries
Cities
Subdivisions of Cities

I wanted to see where the attacks are coming from and share that information. so I ran the program I built, and now have some helpful lists of location information:

Continents

So now I want to take a look at continents.txt.

Well, that's going to be a problem, there are some duplicates.

I can run a quick command and get unique values:



cat continents | sort | uniq

The results should come as no surprise if you've ever looked at a globe:

But I want to see how many attacks from each continent. So I call on my old friend uniq for that:



awk -F '\n' '{print $0}' continents.txt | sort | uniq -c

Pretty sweet, right? So I'll remove the leading spaces, insert a comma after the count and drop it into a text file.



awk -F '\n' '{print $0}' continents.txt | sort | uniq -c | awk '{$1=$1};1' | sed -r 's/\s+/,/'  > contintent-totals.txt

Now I can drop it into Google sheets.

and get this nice chart:

This is the process I repeat for the other locations (country, city, subdivision), so I won't repeat it. So here are my results:

Countries

Here are the top 10 countries attackers are coming from:

China (304)
United States (138)
France (95)
India (46)
Singapore (43)
South Korea (38)
Germany (37)
Russia (37)
Brazil (35)
United Kingdom (29)

Cities

Attacks per city are a little more aggregated.

Beijing (57)
Shanghai (53)
Hefei (25)
Amsterdam (21)
Bengaluru (16)
London (14)
Xinpu (14)
Clifton (10)
North Bergen (9)

But still pretty interesting.

Subdivisions

This one is aggregated even more. But it drills down a bit more. Here are the top 10 subdivisions attackers are coming from:

Beijing (145)
Shanghai (61)
Anhui (26)
England (22)
Jiangsu (22)
New Jersey (22)
North Holland (22)
California (18)
Sao Paulo (18)
Karnataka (16)

Conclusion

Great things always come from curiosity. I'm curious about what other kinds of patterns and data I can extract from this, so I'm going to keep experimenting and playing with it.

If you decide you want to do this for your website, try it these steps, and Let me know if you need any help with it.

Top comments (22)

Ben Overmyer • Feb 9 '20

You should look into two things:

fail2ban, for automatically banning IP addresses. github.com/fail2ban/fail2ban
firewalls above your individual VM. Banning SSH connections from all but the IP addresses you personally access your VM from will make this a non-issue.

Phil Ashby • Feb 9 '20 • Edited

Seconding the fail2ban tooling - excellent support for just about anything you wish to expose to the random Internet, and you can still pull many stats from it's logs for graphing.

Curious as to how many of the IPs you are seeing attacks from are exit nodes for commercial VPNs or Tor? How many are cloud hosted VMs, or zombie personal machines? IP allocation / reverse DNS should help find this stuff out, as would services like this: ipvoid.com/ip-blacklist-check/ :)

In addition to local controls (such as fail2ban), I would also strongly encourage use of a CDN/anti-DDoS/Web App Firewall product (such as Cloudflare, Azure CDN, AWS CDN, Imperva Cloud WAF, etc.) for your outer layer or protection - they will keep up with threats waay faster than one human!

Dave Ceddia • Feb 10 '20

You didn't mention SSH keys and maybe you already do this, but for the benefit of others trying to secure a public-facing machine, it's best to disable password-based authentication and use ssh keys to authenticate instead. Combined with fail2ban (mentioned by Ben) it's a good way to prevent brute force attacks.

DigitalOcean has a good writeup of how to generate SSH keys and configure a Linux machine to use them.

rstamper-ai • Feb 10 '20

First off, very nicely done. Love the breakdown and that IPTables loop script is awesomely useful too.

Don't they often spoof IPs so banning that stops them right now, but they're back in an hour on another IP?

Also, using the auth log, is that a guarantee these are all attacks, or is it possible some were "good guys"? [just playing devil's advocate here]

Gonna do something like this on my servers, see what I get out of it :)

tiff • Feb 14 '20

I was just thinking this. They could definitely use a VPN or some other proxy to mask their IP address.

Kaustubh Phatak • Feb 9 '20

Cool writeup. There are millions of ways to do this ofcourse.
May be you want to look at google sheets API so you can mostly automate all tasks in the above list and can analyze stuff over time directly in google sheets ;)
or you can just use prometheus and grafana...haha

mikcat • Feb 11 '20

Please note that using

iptables -A INPUT -s $line -j DROP

is appending your rule to the end of INPUT rules. This can be a problem if the INPUT chain has any other rules that have been applied before.

To ban an IP is better to prepend the rule at the beginning:

iptables -I INPUT 1 -s $line -j DROP

Althoug I prefer to create a custom chain for them, so you can prepend your custom chain to the beginning of INPUT, and append ips safely to it:

iptables -N bannejats
iptables -A bannejats -j RETURN
# No line specified because it defaults to 1 :)
iptables -I INPUT -p tcp -j bannejats

input="badguyips.txt"
while IFS= read -r line
do
  iptables -I bannejats 1 -s $line -j REJECT --reject-with icmp-port-unreachable
  # Or also safe now because bannejats is on head
  # iptables -A bannejats -s $line -j DROP
done < "$input"

(A note on Drop versus Reject chiark.greenend.org.uk/~peterb/net...)

Alessandro (Ale) Segala • Feb 15 '20

Blocking IP addresses of attackers is a lost battle. It's incredibly easy for attackers to get a different IP...

I wrote (in 2016) an article with 3 simple tricks (that don't require installing any other app, like fail2ban) that have been working great for me to stop tentative brute force attacks: withblue.ink/2016/07/15/stop-ssh-b... TL;DR: use SSH keys, rate-limit connections to port 22 (using iptables), and/or change the SSH default port.

xxzz • Feb 9 '20

sshguard, fail2ban will automate this. You should try ipset instead of blocking via iptables.

rndmh3ro • Feb 10 '20

One of the easiest steps (next to the already mentioned) is using a different port for ssh. This way these automated attacks won't happen anymore.
Additionally fail2ban and you're set.

Disclaimer: I know this won't stop a dedicated attacker, but it will stop the automated attacks and scans.