The Server Was Up. It Was Also Compromised.

The sites were still loading. Forms worked, pages opened, nothing was throwing errors. From the outside, everything looked fine.

It wasn't. Someone else had been living on that server.

What I found

• A rootkit hiding processes from the usual tools
• Backdoor files planted in web directories, made to look like ordinary PHP files
• An SSH key I had never added, sitting in an authorized_keys file
• Cron jobs quietly running malicious scripts on a schedule

None of it broke anything. That was the point. Attackers who break things get noticed and kicked out. Attackers who stay quiet keep access for months.

Cleaning it up

I removed the backdoors, the unknown key and every cron job I couldn't explain, got rid of the rootkit, and audited the installed packages for changed files.

Then I worked on making sure it couldn't happen the same way again:

• SSH: keys only, no passwords, no root login
• Firewall: UFW on, with only the ports that need to be open
• Integrity checks and login audits, so changes and strange logins get noticed
• Least privilege: services run as their own users, not as root

Watching it

A clean server only stays clean if someone notices when it changes. So I added off site backups with rclone, a watchdog that runs every 5 minutes, uptime and SSL checks, and alerts to Discord and email.

That setup now watches 29 client sites.

The checks I run now

Nothing advanced. That's exactly why they get skipped.

Who can log in?

cat /root/.ssh/authorized_keys /home/*/.ssh/authorized_keys

What runs on a schedule?

for u in $(cut -d: -f1 /etc/passwd); do crontab -l -u "$u" 2>/dev/null; done

What is listening?

ss -tulpn

Which PHP files changed this week?

find /var/www -name "*.php" -mtime -7

One warning: if a rootkit is already on the machine, the tools on that machine can lie to you. That's why integrity checks and backups stored somewhere else matter so much.

What I learned

"The site is up" says nothing about whether the server is safe. Uptime monitoring and security monitoring are different jobs.

Attackers don't break things. They move in and stay quiet. The most dangerous compromise looks like a normal day.

If you don't know what normal looks like, you can't spot what isn't. Write down the expected SSH keys, cron jobs and open ports for every server you run, and check against that list.

I'm Jabed, a DevOps engineer who also builds the Laravel, React and Next.js apps I deploy.