One SSH key, one cron line: a 5-minute persistence audit

#security #linux #ssh #incidentresponse

The scenario is fictional (RFC 5737 IPs). Detection only; no attack steps.

You found the intrusion, killed the process, changed the password. Done? Attackers plan to be locked out. Within minutes of getting in they build a second and third way back.

What you'll learn

The two classic persistence spots: SSH keys and cron
Baseline diffs that find them in minutes
How to tell admin changes from attacker changes
Why you preserve before you clean

The story at a glance

Time	What happened
02:20	Root login succeeds after brute force
02:26	An unknown key appears in root's `authorized_keys`
02:27	A cron entry runs every minute
Later	Password reset and process kill don't help

Keys: diff against a saved baseline

find / -name authorized_keys -exec sha256sum {} \; | sort > /var/lib/keys_base.txt   # normal times
find / -name authorized_keys -exec sha256sum {} \; | sort > /var/lib/keys_now.txt    # investigation
diff /var/lib/keys_base.txt /var/lib/keys_now.txt

In my sandbox test only root's file showed up in the diff.

Cron: new files and fetch-and-execute

find /etc/cron.d /var/spool/cron -type f -exec sha256sum {} \; | sort
grep -REl 'curl[^|]*\|[[:space:]]*(ba)?sh' /etc/cron.d /etc/crontab /var/spool/cron

The grep flagged the fake malicious cron and ignored a legit backup job. Legit tools sometimes use curl | bash, so check whether the destination is yours.

Normal vs. attack

	Normal	Attack
Changed by	config management (ansible), uid 0	`www-data` (uid 33) with euid 0
Path	change ticket exists	from the web server process
When	maintenance window	minutes after intrusion

Catch it live: auditd (from the book, not sandbox-tested)

-w /root/.ssh/authorized_keys -p wa -k ssh_key_change
-w /etc/crontab -p wa -k cron_persist
-w /etc/cron.d/ -p wa -k cron_persist
-w /var/spool/cron/ -p wa -k cron_persist
-w /etc/systemd/system/ -p wa -k systemd_persist

Preserve first, clean later

Copy files, hash, record stat times and who changed them. Only then remove the key line and the cron file.

Takeaways

Assume redundancy: keys, cron, systemd, profile.d, other users' ~/.ssh. Save the baseline before you need it.

Next: SSH that skipped the bastion.

Adapted from chapters 26 and 9 of my Japanese book on server defense.

DEV Community