Emergency Server Recovery: A 4-Hour Race Against Time

## The 3AM Wake-Up Call
You know that feeling when your phone buzzes at an ungodly hour and your stomach drops? That was me, staring at a frantic message: "Site is showing weird content. Help!"

I grabbed my laptop. The WordPress site was serving pharmaceutical spam to visitors. Classic compromise. The clock started ticking.

## Hour 1: Damage Assessment (03:00 - 04:00)
First rule of server emergencies: don't panic, but move fast.
What I Did First
SSH into the server, check if I still had access. Good news: credentials still worked. Bad news: everything else.

Check recent file modifications

find /var/www/html -type f -mtime -7 -ls | head -20

Tons of suspicious PHP files scattered everywhere. The wp-content/uploads folder was full of backdoors. Someone had gotten in through an outdated plugin, probably.
Quick Isolation
Pulled the site offline with a maintenance page. Better to show "down for maintenance" than spam pills to your visitors.

Quick nginx block

location / {

return 503;

}

Took a snapshot of everything before touching anything. You need evidence, and you might need to rollback if things go sideways.

## Hour 2: The Cleanup (04:00 - 05:00)
Finding the Entry Point
Checked Apache/Nginx logs for unusual POST requests:

grep -i "POST" /var/log/nginx/access.log | grep -E ".(php|asp|jsp)" | tail -100

Found it. An old contact form plugin with a known vulnerability. They uploaded a shell through a file upload field that wasn't properly validated.
Nuclear Option with Surgical Precision
Here's the thing about compromised WordPress sites: you can't trust anything. But you also can't just delete everything because you need the data.

My approach:

1. Backed up the database (even though it might be compromised)
2. Downloaded all uploaded media files
3. Saved the wp-config.php (to get database credentials)
4. Nuked everything else

Backup first

mysqldump -u dbuser -p dbname > backup_$(date +%Y%m%d_%H%M%S).sql

Fresh WordPress install

wget https://wordpress.org/latest.tar.gz

tar -xzf latest.tar.gz

Database Surgery
The database had malicious entries in these tables:

`wp_options` (autoload hooks)
`wp_posts` (spam content injected)
`wp_users` (unknown admin accounts)

Cleaned them manually. Yes, manually. Running automated scripts on a compromised database is asking for trouble.

-- Remove suspicious admin users

SELECT * FROM wp_users WHERE user_login NOT IN ('known_admin_1', 'known_admin_2');

DELETE FROM wp_users WHERE ID = [suspicious_id];

-- Check for injected JavaScript in posts

SELECT ID, post_title FROM wp_posts

WHERE post_content LIKE '%<script%'

OR post_content LIKE '%iframe%';

## Hour 3: Hardening & Recovery (05:00 - 06:00)

The Rebuild
Fresh WordPress core, clean database, restored media files. Now comes the part most people skip: actually securing the thing.
File Permissions That Make Sense

Directories: 755

find /var/www/html -type d -exec chmod 755 {} \;

Files: 644

find /var/www/html -type f -exec chmod 644 {} \;

wp-config.php: 440

chmod 440 /var/www/html/wp-config.php

chown www-data:www-data /var/www/html/wp-config.php
Disable File Editing
Added to wp-config.php:

define('DISALLOW_FILE_EDIT', true);

define('DISALLOW_FILE_MODS', true);

No more editing themes from the admin panel. If you need to update something, do it through SFTP like a proper developer.
Web Application Firewall
Configured ModSecurity with OWASP rules. Basic stuff:

apt-get install libapache2-mod-security2

cp /etc/modsecurity/modsecurity.conf-recommended /etc/modsecurity/modsecurity.conf

Changed SecRuleEngine DetectionOnly to SecRuleEngine On.

The Forgotten Hero: Fail2Ban
Set up Fail2Ban to block brute force attempts:

/etc/fail2ban/jail.local

[wordpress]

enabled = true

filter = wordpress

logpath = /var/log/auth.log

maxretry = 3

bantime = 3600

## Hour 4: Automation & Insurance (06:00 - 07:00)

Automated Backups That Actually Work
Wrote a bash script for daily backups to remote storage:

!/bin/bash

TIMESTAMP=$(date +%Y%m%d_%H%M%S)

BACKUP_DIR="/backups"

Database

mysqldump -u user -ppassword database > $BACKUP_DIR/db_$TIMESTAMP.sql

Files

tar -czf $BACKUP_DIR/files_$TIMESTAMP.tar.gz /var/www/html

Send to remote (S3, or whatever)

aws s3 cp $BACKUP_DIR/ s3://your-bucket/backups/ --recursive

Keep only last 7 days locally

find $BACKUP_DIR -type f -mtime +7 -delete

Added it to crontab: `0 2 * * * /home/scripts/backup.sh`

Monitoring Setup
Installed basic monitoring so next time we catch things early:

Uptime monitoring

curl -X POST https://cronitor.io/api/monitors \

-H "Content-Type: application/json" \

-d '{"name": "client-site", "url": "https://example.com"}'

File integrity monitoring

apt-get install aide

aide --init
SSL & Security Headers
Fresh SSL certificate with Let's Encrypt:

certbot --nginx -d example.com -d www.example.com

Added security headers to nginx:

add_header X-Frame-Options "SAMEORIGIN" always;

add_header X-Content-Type-Options "nosniff" always;

add_header X-XSS-Protection "1; mode=block" always;

add_header Referrer-Policy "strict-origin-when-cross-origin" always;

## The Aftermath
By 7AM, site was back online. Clean, secured, monitored.

What the client saw: Their site back up, faster than before, with new security measures.

What they didn't see: The 4 hours of SSH sessions, database queries, and three cups of coffee.

## Lessons From The Trenches

### What Worked

- Having a clear mental checklist for compromises
- Not trusting anything on a compromised system
- Taking backups before any action (even if you think you don't need them)
- Hardening during recovery, not after

### What I'd Do Different
- Should have set up monitoring earlier (obviously)
- Could have automated the cleanup scripts better
- Next time: keep a USB drive with common tools ready

### Prevention Is Cheaper Than Cure
After this incident, I set up these things for all client sites:

- Weekly automated backups (tested restores monthly)
- Security plugins with proper configuration
- Update automation for core/plugins/themes
- File integrity monitoring
- Login attempt limiting

### The Technical Stack Behind This Recovery
- OS: Ubuntu 20.04 LTS
- Web Server: Nginx 1.18
- Database: MySQL 8.0
- Backup Storage: AWS S3
- Monitoring: Uptime Robot + custom bash scripts
- Security: ModSecurity, Fail2Ban, Cloudflare WAF

## Final Thoughts
Emergency recoveries are stressful. Your hands shake a bit when you're running rm -rf on a production server at 5AM. But this is what separates someone who just "knows WordPress" from someone who actually understands infrastructure.

The client was happy. The site survived. And I learned (again) that keeping systems updated and monitored is way easier than 4-hour emergency sessions.

Now I keep this checklist printed and stuck to my monitor. Because there will be a next time. There's always a next time.

------

Time taken: 4 hours
Coffee consumed: 3 cups
Client panic level: Reduced from 10/10 to 2/10
Would I do it again: Absolutely. But let's try to avoid it, yeah?