DEV Community

Cover image for Emergency Server Recovery: A 4-Hour Race Against Time
hello world_leo
hello world_leo

Posted on

Emergency Server Recovery: A 4-Hour Race Against Time

#linux #security #wordpress

The 3AM Wake-Up Call

You know that feeling when your phone buzzes at an ungodly hour and your stomach drops? That was me, staring at a frantic message: "Site is showing weird content. Help!"

I grabbed my laptop. The WordPress site was serving pharmaceutical spam to visitors. Classic compromise. The clock started ticking.

Hour 1: Damage Assessment (03:00 - 04:00)

First rule of server emergencies: don't panic, but move fast.
What I Did First
SSH into the server, check if I still had access. Good news: credentials still worked. Bad news: everything else.

#Check recent file modifications

find /var/www/html -type f -mtime -7 -ls | head -20
Enter fullscreen mode Exit fullscreen mode

Tons of suspicious PHP files scattered everywhere. The wp-content/uploads folder was full of backdoors. Someone had gotten in through an outdated plugin, probably.
Quick Isolation
Pulled the site offline with a maintenance page. Better to show "down for maintenance" than spam pills to your visitors.

#Quick nginx block

location / {

    return 503;

}
Enter fullscreen mode Exit fullscreen mode

Took a snapshot of everything before touching anything. You need evidence, and you might need to rollback if things go sideways.

Hour 2: The Cleanup (04:00 - 05:00)

Finding the Entry Point
Checked Apache/Nginx logs for unusual POST requests:

grep -i "POST" /var/log/nginx/access.log | grep -E "\.(php|asp|jsp)" | tail -100
Enter fullscreen mode Exit fullscreen mode

Found it. An old contact form plugin with a known vulnerability. They uploaded a shell through a file upload field that wasn't properly validated.
Nuclear Option with Surgical Precision
Here's the thing about compromised WordPress sites: you can't trust anything. But you also can't just delete everything because you need the data.

My approach:

  1. Backed up the database (even though it might be compromised)
  2. Downloaded all uploaded media files
  3. Saved the wp-config.php (to get database credentials)
  4. Nuked everything else 
#### Backup first

mysqldump -u dbuser -p dbname > backup_$(date +%Y%m%d_%H%M%S).sql

#### Fresh WordPress install

wget https://wordpress.org/latest.tar.gz

tar -xzf latest.tar.gz
Enter fullscreen mode Exit fullscreen mode

Database Surgery
The database had malicious entries in these tables:

wp_options (autoload hooks)
wp_posts (spam content injected)
wp_users (unknown admin accounts)

Cleaned them manually. Yes, manually. Running automated scripts on a compromised database is asking for trouble.

-- Remove suspicious admin users

SELECT * FROM wp_users WHERE user_login NOT IN ('known_admin_1', 'known_admin_2');

DELETE FROM wp_users WHERE ID = [suspicious_id];

-- Check for injected JavaScript in posts

SELECT ID, post_title FROM wp_posts 

WHERE post_content LIKE '%<script%' 

OR post_content LIKE '%iframe%';
Enter fullscreen mode Exit fullscreen mode

Hour 3: Hardening & Recovery (05:00 - 06:00)

The Rebuild
Fresh WordPress core, clean database, restored media files. Now comes the part most people skip: actually securing the thing.
File Permissions That Make Sense

# Directories: 755

find /var/www/html -type d -exec chmod 755 {} \;

# Files: 644

find /var/www/html -type f -exec chmod 644 {} \;

# wp-config.php: 440

chmod 440 /var/www/html/wp-config.php

chown www-data:www-data /var/www/html/wp-config.php
Disable File Editing
Added to wp-config.php:

define('DISALLOW_FILE_EDIT', true);

define('DISALLOW_FILE_MODS', true);
Enter fullscreen mode Exit fullscreen mode

No more editing themes from the admin panel. If you need to update something, do it through SFTP like a proper developer.
Web Application Firewall
Configured ModSecurity with OWASP rules. Basic stuff:

apt-get install libapache2-mod-security2

cp /etc/modsecurity/modsecurity.conf-recommended /etc/modsecurity/modsecurity.conf
Enter fullscreen mode Exit fullscreen mode

Changed SecRuleEngine DetectionOnly to SecRuleEngine On.

The Forgotten Hero: Fail2Ban
Set up Fail2Ban to block brute force attempts:

# /etc/fail2ban/jail.local

[wordpress]

enabled = true

filter = wordpress

logpath = /var/log/auth.log

maxretry = 3

bantime = 3600
Enter fullscreen mode Exit fullscreen mode

Hour 4: Automation & Insurance (06:00 - 07:00)

Automated Backups That Actually Work
Wrote a bash script for daily backups to remote storage:

#!/bin/bash

TIMESTAMP=$(date +%Y%m%d_%H%M%S)

BACKUP_DIR="/backups"

# Database

mysqldump -u user -ppassword database > $BACKUP_DIR/db_$TIMESTAMP.sql

# Files

tar -czf $BACKUP_DIR/files_$TIMESTAMP.tar.gz /var/www/html

# Send to remote (S3, or whatever)

aws s3 cp $BACKUP_DIR/ s3://your-bucket/backups/ --recursive

# Keep only last 7 days locally

find $BACKUP_DIR -type f -mtime +7 -delete
Enter fullscreen mode Exit fullscreen mode

Added it to crontab: 0 2 * * * /home/scripts/backup.sh

Monitoring Setup
Installed basic monitoring so next time we catch things early:

# Uptime monitoring

curl -X POST https://cronitor.io/api/monitors \

  -H "Content-Type: application/json" \

  -d '{"name": "client-site", "url": "https://example.com"}'

# File integrity monitoring

apt-get install aide

aide --init
SSL & Security Headers
Fresh SSL certificate with Let's Encrypt:

certbot --nginx -d example.com -d www.example.com

Added security headers to nginx:

add_header X-Frame-Options "SAMEORIGIN" always;

add_header X-Content-Type-Options "nosniff" always;

add_header X-XSS-Protection "1; mode=block" always;

add_header Referrer-Policy "strict-origin-when-cross-origin" always;
Enter fullscreen mode Exit fullscreen mode

The Aftermath

By 7AM, site was back online. Clean, secured, monitored.

What the client saw: Their site back up, faster than before, with new security measures.

What they didn't see: The 4 hours of SSH sessions, database queries, and three cups of coffee.

Lessons From The Trenches

What Worked

  • Having a clear mental checklist for compromises
  • Not trusting anything on a compromised system
  • Taking backups before any action (even if you think you don't need them)
  • Hardening during recovery, not after

What I'd Do Different

  • Should have set up monitoring earlier (obviously)
  • Could have automated the cleanup scripts better
  • Next time: keep a USB drive with common tools ready

Prevention Is Cheaper Than Cure

After this incident, I set up these things for all client sites:

  • Weekly automated backups (tested restores monthly)
  • Security plugins with proper configuration
  • Update automation for core/plugins/themes
  • File integrity monitoring
  • Login attempt limiting

The Technical Stack Behind This Recovery

  • OS: Ubuntu 20.04 LTS
  • Web Server: Nginx 1.18
  • Database: MySQL 8.0
  • Backup Storage: AWS S3
  • Monitoring: Uptime Robot + custom bash scripts
  • Security: ModSecurity, Fail2Ban, Cloudflare WAF

Final Thoughts

Emergency recoveries are stressful. Your hands shake a bit when you're running rm -rf on a production server at 5AM. But this is what separates someone who just "knows WordPress" from someone who actually understands infrastructure.

The client was happy. The site survived. And I learned (again) that keeping systems updated and monitored is way easier than 4-hour emergency sessions.

Now I keep this checklist printed and stuck to my monitor. Because there will be a next time. There's always a next time.

Time taken: 4 hours
Coffee consumed: 3 cups
Client panic level: Reduced from 10/10 to 2/10
Would I do it again: Absolutely. But let's try to avoid it, yeah?

Top comments (0)