DEV Community

Cover image for We've Been Hacked!
StĂŠphan Zych
StĂŠphan Zych

Posted on • Edited on

We've Been Hacked!

🏴‍☠️ We’ve Been Boarded!

Billions of blistering barnacles! One of our ships has been boarded.

A calm, professional, pirate‑approved guide to surviving a security incident without losing the ship, the cargo, or your sanity.

Purpose of This Logbook

This document is an incident‑response playbook for web applications in general.

While command snippets may reference common tools found in many environments (Linux servers, web servers, databases, containers), the procedure itself is deliberately stack‑agnostic and applies to any traditional web application stack.

The intent is to describe what must be done and in which order; adapt the commands to your vessel, sails, and seas.

Its goals are to:

  • De‑escalate stressful situations (for clients and developers)
  • Preserve evidence for technical or legal needs
  • Protect user and client data
  • Restore service safely
  • Ensure we almost never lose data thanks to modern practices

Golden Rule of the Seas: Panic sinks ships. Procedure keeps them afloat.

🧭 The Pirate Code (Read Before Acting)

  1. Stop the bleeding before swabbing the deck
  2. Preserve evidence before fixing anything
  3. Assume the ship is compromised until proven otherwise
  4. Restore from trusted charts, don’t patch rotten planks
  5. Say only what you know — speculation feeds mutiny

⚓ Phase 1 — Freeze the Moment (Snapshots & Evidence)

Goal: Capture everything before the winds shift.

1.1 Open an Incident Chest 

INCIDENT_DIR="/quarantine/incident-$(date +%F-%H%M)"
mkdir -p "$INCIDENT_DIR"
chmod 700 "$INCIDENT_DIR"
Enter fullscreen mode Exit fullscreen mode

This chest is read‑only evidence. Never clean or modify it.

1.2 Secure the Cargo Hold (Application Files) 

rsync -aHAX --numeric-ids \
  /var/www/ "$INCIDENT_DIR/www/"
Enter fullscreen mode Exit fullscreen mode

Why:

  • Gives a forensic snapshot
  • Protects you from accidental data loss
  • Buys time to think clearly

1.3 Bottle the Rum (Database Backup)

MySQL / MariaDB

mysqldump --single-transaction --routines --events \
  --all-databases > "$INCIDENT_DIR/db.sql"
Enter fullscreen mode Exit fullscreen mode

PostgreSQL

pg_dumpall > "$INCIDENT_DIR/db.sql"
Enter fullscreen mode Exit fullscreen mode

1.4 Capture the Ship’s Logbooks (Logs) 

cp -a /var/log/nginx "$INCIDENT_DIR/"
cp -a /var/log/auth.log "$INCIDENT_DIR/"
journalctl --since "30 days ago" > "$INCIDENT_DIR/journal.log"
Enter fullscreen mode Exit fullscreen mode

Logs are often more valuable than code when storms reach lawyers.

1.5 Note the Crew & Rigging (System State) 

ps auxfw > "$INCIDENT_DIR/processes.txt"
ss -plant > "$INCIDENT_DIR/network.txt"
crontab -l > "$INCIDENT_DIR/crontab-root.txt"
Enter fullscreen mode Exit fullscreen mode

1.6 Preserve the Captain’s Orders (Command History)

Shell history can reveal attacker activity or emergency commands run during the incident.

Capture command histories before logging out or restarting shells.

# Root history
cp -a /root/.bash_history "$INCIDENT_DIR/bash_history_root" 2>/dev/null || true

# Current user history
cp -a ~/.bash_history "$INCIDENT_DIR/bash_history_user" 2>/dev/null || true
Enter fullscreen mode Exit fullscreen mode

If other shells are in use (zsh, fish), capture their history files as well.

Why this matters:

  • attackers sometimes leave traces via interactive shells
  • helps reconstruct timelines
  • protects operators from "who ran what" uncertainty
ps auxfw > "$INCIDENT_DIR/processes.txt"
ss -plant > "$INCIDENT_DIR/network.txt"
crontab -l > "$INCIDENT_DIR/crontab-root.txt"
Enter fullscreen mode Exit fullscreen mode

🚧 Phase 2 — Containment (Batten Down the Hatches)

Goal: Stop further damage while looking calm and in control.

2.1 Raise the Maintenance Flag (Maintenance Mode)

Serve a static maintenance page while the crew works below deck.

How this is done depends on your stack:

  • web‑server level redirect
  • application maintenance mode
  • load‑balancer / gateway rule

Principle: stop user traffic cleanly while presenting a calm, professional message.

Client message should always be:

  • calm
  • short
  • confident

2.2 Full Quarantine (If the Kraken Is Active) 

iptables -I INPUT -p tcp --dport 80 -j DROP
iptables -I INPUT -p tcp --dport 443 -j DROP
Enter fullscreen mode Exit fullscreen mode

Use only when:

  • live exploitation is suspected
  • data exfiltration is possible

🗣️ Phase 3 — Control the Message

Loose tongues sink ships.

Internal

  • “Incident contained”
  • “Evidence preserved”
  • “Investigation underway”

Client-facing

  • No speculation
  • No blame
  • Emphasize protection of data and professionalism

🔍 Phase 4 — Find How They Boarded

We don’t just clean the mess — we find the hole in the hull.

4.1 Timeline Recon 

find /var/www -type f -mtime -7 -ls > recent-files.txt
Enter fullscreen mode Exit fullscreen mode

4.2 Black Magic Detection (Code Scanning) 

rg -i "(eval\(|base64_decode|shell_exec|system\(|passthru|assert\(|gzinflate)" \
  /var/www
Enter fullscreen mode Exit fullscreen mode

4.3 Access Log Recon

Review web access logs for suspicious activity.

# Adapt path and tooling to your environment
zgrep -i "POST\|upload\|\.php" /var/log/*access*
Enter fullscreen mode Exit fullscreen mode

Common boarding points:

  • file uploads
  • admin endpoints
  • executable files in upload directories

4.4 Hidden Persistence 

crontab -l
ls -lah /etc/cron*
Enter fullscreen mode Exit fullscreen mode

🔑 Phase 5 — Roll All Credentials

Assume the enemy copied the keys.

Rotate everything:

  • Database users
  • Application secrets (.env)
  • API keys
  • OAuth tokens
  • JWT secrets
  • Admin passwords
  • SSH keys

Example:

ALTER USER 'appuser' IDENTIFIED BY 'NEW_STRONG_PASSWORD';
FLUSH PRIVILEGES;
Enter fullscreen mode Exit fullscreen mode

🧱 Phase 6 — Restore From Trusted Charts

Never patch a rotten plank.

6.1 Code from Version Control Only 

git fetch --all
git reset --hard origin/main
git clean -fdx
Enter fullscreen mode Exit fullscreen mode

Modern Rule: Anything not in Git is considered lost at sea.

6.2 Reinstall Dependencies 

composer install --no-dev --optimize-autoloader
npm ci --production
Enter fullscreen mode Exit fullscreen mode

6.3 Carefully Reintroduce User Cargo

Restore only:

  • media uploads
  • verified database rows

Never restore:

  • executable files
  • modified core files

🐳 Phase 7 — Containerized Ships

Containers are disposable rowboats, not galleons.

If your application runs in containers:

  • Do not attempt in‑place cleanup
  • Assume images and runtime are tainted

If compromised:

# Stop and destroy
container stop <container>
container rm <container>

# Rebuild from trusted sources
container build --no-cache -t app:clean .
container run ...
Enter fullscreen mode Exit fullscreen mode

Notes:

  • Distroless or minimal images offer fewer attack surfaces
  • Forensics usually happen on the host, volumes, and CI pipeline
  • Rotate all secrets
  • Rebuild images from known‑good sources

🔐 Phase 8 — Guarding the Crew’s Data

8.1 Encryption at Rest

  • Encrypted disks (LUKS / cloud KMS)
  • Encrypted backups: 
gpg --encrypt --recipient security@example.com db.sql
Enter fullscreen mode Exit fullscreen mode

8.2 Encryption in the App

  • Passwords: bcrypt or argon2id
  • Sensitive user data: application‑level encryption

  • Keys:

    • never committed
    • rotated
    • stored in secret managers

8.3 Backup Strategy (Modern Seamanship)

Back up

  • databases
  • user uploads
  • secrets (encrypted)

Do NOT back up

  • vendor directories
  • build artifacts
  • containers

Rules:

  • off‑site
  • encrypted
  • versioned
  • restore tested regularly

✅ Phase 9 — Final Inspection 

clamscan -r /var/www
rg "(eval\(|base64_decode)" /var/www
Enter fullscreen mode Exit fullscreen mode
  • re‑enable traffic
  • monitor logs closely
  • watch for unexpected writes

📜 Phase 10 — Captain’s Report (Non‑Optional)

Include:

  • timeline
  • entry point (or unknown)
  • affected data scope
  • credentials rotated
  • preventive actions taken

This report protects:

  • the client
  • the crew
  • future captains

🍻 Final Words from the Captain

A good incident response doesn’t prove the ship was unsinkable.

It proves:

  • discipline
  • professionalism
  • and that even when boarded…

you stayed in command of the seas. Celebrate with a liter of rum. 🥃

Top comments (0)