We've Been Hacked!

Stéphan Zych — Tue, 29 Jul 2025 08:10:37 +0000

🏴‍☠️ We’ve Been Boarded!

Billions of blistering barnacles! One of our ships has been boarded.

A calm, professional, pirate‑approved guide to surviving a security incident without losing the ship, the cargo, or your sanity.

Purpose of This Logbook

This document is an incident‑response playbook for web applications in general.

While command snippets may reference common tools found in many environments (Linux servers, web servers, databases, containers), the procedure itself is deliberately stack‑agnostic and applies to any traditional web application stack.

The intent is to describe what must be done and in which order; adapt the commands to your vessel, sails, and seas.

Its goals are to:

De‑escalate stressful situations (for clients and developers)
Preserve evidence for technical or legal needs
Protect user and client data
Restore service safely
Ensure we almost never lose data thanks to modern practices

Golden Rule of the Seas: Panic sinks ships. Procedure keeps them afloat.

🧭 The Pirate Code (Read Before Acting)

Stop the bleeding before swabbing the deck
Preserve evidence before fixing anything
Assume the ship is compromised until proven otherwise
Restore from trusted charts, don’t patch rotten planks
Say only what you know — speculation feeds mutiny

⚓ Phase 1 — Freeze the Moment (Snapshots & Evidence)

Goal: Capture everything before the winds shift.

1.1 Open an Incident Chest

INCIDENT_DIR="/quarantine/incident-$(date +%F-%H%M)"
mkdir -p "$INCIDENT_DIR"
chmod 700 "$INCIDENT_DIR"

This chest is read‑only evidence. Never clean or modify it.

1.2 Secure the Cargo Hold (Application Files)

rsync -aHAX --numeric-ids \
  /var/www/ "$INCIDENT_DIR/www/"

Why:

Gives a forensic snapshot
Protects you from accidental data loss
Buys time to think clearly

1.3 Bottle the Rum (Database Backup)

MySQL / MariaDB

mysqldump --single-transaction --routines --events \
  --all-databases > "$INCIDENT_DIR/db.sql"

PostgreSQL

pg_dumpall > "$INCIDENT_DIR/db.sql"

1.4 Capture the Ship’s Logbooks (Logs)

cp -a /var/log/nginx "$INCIDENT_DIR/"
cp -a /var/log/auth.log "$INCIDENT_DIR/"
journalctl --since "30 days ago" > "$INCIDENT_DIR/journal.log"

Logs are often more valuable than code when storms reach lawyers.

1.5 Note the Crew & Rigging (System State)

ps auxfw > "$INCIDENT_DIR/processes.txt"
ss -plant > "$INCIDENT_DIR/network.txt"
crontab -l > "$INCIDENT_DIR/crontab-root.txt"

1.6 Preserve the Captain’s Orders (Command History)

Shell history can reveal attacker activity or emergency commands run during the incident.

Capture command histories before logging out or restarting shells.

# Root history
cp -a /root/.bash_history "$INCIDENT_DIR/bash_history_root" 2>/dev/null || true

# Current user history
cp -a ~/.bash_history "$INCIDENT_DIR/bash_history_user" 2>/dev/null || true

If other shells are in use (zsh, fish), capture their history files as well.

Why this matters:

attackers sometimes leave traces via interactive shells
helps reconstruct timelines
protects operators from "who ran what" uncertainty

ps auxfw > "$INCIDENT_DIR/processes.txt"
ss -plant > "$INCIDENT_DIR/network.txt"
crontab -l > "$INCIDENT_DIR/crontab-root.txt"

🚧 Phase 2 — Containment (Batten Down the Hatches)

Goal: Stop further damage while looking calm and in control.

2.1 Raise the Maintenance Flag (Maintenance Mode)

Serve a static maintenance page while the crew works below deck.

How this is done depends on your stack:

web‑server level redirect
application maintenance mode
load‑balancer / gateway rule

Principle: stop user traffic cleanly while presenting a calm, professional message.

Client message should always be:

calm
short
confident

2.2 Full Quarantine (If the Kraken Is Active)

iptables -I INPUT -p tcp --dport 80 -j DROP
iptables -I INPUT -p tcp --dport 443 -j DROP

Use only when:

live exploitation is suspected
data exfiltration is possible

🗣️ Phase 3 — Control the Message

Loose tongues sink ships.

Internal

“Incident contained”
“Evidence preserved”
“Investigation underway”

Client-facing

No speculation
No blame
Emphasize protection of data and professionalism

🔍 Phase 4 — Find How They Boarded

We don’t just clean the mess — we find the hole in the hull.

4.1 Timeline Recon

find /var/www -type f -mtime -7 -ls > recent-files.txt

4.2 Black Magic Detection (Code Scanning)

rg -i "(eval\(|base64_decode|shell_exec|system\(|passthru|assert\(|gzinflate)" \
  /var/www

4.3 Access Log Recon

Review web access logs for suspicious activity.

# Adapt path and tooling to your environment
zgrep -i "POST\|upload\|\.php" /var/log/*access*

Common boarding points:

file uploads
admin endpoints
executable files in upload directories

4.4 Hidden Persistence

crontab -l
ls -lah /etc/cron*

🔑 Phase 5 — Roll All Credentials

Assume the enemy copied the keys.

Rotate everything:

Database users
Application secrets (.env)
API keys
OAuth tokens
JWT secrets
Admin passwords
SSH keys

Example:

ALTER USER 'appuser' IDENTIFIED BY 'NEW_STRONG_PASSWORD';
FLUSH PRIVILEGES;

🧱 Phase 6 — Restore From Trusted Charts

Never patch a rotten plank.

6.1 Code from Version Control Only

git fetch --all
git reset --hard origin/main
git clean -fdx

Modern Rule: Anything not in Git is considered lost at sea.

6.2 Reinstall Dependencies

composer install --no-dev --optimize-autoloader
npm ci --production

6.3 Carefully Reintroduce User Cargo

Restore only:

media uploads
verified database rows

Never restore:

executable files
modified core files

🐳 Phase 7 — Containerized Ships

Containers are disposable rowboats, not galleons.

If your application runs in containers:

Do not attempt in‑place cleanup
Assume images and runtime are tainted

If compromised:

# Stop and destroy
container stop <container>
container rm <container>

# Rebuild from trusted sources
container build --no-cache -t app:clean .
container run ...

Notes:

Distroless or minimal images offer fewer attack surfaces
Forensics usually happen on the host, volumes, and CI pipeline
Rotate all secrets
Rebuild images from known‑good sources

🔐 Phase 8 — Guarding the Crew’s Data

8.1 Encryption at Rest

Encrypted disks (LUKS / cloud KMS)
Encrypted backups:

gpg --encrypt --recipient security@example.com db.sql

8.2 Encryption in the App

Passwords: bcrypt or argon2id
Sensitive user data: application‑level encryption
Keys:
- never committed
- rotated
- stored in secret managers

8.3 Backup Strategy (Modern Seamanship)

Back up

databases
user uploads
secrets (encrypted)

Do NOT back up

vendor directories
build artifacts
containers

Rules:

off‑site
encrypted
versioned
restore tested regularly

✅ Phase 9 — Final Inspection

clamscan -r /var/www
rg "(eval\(|base64_decode)" /var/www

re‑enable traffic
monitor logs closely
watch for unexpected writes

📜 Phase 10 — Captain’s Report (Non‑Optional)

Include:

timeline
entry point (or unknown)
affected data scope
credentials rotated
preventive actions taken

This report protects:

the client
the crew
future captains

🍻 Final Words from the Captain

A good incident response doesn’t prove the ship was unsinkable.

It proves:

discipline
professionalism
and that even when boarded…

you stayed in command of the seas. Celebrate with a liter of rum. 🥃

DEV Community: Stéphan Zych