We rebuilt a compromised server. Six hours later it was infected again.

Rebuilding a compromised server from scratch is the textbook response. It is also not enough on its own. This is the story of a website server I rebuilt cleanly, watched get re-infected within six hours, and the one command that explained why.

Details that would identify the organisation, hosts or attacker infrastructure have been removed.

The first compromise

The server was a small Linux container running a marketing website: a Next.js frontend and a Strapi CMS, both managed by PM2. During a routine check I found it in bad shape:

• A cryptominer running from /tmp at 200% CPU for about five days, talking to a mining pool on port 33333.
• A world-writable binary in /usr/sbin registered as a systemd service, holding a connection open on port 443 (almost certainly command-and-control).
• A root crontab that recreated a hidden UID-0 user every 30 minutes, restarted a hidden Docker container, and every midnight fetched a fresh script from the internet and piped it to /bin/sh using whichever of curl, wget, python3 or perl was available.

That last item settled the response. Any in-place cleanup would have been silently undone at 00:00. So the plan was: contain, preserve evidence, rebuild.

1. Contain. I took the container's network interface down from inside. On Proxmox, pct exec and pct pull go through the host, so I could still work on it with no network.
2. Preserve. I pulled the application data out through the host and took a full forensic backup of the container, malware included, kept offline.
3. Rotate. I rotated every secret that had lived on the box: the database role password and all the CMS signing keys.
4. Rebuild. I destroyed the container and rebuilt it with the same address. It came back without Docker-in-container support (which the site never needed), with key-only SSH and a host firewall.

Later the same day I hardened it further:

• No root at runtime. The app moved from root's PM2 to an unprivileged app user, and the user's passwordless sudo rule was removed.
• Nothing executes from temp. /tmp and /var/tmp became noexec tmpfs mounts, set at the hypervisor level because an unprivileged container can't remount them itself.
• Egress filtering. Outbound traffic was default-deny, with an allow-list of DNS, HTTP/S, NTP, mail and the one database endpoint.
• Integrity checks. A 15-minute timer now watches for extra UID-0 users, new crontabs, and Docker binaries reappearing.

The site came back up and every check passed.

Two days later

Two days later I looked again. A miner was running, this time from the app user's home directory, with a second one staged in /tmp and waiting.

The app user's crontab had the exact same persistence pattern as before: create the hidden root user, start the hidden container. The system log showed that cron entry had been firing every 30 minutes since roughly six hours after the rebuild.

But this time it had achieved almost nothing:

• The hidden root user was never created. app had no sudo left, so every attempt failed.
• The hidden container never started. There was no Docker on the box.
• The staged miner in /tmp could not run, because /tmp was noexec.
• No SSH sessions appeared in the login records at all.

The hardening had done its job: the attacker was stuck at the application user's privilege level. But something was clearly still letting them in.

Following the wrong lead first

My first theory was stolen credentials. Around the same time we had found malware on an admin workstation, so a reused host credential was plausible. It would also explain the missing login records, because commands run from the hypervisor don't appear in the container's login history.

I couldn't prove or disprove it from the logs. So I tested the other obvious theory: a poisoned npm dependency. Nothing in either node_modules tree had changed since the rebuild's install, and the only postinstall script was a well-known benign one.

The answer was one command away

Then I ran the check I should have run first:

npm audit

The deployed frontend was pinned to a Next.js version inside the affected range of GHSA-9qr9-h5gf-34mp: a critical, unauthenticated remote code execution in the React flight protocol.

That one finding explained everything:

Symptom	Explanation
No login records	It was an HTTP exploit, not a login
Compromise limited to the app user	That is exactly the privilege the Next.js process runs with
Reinfected about 6 hours after the rebuild	The rebuild redeployed the same vulnerable version, and scanners for a public RCE find exposed sites quickly

The rebuild had cleaned the server and then reinstalled the hole.

The fix

I bumped Next.js to a patched version on the live deployment, rebuilt the app, and confirmed with npm audit that the critical finding was gone. The CMS had its own critical advisories (an SQL injection among them), so it got the same treatment.

One trap: a hotfix on the server is not a fix in the repository. Until the version bump is committed, the next pipeline deploy quietly reinstalls the vulnerable version. That handoff to the developers matters as much as the patch.

What I took away

1. A clean rebuild of a vulnerable app is a vulnerable app. Before redeploying after an incident, audit the exact versions you are about to ship, not just the server underneath them.
2. Check the cheap, likely explanation first. A known public RCE in an internet-facing framework is far more likely than an attacker with stolen keys, and npm audit takes seconds.
3. Defence in depth bought me time. A non-root runtime, no sudo, no Docker, noexec temp and egress filtering turned a second root compromise into a contained, app-level nuisance.
4. Make the persistence pattern your detection. Attackers reuse playbooks. The same crontab lines showed up twice. Watching for them (new UID-0 users, unexpected crontabs) is cheap and specific.
5. Alert on boring signals. The first miner ran at 200% CPU for about five days unnoticed. A sustained-CPU alert would have caught it on day one.

If you run Next.js or Strapi on the open internet, run npm audit against what is actually deployed today. Not what's in your repo: what's running.

Comments

[cryptostrapon]

Point 1 is the one teams skip most: the rebuild comes from the same lockfile that got them owned. An npm audit gate in the deploy pipeline stops the vulnerable version before it ever reaches the new box.