DEV Community: manuelerastus

We rebuilt a compromised server. Six hours later it was infected again.

manuelerastus — Wed, 04 Nov 2026 08:00:59 +0000

Rebuilding a compromised server from scratch is the textbook response. It is also not enough on its own. This is the story of a website server I rebuilt cleanly, watched get re-infected within six hours, and the one command that explained why.

Details that would identify the organisation, hosts or attacker infrastructure have been removed.

The first compromise

The server was a small Linux container running a marketing website: a Next.js frontend and a Strapi CMS, both managed by PM2. During a routine check I found it in bad shape:

A cryptominer running from /tmp at 200% CPU for about five days, talking to a mining pool on port 33333.
A world-writable binary in /usr/sbin registered as a systemd service, holding a connection open on port 443 (almost certainly command-and-control).
A root crontab that recreated a hidden UID-0 user every 30 minutes, restarted a hidden Docker container, and every midnight fetched a fresh script from the internet and piped it to /bin/sh using whichever of curl, wget, python3 or perl was available.

That last item settled the response. Any in-place cleanup would have been silently undone at 00:00. So the plan was: contain, preserve evidence, rebuild.

Contain. I took the container's network interface down from inside. On Proxmox, pct exec and pct pull go through the host, so I could still work on it with no network.
Preserve. I pulled the application data out through the host and took a full forensic backup of the container, malware included, kept offline.
Rotate. I rotated every secret that had lived on the box: the database role password and all the CMS signing keys.
Rebuild. I destroyed the container and rebuilt it with the same address. It came back without Docker-in-container support (which the site never needed), with key-only SSH and a host firewall.

Later the same day I hardened it further:

No root at runtime. The app moved from root's PM2 to an unprivileged app user, and the user's passwordless sudo rule was removed.
Nothing executes from temp. /tmp and /var/tmp became noexec tmpfs mounts, set at the hypervisor level because an unprivileged container can't remount them itself.
Egress filtering. Outbound traffic was default-deny, with an allow-list of DNS, HTTP/S, NTP, mail and the one database endpoint.
Integrity checks. A 15-minute timer now watches for extra UID-0 users, new crontabs, and Docker binaries reappearing.

The site came back up and every check passed.

Two days later

Two days later I looked again. A miner was running, this time from the app user's home directory, with a second one staged in /tmp and waiting.

The app user's crontab had the exact same persistence pattern as before: create the hidden root user, start the hidden container. The system log showed that cron entry had been firing every 30 minutes since roughly six hours after the rebuild.

But this time it had achieved almost nothing:

The hidden root user was never created. app had no sudo left, so every attempt failed.
The hidden container never started. There was no Docker on the box.
The staged miner in /tmp could not run, because /tmp was noexec.
No SSH sessions appeared in the login records at all.

The hardening had done its job: the attacker was stuck at the application user's privilege level. But something was clearly still letting them in.

Following the wrong lead first

My first theory was stolen credentials. Around the same time we had found malware on an admin workstation, so a reused host credential was plausible. It would also explain the missing login records, because commands run from the hypervisor don't appear in the container's login history.

I couldn't prove or disprove it from the logs. So I tested the other obvious theory: a poisoned npm dependency. Nothing in either node_modules tree had changed since the rebuild's install, and the only postinstall script was a well-known benign one.

The answer was one command away

Then I ran the check I should have run first:

npm audit

The deployed frontend was pinned to a Next.js version inside the affected range of GHSA-9qr9-h5gf-34mp: a critical, unauthenticated remote code execution in the React flight protocol.

That one finding explained everything:

Symptom	Explanation
No login records	It was an HTTP exploit, not a login
Compromise limited to the `app` user	That is exactly the privilege the Next.js process runs with
Reinfected about 6 hours after the rebuild	The rebuild redeployed the same vulnerable version, and scanners for a public RCE find exposed sites quickly

The rebuild had cleaned the server and then reinstalled the hole.

The fix

I bumped Next.js to a patched version on the live deployment, rebuilt the app, and confirmed with npm audit that the critical finding was gone. The CMS had its own critical advisories (an SQL injection among them), so it got the same treatment.

One trap: a hotfix on the server is not a fix in the repository. Until the version bump is committed, the next pipeline deploy quietly reinstalls the vulnerable version. That handoff to the developers matters as much as the patch.

What I took away

A clean rebuild of a vulnerable app is a vulnerable app. Before redeploying after an incident, audit the exact versions you are about to ship, not just the server underneath them.
Check the cheap, likely explanation first. A known public RCE in an internet-facing framework is far more likely than an attacker with stolen keys, and npm audit takes seconds.
Defence in depth bought me time. A non-root runtime, no sudo, no Docker, noexec temp and egress filtering turned a second root compromise into a contained, app-level nuisance.
Make the persistence pattern your detection. Attackers reuse playbooks. The same crontab lines showed up twice. Watching for them (new UID-0 users, unexpected crontabs) is cheap and specific.
Alert on boring signals. The first miner ran at 200% CPU for about five days unnoticed. A sustained-CPU alert would have caught it on day one.

If you run Next.js or Strapi on the open internet, run npm audit against what is actually deployed today. Not what's in your repo: what's running.

Your FortiGate static route is perfect. SD-WAN is ignoring it.

manuelerastus — Wed, 04 Nov 2026 07:57:59 +0000

I lost the better part of three days to one FortiOS behaviour while connecting two sites over IPsec. It bit me four times: three different destination subnets, then once more in the return direction. Each time the firewall policy, the phase 2 selectors and the static route all looked correct, and traffic still vanished or leaked out to the internet.

This post is the explanation I wish I'd had, and the diagnostic order that finds it in minutes. Addresses and interface names are changed.

The setup

Two sites, each behind a FortiGate, joined by route-based IPsec tunnels. Site A has two internet links bundled into SD-WAN. Someone had added an SD-WAN rule to load-balance internet traffic across both:

config system sdwan
    config service
        edit 1
            set name "wan_load_balance"
            set mode load-balance
            set dst "all"
            set src "all"
            ...
        next
    end
end

Then I added a new remote subnet, 10.2.0.0/24, behind the tunnel to-siteB:

config router static
    edit 0
        set dst 10.2.0.0 255.255.255.0
        set device "to-siteB"
    next
end

Firewall policy allowing LAN to the tunnel: done. Phase 2 selectors covering both subnets: done. Tunnel up: yes.

From a real host on the LAN, nothing reached Site B.

Why the static route loses

FortiOS doesn't go straight to the routing table. Roughly, the forwarding decision checks, in order:

Policy routes (config router policy)
SD-WAN service rules
The routing table (FIB)

SD-WAN service rules are policy routes under the hood. A rule matching src all / dst all therefore claims every forwarded packet before the routing table is consulted. However specific your static route is, it never gets a vote.

The packet goes out of an SD-WAN member link, source-NATed to that link's public IP, towards a private address on the internet. It doesn't arrive anywhere, and nothing logs a "deny", because by its own rules nothing was denied.

Note that the built-in implicit SD-WAN rule follows the routing table. The trap is an explicit catch-all rule, which many deployments add for load balancing.

Why it looked fine at first

My first test was a ping from the FortiGate itself:

execute ping-options source 10.1.0.1
execute ping 10.2.0.10

It worked. Self-originated traffic isn't forwarded through the same policy-route path as traffic arriving from a LAN host, so it takes the static route and goes into the tunnel.

Lesson one: test with real traffic from a real host. A ping from the firewall proves the tunnel works, not that forwarding works.

The fix, part one: outbound

Add a policy route ahead of the SD-WAN rules for the new destination:

config router policy
    edit 0
        set input-device "lan"
        set src "10.1.0.0/255.255.255.0"
        set dst "10.2.0.0/255.255.255.0"
        set output-device "to-siteB"
    next
end

Traffic from the LAN now entered the tunnel. The remote side saw the packets arrive and replied.

Replies still never reached the LAN host.

The fix, part two: the return path

This is the one that cost the most time. A reply comes out of the tunnel, gets decrypted, and then needs a second routing decision to reach the directly connected LAN. That decision goes through the same order, and the catch-all SD-WAN rule claims it too.

The symptoms looked like a hardware fault:

Tunnel counters showed decrypted packets increasing.
The sniffer showed nothing leaving the LAN interface.
No session, no log entry.

diagnose debug flow finally showed what was happening:

Match policy routing id=...: to 10.1.0.25 via ifindex-<wan member>
find a route: gw-<isp gateway> via <wan member>
...
policy-0 is matched, act-drop
Denied by forward policy check (policy 0)

Policy 0 is FortiOS's implicit deny. The packet only reached it because routing had already chosen the WAN link for a reply meant for the LAN, and no firewall policy allows tunnel-to-WAN.

The fix is the mirror image:

config router policy
    edit 0
        set input-device "to-siteB"
        set src "10.2.0.0/255.255.255.0"
        set dst "10.1.0.0/255.255.255.0"
        set output-device "lan"
    next
end

Lesson two: every new tunnel destination needs a policy route in both directions. The return direction is not inherited.

The diagnostic order that finds it fast

Use a real host. Generate traffic from a LAN machine, not the firewall.
Sniff it.

   diagnose sniffer packet any "host 10.2.0.10" 4 30

If you see it leaving a WAN member with a NATed source instead of the tunnel, SD-WAN has it.

Check which policy routes are being hit.

   diagnose firewall proute list

A hit_count of 0 on your new rule means it never matches. Compare it with a working sibling rule.

Clear stale sessions before retesting. Offloaded sessions keep their old path until cleared:

   diagnose sys session filter dst 10.2.0.10
   diagnose sys session clear

Trace the flow. Still stuck? diagnose debug flow names the exact policy route and gateway chosen. It's easiest to capture from the GUI's debug flow tool, because streaming live debug over a non-interactive SSH session is unreliable.
Trust get over show. In a few edge cases show omitted fields that were set. get inside edit <id> gives the live values.

What didn't help

I tried these before finding the cause. None of them touch the routing decision, so none of them helped:

Disabling ASIC/NPU offload on the firewall policy
Disabling NPU offload on the phase 1 interface
Widening the firewall policy to all / all

If the sniffer shows the packet leaving the wrong interface, the problem is routing. Firewall policy changes won't fix it.

Takeaways

An SD-WAN rule matching dst all outranks every static route for forwarded traffic.
Add explicit policy routes for internal and tunnel destinations, in both directions.
Prove forwarding with traffic from a real host. A firewall-originated ping proves only the tunnel.
When a decrypted packet "disappears", trace the second routing decision before blaming the hardware.

Better still, scope the load-balancing rule to internet destinations in the first place, for example with a negated RFC 1918 destination group. Then internal traffic never matches it.