Security incidents often arrive quietly, without alarm bells or dramatic warnings. They sneak into production environments when we least expect them, on weekends, during moments of downtime, or when routine vigilance slips just a little.
This is the story of how my production VPS was compromised by attackers who deployed crypto-mining malware and persistent agents. What followed was a long learning process filled with investigation, cleanup attempts, unexpected failures, and finally, a deep dive into what true production-grade security really means.
A Sunday Evening Shock
It began innocently. On a calm Sunday evening, I opened my production app to check something. Except… it didn’t load. No error page. No timeout. Just a silent blank screen nothing except the slpash screen.
I assumed the server might be having a hiccup. Maybe a deployment issue or that core just again said I will not work. Maybe some error or the hosting service down (which is happening a lot recently) but I was so wrong.
My first thought today is sunday lets leave it for tomorrow but curiousity began in my mind so I got to my workstation and tried to SSH into the VPS. Connection timed out. Tried again. Timed out again I thought hmm... maybe connection failure or may the vps have been shutdown due to overdue bill I supposed. again I was so wrong.
After multiple attempts, the SSH prompt finally appeared. the machine was slow and sluggish like running ls was also taking 3 - 4 seconds that sluggishness became my first real clue
The machine wasn’t down, it was overwhelmed.
Once I logged in, the system felt painfully sluggish. Even typing commands felt like wading through molasses.
I ran:
top
And saw something that changed the tone of the night and runined the few hours of my weekend
The Bad Actor
A process named rysyslo\, located in /usr/bin/rsyslo, was consuming roughly 80% CPU and a significant amount of RAM and almost all of the disk space
I had never installed any binary by that name. Ubuntu the os, doesn’t ship such a file.
A quick chat and search confirmed it. It was malicious.
My server had been breached.
First Cleanup Attempt; Sense of a False Victory
I deleted the suspicious binary. Removed unfamiliar services. Killed the rogue processes. The load dropped. The system behaved normally. Everything felt fine. But deep down in my mind I knew that it may not have been gone with such simple attempts (spoiler alert: I was right) but still I left it as I thought I will backup the data tomorrow
The Next Day Things Get Worse, Not Better
The following morning, I again opned the app, still unavailable. Over 16 hours of downtime at this point.
SSH once again behaved erratically — refusing connections for minutes at a time. I even had to ask chat gpt to get me script to constantly try to get in using ssh because it just kept on timing out.
When I finally got in, the situation had escalated dramatically.
Running top revealed:
- xmrig — a notorious crypto-mining tool
- Multiple disguised binaries with random names
- Fake kernel threads impersonating system processes
- A mysterious program called monarx-agent consuming memory
- Node processes I had not launched
The first obvious solution that came to my mind was to kill them but I knew this was not going to do anything because they must have set backdoors and other ways to restart the service. The system was no longer merely “infected.” It was owned.
The Attackers’ Footprint
I still am very beginner in devops and linux systems even though I use it as my daily driver. I asked chat gpt how can I clean the system follwed the steps closely and this bought me some time to backup any server data that was there.
First I did basic scans to search for malicious services and found alive.service and lived.service these are too not created by the os and ofcourse not by me hence: malicious they both pointed to a script file located in the following locations:
/tmp/runnv/alive.sh
/tmp/runnv/lived.sh
Both services were configured to:
Restart=always
RestartSec=5s
These scripts continuously fetched, restored, and relaunched the miner — even seconds after deletion. This explained everything: No matter how many times I killed a process or removed a binary, the malicious services resurrected them.
This wasn’t a small compromise. It was a full-scale, multi-vector persistence setup.
Services Begin to Break
Over time, the machine became unstable:
- SSH stalled or dropped repeatedly
- System operations slowed to a crawl
- Memory usage spiked unpredictably
- System logs showed irregularities
- A fake agent (monarx-agent) reappeared no matter how many times it was removed
- The machine would barely stay functional long enough for basic commands
There was no longer any pretense of stability it became like the VPS was setup intentionally for mining bitcoin.
The attackers had established root-level foothold (this was due to my fault but more on this later), and the operating system had been modified in ways I could no longer trust.
My Response to the things
At this stage it was not about that how can I fix the server but the important server data that's on there should be extracted asap and the user data should be checked to see that it did not have a breech
I began systematically backing up:
- PostgreSQL database dumps using pgdump
- Application server files
- Media directories
- Configuration files
- User uploads
- Essential environment variables
- Reverse proxy configs
- Logs for analysis
Because the system was barely responsive, everything required workarounds,
For anything related to files I used rsync and copied the files to my local machine but this was not the usual way I had to write scripts to check the access of the ssh and then copy the files because the ssh connection kept on dropping.
Eventually, I succeed in the task every important asset was safely backed up.
After confirming data integrity and to my surprise the application data was untouched by the hackers they just wanted to use my machine to mine crypto, so I made the decision to shut down the compromised machine permanently and migrate to a fresh VPS instance because this was destroyed beyod repair.
Searching for the Entry Point
After all the chaos I was geniunly curous that how did they get access to my server, I never had my keys leaked nor the IP so I began looking into this.
I began by reviewing all forensic data:
- /var/log/auth.log
- SSH access attempts
- Systemd unit file timestamps
- Root-level process creation logs
- Cron jobs
- Unknown script origins
- Unfamiliar executables
- Running port lists
- Binaries in /usr/bin, /tmp, /dev/shm, /usr/share/updater
What I discovered:
I was right I did not had my keys leaked or something like that. The log only showed rejected SSH requests which I know happen because some people just leave everything to the default (I am one of those but not at this point)
Everything was good but annoyingly:
❗ There was no definitive entry point.
So where did they get in?
The prime suspect is the very recently disclosed React2Shell vulnerability (CVE-2025-55182) revealed just a day before the attack. This vulnerability allows remote code execution under certain misconfigured environments.
Web servers or exposed ports would have been particularly vulnerable. Another major factor:
And yes I was right I checked the nginx logs for the server and there I see someone forcefully did a post request to every endpoint they could've found and there might been vulnerable endpoints which gave them access to the machine.
Hostinger VPS firewall defaults to allow ALL inbound and outbound traffic.
Unlike cloud platforms with strict security groups (AWS, GCP), this VPS was wide open at the network level. this was the default thing I never touched.
This alone dramatically increases the attack surface.
The Biggest Lesson — Never Run Applications as root
One big mistake I had made knowingly that not to run apps with root privelleges that they can modify the system in anyway. My PostgreSQL server, which ran under a non-root system user, remained untouched by the attacker, I know they were not looking for it but in case then it would've still remained safe No tampered files. No altered configurations. No corrupt data. No suspicious access. Even though the system was compromised, the database was safe.
This incident reinforced a principle I will never ignore again:
Never run applications as root. Ever.
Give them the smallest permissions required to function and nothing more. not even modification outside the directory (unless your app has to access).
The Rebuild
After recovering and moving to a fresh VPS, I rebuilt the environment with proper security measures:
Firstly I ensured that I do not run the process as the root user I create users for each module like for the server for the frontend or for the api. NO compromise should be made in setting users up.
I enforced strict firewall rules only allow http and https ports to be expoed to the public because my app does run postgres but as an internal tool and the database should not be accisble publically all traffic is handled via nginx so other ports should not be open.
This wasn’t just “rebuilding a server.” It was rebuilding an entire philosophy about production infrastructure.
Final Thoughts
Security incidents can be unsettling, especially when a server is actively failing under the strain of malware and critical data must be preserved in a limited window of time. Yet, despite the pressure, this experience provided a deeper and more practical understanding of DevOps and cybersecurity than any structured lesson could have offered. Real incidents force you to confront the consequences of design decisions, configuration choices, and operational habits in a way theoretical learning never does.
Key lessons learned:
- A VPS exposed with open ports is an open invitation for attackers. Any publicly accessible service increases the attack surface. Without strict access controls, monitoring, and proper configuration, automated scanners and bots will quickly identify and target vulnerabilities.
- The presence of a crypto miner is a clear indicator of full system compromise. Attackers who deploy miners rarely stop there; they often modify system files, escalate privileges, and establish hidden footholds. Once this occurs, trust in the system is lost. Rebuilding from a clean state is the only reliable response.
- Attackers rely on persistence mechanisms that are deeper than a single malicious process. Modern attacks may involve cron jobs, rootkits, altered binaries, SSH backdoors, and hidden user accounts. Removing one visible component does not eliminate the underlying compromise.
- Least privilege is a critical layer of defense. Systems designed so that applications do not run as root help ensure that, even if an intrusion occurs, the damage is limited. Proper isolation, permissions, and role separation can prevent attackers from accessing sensitive data or making irreversible changes.
- Backups are invaluable during emergencies. When a system is compromised, having recent, verified, and off-server backups can mean the difference between a smooth recovery and catastrophic data loss. Backup integrity and testing should be treated as core operational practices.
- Default configurations are rarely secure. Vendor defaults prioritize usability and compatibility, not security. Hardening a system requires deliberate configuration: limiting access, disabling unnecessary services, enforcing strong authentication, and enabling proper logging.
- Incident response demands composure and methodical action. In high-pressure situations, rushing often leads to mistakes. A structured approach—assess, contain, preserve evidence, recover, and improve—ensures that decisions are informed and effective rather than reactive.
Ultimately, this was far more than an unplanned lesson in server hardening. It served as a reminder that security is not a checkbox or a one-time setup. It is an ongoing discipline that requires continuous awareness, proactive measures, and a mindset of vigilance.
TL;DR
My production VPS was fully compromised by attackers who deployed crypto-mining malware and multiple persistence mechanisms. Initial cleanup attempts failed because the system had already been taken over at the root level. After identifying disguised services, malicious binaries, and severe instability, I focused on backing up critical data before rebuilding everything on a fresh server.
The investigation showed that open ports, default firewall settings, and a likely exploitation of a newly disclosed vulnerability created the entry point. The incident reinforced essential lessons: never expose unnecessary services, never run applications as root, always enforce least privilege, and treat security as an ongoing discipline—not a one-time setup.
Top comments (0)