How We Removed a Crypto Miner Malware That Exploited Linux File Attributes

sachinravindran — Mon, 03 Feb 2025 08:36:52 +0000

Introduction

Imagine walking into a Monday morning only to find your company’s server grinding to a halt. CPU and memory usage are maxed out, services are timing out, and your team is panicking. This was my reality when I discovered a crypto miner malware (perfectl) had hijacked our server. What followed was a battle against resource-hogging processes, undeletable files, and a crash course in Linux file attributes. Here’s how I fought back—and what I learned along the way.

The Discovery: htop Reveals the Culprit

The first clue was the server’s sluggish performance. Running htop, I spotted a suspicious process named perfectl guzzling 90%+ of the CPU. Crypto miners are notorious for this—they silently hijack resources to mine cryptocurrency for attackers.

Key Takeaway:

Monitor resource usage: Tools like htop, top, or glances are critical for spotting abnormal processes.

The Roadblock: "Operation Not Permitted" Errors

After killing the malicious process, I traced its files to delete them. But even as root, deletion failed with cryptic errors:

rm: cannot remove 'malicious-file': Operation not permitted

A quick lsattr check revealed the malware’s sneaky trick: immutable files. The i attribute was set, preventing deletion or modification:

lsattr malicious-file  
----i---------e---- malicious-file

Why This Matters:

The i (immutable) flag locks files, even for root. Attackers use this to protect their malware from being removed.

The Fix: chattr to the Rescue

To remove the files, I first had to strip the i attribute using chattr:

sudo chattr -i malicious-file

Then deletion worked:

sudo rm -rf malicious-file

Important Notes:

Be thorough: Malware often spreads across multiple files/directories. Use lsattr -R /path/to/suspicious/dir to check recursively.
Audit cron jobs/services: The malware likely had a persistence mechanism (e.g., a cron job respawning it).

Lessons Learned: Protect Your Server

Monitor Actively: Use tools like htop, netstat, and audit logs.
Lock Down Permissions: Restrict write access to critical directories.
Know Linux Attributes: The i (immutable), a (append-only), and e (extent format) flags can be abused by attackers.
Automate Scans: Tools like chkrootkit, rkhunter, or modern EDR solutions can flag hidden threats.
Backup & Test: Ensure immutable backups exist to recover from attacks.

Final Thoughts

This incident was a wake-up call. Attackers are getting craftier—using legitimate Linux features (like chattr) against us. Vigilance, layered security, and understanding system fundamentals are your best defense.``