A small Python script that turned into a live anomaly detector and accidentally taught me how attackers actually live inside a Linux machine.
The Moment This Tool Became Necessary
If you’ve ever solved a Linux forensics CTF, you know the ritual.
You SSH into the machine.
Everything looks normal.
Then you notice:
- CPU usage feels weird
- There’s network traffic
- Logs don’t line up
So you begin the ancient ceremony:
ps aux | grep -v grep
netstat -tulnp
ls -la /tmp
cat /proc/<pid>/cmdline
You scroll.
You guess.
You re-run commands.
Somewhere in those 200+ processes is a reverse shell, a dropper, or a cryptominer… and you are manually playing Where’s Waldo: Incident Response Edition.
After one challenge where I lost 40 minutes hunting a shell hiding in /dev/shm, I realized:
I wasn’t solving a cybersecurity problem.
I was solving a visibility problem.
So I built a tool.
That tool became ProcWatch.
What ProcWatch Actually Is
ProcWatch is a Python-based process behavior scanner.
Not antivirus.
Not signature detection.
It asks a different question:
Is any process on this system behaving like an attacker would?
Instead of checking files, it watches process behavior patterns used in real intrusions.
The Threat Model
Before writing detections, I mapped how attackers typically behave after gaining Linux access:
| Stage | What attackers do |
|---|---|
| Foothold | Drop binaries in /tmp or /dev/shm
|
| Control | Spawn reverse shells |
| Escalation | Abuse SUID or LD_PRELOAD |
| Persistence | Run scripts via interpreters |
| Monetization | Install crypto miners |
| Evasion | Delete binaries after execution |
Every ProcWatch detection maps directly to one of these.
Detection 1 — Executing from Writable Directories (CRITICAL)
SUSPICIOUS_LOCATIONS = ["/tmp", "/dev/shm", "/var/tmp", "/run/user", "/dev/mqueue"]
Malware loves writable directories because:
- No permissions required
- Rarely monitored
- Easy to clean up
/dev/shm is especially interesting.
It’s RAM-backed storage. Reboot the machine and evidence disappears.
Legitimate software almost never runs from here.
If you see:
/dev/shm/hidden_binary
You didn’t find a program.
You found the attacker.
Detection 2 — Suspicious Interpreter Usage
SUSPICIOUS_NAMES = ["bash","sh","nc","python","perl","ruby","socat"]
On its own this means nothing.
But combined with location or network activity, it becomes powerful.
A python3 process is normal.
A python3 process in /tmp connected to port 4444 is not.
This is called living-off-the-land.
Attackers use legitimate tools so they don’t need to upload malware.
Detection 3 — Privilege Escalation Indicators
if uids.real != uids.effective:
Linux processes have:
- real UID
- effective UID
If they differ, you just caught a SUID privilege escalation in action.
Even more suspicious:
root process running from /home/user
Root processes belong in /usr/bin, not a user’s Downloads folder.
Detection 4 — Reverse Shell & C2 Detection
4444, 5555, 7777, 31337
These ports are classics for reverse shells and C2 listeners.
ProcWatch distinguishes between:
- LISTEN → suspicious
- ESTABLISHED outbound connection → almost certain compromise
A process connecting outward to port 4444 is basically waving a flag saying:
“someone else is controlling me.”
Detection 5 — Cryptominer Detection
Two techniques:
- Keyword detection
xmrig, monero, stratum, pool
- Behavioral detection
CPU usage > 85%
Why this matters:
Modern attackers often don’t destroy systems.
They monetize them.
Many real breaches are discovered because servers suddenly run at 100% CPU 24/7.
Detection 6 — Deleted Executable Trick
Attackers often do:
- Run malware
- Delete the file
Linux keeps the process alive in memory.
The file disappears from disk but continues executing.
Kernel hint:
/proc/<pid>/exe -> binary (deleted)
ProcWatch detects exactly this.
You can even recover it:
cp /proc/<pid>/exe recovered_binary
That’s a real forensic technique.
Detection 7 — LD_PRELOAD Injection (Advanced)
LD_PRELOAD=/tmp/libevil.so
This is powerful.
Instead of malware, attackers load a malicious library that intercepts system calls:
- hide files
- steal passwords
- fake authentication
This is how many user-space rootkits work.
Catching this is almost always a confirmed compromise.
The CLI
I designed the tool like git:
procwatch scan
procwatch scan -v -j
procwatch watch
procwatch info <pid>
procwatch list
The most useful mode
procwatch watch
It alerts only on new suspicious processes, so you get a live feed of attacker activity.
It feels surprisingly close to a real SOC monitoring console.
How This Helps in CTFs
Immediately after login:
procwatch scan -j
You now have a timestamped snapshot of the system’s process state.
Instead of investigating 200 processes, you investigate 3.
That changes a 45-minute challenge into a 5-minute one.
Limitations
This tool operates in user space via /proc.
So it cannot detect kernel rootkits.
If the kernel itself lies, ProcWatch will believe it.
Also:
- Some interpreter alerts are false positives
- Containers hide processes via namespaces
Future upgrades I’m considering:
- YARA memory scanning
- eBPF syscall monitoring
- ptrace detection
Running It
pip install psutil
sudo python3 procwatch.py scan -v
(Root gives full visibility into environment variables and connections.)
What I Learned
I started this to automate a CTF workflow.
Instead, I learned something important:
Attackers don’t hide files.
They hide behavior inside normal processes.
And behavior is much harder to fake.
ProcWatch doesn’t replace investigation.
It just shines a flashlight directly where you should start looking.
Top comments (2)
That moment when 'CPU usage feels weird' turns into 'oh no, we've got a cryptominer' — love that you built something for the paranoid sysadmin in all of us. How deep does the process inspection go? Can it detect injected code or just anomalous behavior?
Good question! ProcWatch is mainly behavior-based, not deep memory forensics. It doesn’t directly prove code injection inside a process, but it detects the effects injected or malicious code usually causes like
LD_PRELOADlibrary hijacking, UID privilege anomalies, processes running from deleted binaries, reverse-shell network connections, and miners abusing CPU. So instead of saying “this exact shellcode is here,” it answers “this process is acting like an attacker.” Detecting pure in-memory shellcode would need kernel or memory inspection tools (eBPF, ptrace, YARA), which I’m planning to explore next.