DEV Community

Cover image for Inside a Real Production Server Breach
ALI MANSOOR
ALI MANSOOR

Posted on

Inside a Real Production Server Breach

Just a normal day.
23rd May, 2026.

Wake up in the morning, pick up my friend from his house, head to the gym.

Somewhere between sets, he casually mentions:

"One of my client's apps went down. I've been awake for the last 2-3 hours trying to fix it."

We start discussing it casually.

He tells me:

  • git status showed a huge number of untracked files
  • index.php had been replaced
  • he quickly stashed changes
  • restarted apache2
  • the app came back online

At that point, it was obvious:

Someone had access to the server.


The Investigation Begins

Fast forward to the evening.

We hop on a call and start investigating properly.


Step 1 — Check for Suspicious Processes

First thing:

ps aux --sort=-%mem | head
Enter fullscreen mode Exit fullscreen mode
top
Enter fullscreen mode Exit fullscreen mode
htop
Enter fullscreen mode Exit fullscreen mode

We also checked for suspicious network connections:

ss -tulpn
Enter fullscreen mode Exit fullscreen mode
netstat -antp
Enter fullscreen mode Exit fullscreen mode

Nothing unusual.

No obvious crypto miners.
No rogue reverse shells.
No suspicious high CPU usage.


Step 2 — Check Active Users & Login History

Now we move toward authentication logs.

who
Enter fullscreen mode Exit fullscreen mode
w
Enter fullscreen mode Exit fullscreen mode
last -a
Enter fullscreen mode Exit fullscreen mode
lastlog
Enter fullscreen mode Exit fullscreen mode

Then:

cat /var/log/auth.log | grep "Accepted"
Enter fullscreen mode Exit fullscreen mode

And there it was.

A suspicious login around the exact same time the server went down.

Even more interesting:

The attacker was exploring the server manually.

We saw commands like:

pm2 list
Enter fullscreen mode Exit fullscreen mode
docker ps
Enter fullscreen mode Exit fullscreen mode
cd /var/www
ls -la
Enter fullscreen mode Exit fullscreen mode
find . -type f
Enter fullscreen mode Exit fullscreen mode

This wasn't automated malware blindly running.

Someone had interactive shell access.


The Weird Part

The traffic was coming from an Amazon IP address.

And not just any region.

It originated from the same country as the client.

That initially made things confusing because AWS IPs often look "legitimate" during quick inspection.


Step 3 — Inspect the Web Directory

Now we check the actual application files.

git status
Enter fullscreen mode Exit fullscreen mode

Absolute chaos.

Dozens of unexpected PHP files.

Mostly inside:

/var/www/public_html
Enter fullscreen mode Exit fullscreen mode

Some had random names like:

x.php
shell.php
cache.php
1index.php
Enter fullscreen mode Exit fullscreen mode

Classic indicators of uploaded web shells.

At that moment, one thing immediately came to mind:

LFI / Arbitrary File Upload

If attackers can upload executable PHP files into a web-accessible directory, game over.


Step 4 — Trace the Upload Vector

I ask him:

"Can users upload files anywhere on the platform?"

He says:

"Only paid users can upload."

Then adds:

"But there is a free trial."

Bingo.


Step 5 — Find the Attacker Account

We inspect recent signups.

Sure enough:

A suspicious user registered around 3-4 days earlier using the free trial flow.

Now we know where to look.


Step 6 — Check User Uploads

We inspect the upload directories.

ls -la /var/www/assets/uploads/
Enter fullscreen mode Exit fullscreen mode

Then:

find /var/www/assets/uploads -type f | grep ".php"
Enter fullscreen mode Exit fullscreen mode

And there it was.

A PHP script uploaded directly into the user's upload directory.

/assets/uploads/1234/xannyanaxium.php
Enter fullscreen mode Exit fullscreen mode

The attacker was simply visiting:

https://example.com/assets/uploads/1234/xannyanaxium.php
Enter fullscreen mode Exit fullscreen mode

And it presented with a kind of a control panel, to upload further files, execute commands and whatnot.


What Happened

The application allowed users to upload files.

But:

  • file extensions were not validated properly
  • uploaded files were stored in a publicly accessible directory
  • PHP execution was allowed inside uploads
  • attackers could execute arbitrary code

This resulted in:

Remote Code Execution (RCE)

Once the attacker uploaded a PHP web shell, they effectively had server access.


Example of a Dangerous Upload

A minimal malicious PHP shell can be as small as:

<?php system($_GET['cmd']); ?>
Enter fullscreen mode Exit fullscreen mode

Which allows attackers to run:

shell.php?cmd=id
Enter fullscreen mode Exit fullscreen mode

Or worse:

shell.php?cmd=rm+-rf+/
Enter fullscreen mode Exit fullscreen mode

Immediate Remediation Steps

1. Disable PHP Execution in Upload Directories

Inside uploads directory:

nano /var/www/assets/uploads/.htaccess
Enter fullscreen mode Exit fullscreen mode

Add:

php_flag engine off
Enter fullscreen mode Exit fullscreen mode

Or for Nginx:

location /uploads/ {
    location ~ \.php$ {
        deny all;
    }
}
Enter fullscreen mode Exit fullscreen mode

2. Restrict Upload Extensions

Allow only:

jpg
jpeg
png
pdf
webp
Enter fullscreen mode Exit fullscreen mode

Never trust client-side validation.

Validate server-side.

Example in PHP:

$allowed = ['jpg', 'jpeg', 'png', 'pdf'];

$ext = strtolower(pathinfo($_FILES['file']['name'], PATHINFO_EXTENSION));

if (!in_array($ext, $allowed)) {
    die("Invalid file type");
}
Enter fullscreen mode Exit fullscreen mode

3. Rename Uploaded Files

Never preserve original filenames.

Instead:

$newName = bin2hex(random_bytes(16)) . ".jpg";
Enter fullscreen mode Exit fullscreen mode

4. Store Uploads Outside Web Root

Bad:

/var/www/public_html/uploads
Enter fullscreen mode Exit fullscreen mode

Better:

/var/app_uploads
Enter fullscreen mode Exit fullscreen mode

Serve files through application logic instead.


5. Hunt for Persistence

Attackers often leave backdoors.

Search aggressively:

find /var/www -type f -name "*.php" | grep -E "(shell|cmd|cache|tmp)"
Enter fullscreen mode Exit fullscreen mode

Search recently modified files:

find /var/www -mtime -7
Enter fullscreen mode Exit fullscreen mode

Search suspicious functions:

grep -R "system(" /var/www
grep -R "exec(" /var/www
grep -R "base64_decode" /var/www
Enter fullscreen mode Exit fullscreen mode

6. Rotate Everything

Assume compromise.

Rotate:

  • SSH keys
  • database passwords
  • API keys
  • JWT secrets
  • cloud credentials
  • payment provider keys

Everything.


It all came down to one thing:

Never allow executable uploads into a public directory.

Top comments (0)