The LiteLLM Supply Chain Attack: How a Poisoned Security Scanner Stole Credentials from Thousands of AI Environments

A deep dive into one of the most sophisticated software supply chain attacks of 2026 — and what every developer can learn from it.

---

On March 24, 2026, a developer named Callum McMahon at FutureSearch was testing a Cursor MCP plugin. Minutes after Python installed a fresh dependency, his machine ground to a halt — RAM usage spiking to 100%, processes forking uncontrollably. What he'd stumbled into wasn't a bug. It was a backdoor.

Two versions of litellm, a popular Python package downloaded 3.4 million times per day, had been poisoned. The malicious code was quietly harvesting SSH keys, cloud credentials, Kubernetes secrets, and cryptocurrency wallets — then encrypting everything and shipping it to an attacker-controlled server.

The kicker? The attackers got in by compromising a security scanner.

What Is LiteLLM?

LiteLLM is an open-source Python library that acts as a unified gateway to 100+ LLM providers — OpenAI, Anthropic, Cohere, Hugging Face, and more. With roughly 97 million monthly downloads and 2,000+ dependent packages (including DSPy, MLflow, and Open Interpreter), it sits at a critical junction in the AI developer ecosystem.

When deployed as a proxy server, LiteLLM holds API keys for multiple model providers in one place. That makes it an incredibly high-value target.

The Attack Chain: Five Days, Three Tools, One Threat Actor

The attack on LiteLLM didn't start on March 24. It started five days earlier.

Phase 1: Trivy (March 19)

Trivy is a widely-used open-source security scanner from Aqua Security. On March 19, attackers exploited a pull_request_target workflow vulnerability in Trivy's CI to exfiltrate the aqua-bot credentials. They then rewrote the Git tags for trivy-action v0.69.4 to point to a malicious release carrying a credential-harvesting payload.

Yes — a security scanner was the first domino.

Phase 2: Checkmarx KICS (March 23)

Using infrastructure from the Trivy compromise, the same attackers hit Checkmarx KICS (Keep Infrastructure as Code Secure). The C2 domain checkmarx.zone — designed to impersonate the Checkmarx security company — was registered and activated.

Phase 3: LiteLLM (March 24)

LiteLLM's CI/CD pipeline ran Trivy as part of its build process, pulling it from apt without a pinned version. The compromised Trivy action exfiltrated the PYPI_PUBLISH token from the GitHub Actions runner environment.

With that token, the attackers published:

• litellm 1.82.7 at 10:39 UTC
• litellm 1.82.8 at 10:52 UTC (just 13 minutes later)

Both contained malicious payloads. Different delivery mechanisms, same devastating effect.

Two Delivery Mechanisms

Version 1.82.7: Source Code Injection

The payload was base64-encoded and embedded directly inside litellm/proxy/proxy_server.py. It executes whenever anything imports litellm.proxy — the standard import path for LiteLLM's proxy server mode.

Version 1.82.8: The .pth Nuclear Option

This version added a file called litellm_init.pth to site-packages/. Python's .pth mechanism fires on every interpreter startup — no import needed. This means the payload runs when you execute pip, python -c, or even when your IDE's language server starts Python.

The .pth file was correctly declared in the wheel's RECORD file, so it passed pip's hash verification. The package passed all standard integrity checks because it was published using legitimate credentials.

CPython maintainers have acknowledged the security risk of .pth startup hooks in issue #113659, but no patch has been applied as of this writing. — Snyk

The Three-Stage Payload

Stage 1: Credential Harvesting

The script systematically collects:

• System data: hostname, environment variables, running processes, network routing, auth logs
• SSH keys: all private keys from ~/.ssh/
• Cloud credentials: AWS credentials/config, GCP and Azure service account files, cloud metadata via IMDS queries (with full IMDSv2 signing implemented), AWS Secrets Manager, and SSM Parameter Store
• Container/orchestration secrets: Docker config.json, Kubernetes kubeconfig, service account tokens, secrets, and certificates
• CI/CD configs: Jenkins, Travis CI, Terraform state files
• Communication tokens: Slack and Discord webhook tokens
• Cryptocurrency: Bitcoin, Ethereum, Solana, Cardano, Monero wallet files, seed phrases, and Ledger hardware wallet configurations
• Everything else: .env files, Git credentials, API keys from shell history, password hashes from /etc/shadow

Stage 2: Encrypted Exfiltration

The collected data is encrypted before transmission:

1. Generate a 32-byte AES-256 session key via openssl rand
2. Encrypt data with AES-256-CBC (PBKDF2 key derivation)
3. Encrypt the session key with a hardcoded 4096-bit RSA public key (OAEP padding)
4. Bundle everything into tpcp.tar.gz
5. POST the bundle to https://models.litellm.cloud/

The domain models.litellm.cloud was registered on March 23 — one day before the attack. The RSA public key in this payload is identical to the one found in the Trivy and KICS payloads, providing ironclad attribution.

Stage 3: Persistence and Lateral Movement

Local persistence:

• Writes a backdoor to ~/.config/sysmon/sysmon.py
• Creates a systemd user service named "System Telemetry Service"
• The service polls https://checkmarx.zone/raw every 5 minutes for commands

Kubernetes lateral movement:
If the script finds a K8s service account token, it:

1. Reads all secrets across every namespace
2. Deploys a privileged pod to every node in kube-system using alpine:latest
3. These pods mount the host filesystem and install the backdoor on the underlying node

Pods are named node-setup-{node_name}, making them look like legitimate cluster operations at first glance.

The Discovery: A Bug Saved Us

The malicious payload had a critical flaw. The .pth mechanism fires when Python starts, and the payload spawns a new Python subprocess. That subprocess also triggers .pth execution, creating an unintended fork bomb that caused exponential RAM growth.

McMahon traced the OOM crash to the newly installed litellm package, found litellm_init.pth (a 34,628-byte double base64-encoded file), and published his findings on futuresearch.ai. The disclosure spread to r/LocalLLaMA, r/Python, and Hacker News within the hour.

If the attacker hadn't accidentally created a fork bomb, this could have run silently for weeks.

The Cover-Up Attempt

When the community began reporting the compromise in GitHub issue #24512, the attackers responded with brute force:

• 88 bot comments from 73 unique accounts posted in a 102-second window (12:44-12:46 UTC)
• The accounts were previously compromised developer accounts, not purpose-created profiles
• Using the compromised maintainer account (krrishdholakia), the attackers closed the issue as "not planned"
• They committed messages to unrelated repositories reading "teampcp update"

76% of the bot accounts overlapped with the botnet used during the Trivy disclosure. The community routed around the censorship by opening a parallel tracking issue (#24518) and continuing discussion on Hacker News, where the thread reached 324 points.

The Blast Radius

The poisoned versions were on PyPI for approximately 3 hours. In that window, projects that filed security PRs included:

Project	Response
DSPy	PR #9498 merged; CI failure report
MLflow	PR #21971 merged
OpenHands	PR #13569
CrewAI	PR #5040 (decoupled from litellm)
langwatch	PR #2577
Arize Phoenix	PR #12342
Aider	Confirmed safe (pins litellm==1.82.3)

Remember: the .pth mechanism fires when any Python process starts, including pip itself. In CI/CD environments, this means the payload executes during build steps, not just at application runtime.

Per Wiz, LiteLLM is present in 36% of all cloud environments.

Who Is TeamPCP?

TeamPCP (also known as PCPcat, Persy_PCP, ShellForce, and DeadCatx3) has been active since at least December 2025. They maintain Telegram channels and embed the string "TeamPCP Cloud stealer" in their payloads.

The LiteLLM compromise is Phase 09 of an ongoing campaign spanning five ecosystems: GitHub Actions, Docker Hub, npm, Open VSX, and PyPI. All three domains in this operation share the same registrar (Spaceship, Inc.) and hosting provider (DEMENIN B.V.).

Most alarmingly, per The Hacker News, TeamPCP is now collaborating with LAPSUS$, the notorious extortion group. In a Telegram post, TeamPCP stated:

"These companies were built to protect your supply chains yet they can't even protect their own. The state of modern security research is a joke. As a result, we're gonna be around for a long time stealing terabytes of trade secrets with our new partners."

They've also deployed CanisterWorm, which uses the Internet Computer Protocol (ICP) as a C2 channel — the first observed use of ICP as C2 in a supply chain campaign, making takedowns nearly impossible.

A component called hackerbot-claw uses an AI agent for automated attack targeting — one of the first documented cases of AI being used operationally in a supply chain attack.

Karpathy's Response

Andrej Karpathy, whose tweet about the incident garnered 36.7 million views, used this as a rallying point for his stance on dependency minimalism:

"I don't like adding dependencies to my projects. Instead I prefer to use an LLM to 'yoink' the code I need... I get just the code I need with no transitive dependency surprises."

Whether you agree with Karpathy's radical dependency avoidance or not, the sentiment resonated: this attack was possible because a build pipeline trusted an unpinned dependency, which trusted another unpinned dependency, creating a cascading compromise chain.

Are You Affected? Check Now

Step 1: Check your installed version

pip show litellm | grep Version

If you see 1.82.7 or 1.82.8, treat the system as compromised. Don't just upgrade — the payload may have already run.

Step 2: Check for persistence artifacts

# Check for the sysmon backdoor
ls -la ~/.config/sysmon/sysmon.py 2>/dev/null && echo "BACKDOOR FOUND"
ls -la /root/.config/sysmon/sysmon.py 2>/dev/null && echo "ROOT BACKDOOR FOUND"

# Check for the systemd persistence service
systemctl --user status sysmon.service 2>/dev/null
ls -la ~/.config/systemd/user/sysmon.service 2>/dev/null && echo "PERSISTENCE SERVICE FOUND"

# Check for exfiltration archive remnants
ls /tmp/tpcp.tar.gz /tmp/session.key /tmp/payload.enc /tmp/session.key.enc 2>/dev/null && echo "EXFIL ARTIFACTS FOUND"

Step 3: Check for malicious .pth files

find $(python3 -c "import site; print(' '.join(site.getsitepackages()))") \
  -name "*.pth" -exec grep -l "base64\|subprocess\|exec" {} \;

Step 4: Verify file hashes

# Check proxy_server.py (1.82.7)
find / -path "*/litellm/proxy/proxy_server.py" 2>/dev/null -exec shasum -a 256 {} \;
# Malicious hash: a0d229be8efcb2f9135e2ad55ba275b76ddcfeb55fa4370e0a522a5bdee0120b

# Check litellm_init.pth (1.82.8)
find / -name "litellm_init.pth" 2>/dev/null -exec shasum -a 256 {} \;
# Malicious hash: 71e35aef03099cd1f2d6446734273025a163597de93912df321ef118bf135238

Step 5: Check network indicators

grep "litellm.cloud\|checkmarx.zone" /etc/hosts
grep "models.litellm.cloud\|checkmarx.zone" /var/log/syslog 2>/dev/null

Step 6: Check Kubernetes

kubectl get pods -A | grep "node-setup-"

If You're Compromised: Full Remediation

1. Remove persistence: Delete ~/.config/sysmon/, disable sysmon.service, remove /tmp/tpcp.tar.gz and related files
2. Rotate ALL credentials: SSH keys, AWS/GCP/Azure keys, API keys, Docker registry creds, K8s configs, database passwords, Git credentials, cryptocurrency wallet seed phrases
3. Audit cloud services: AWS Secrets Manager, SSM Parameter Store, all namespaced K8s secrets
4. Check for rogue pods: Look for node-setup-* pods in kube-system
5. Reinstall clean: Use a fresh environment, pin to litellm<=1.82.6

Lessons Learned

1. Pin Your CI/CD Dependencies

LiteLLM pulled Trivy from apt without a pinned version. This is how the compromised Trivy action entered the pipeline. Every dependency in CI/CD should be pinned by hash or version.

2. Security Tools Are High-Value Targets

The attackers deliberately targeted tools with elevated access: a container scanner (Trivy), an infrastructure scanner (KICS), and an LLM gateway (LiteLLM). Each tool has broad read access to credentials by design.

3. .pth Files Are a Known Python Threat Vector

Python's .pth startup hooks execute arbitrary code on every interpreter startup. Despite being flagged in CPython issue #113659, no mitigation exists. Watch for packages that install .pth files.

4. Hash Verification Is Not Enough

pip install --require-hashes would have passed for both malicious versions because the malicious content was published using legitimate credentials. The hashes match what PyPI advertised — but what PyPI advertised was malicious.

5. The Snowball Effect Is Real

As Wiz researcher Gal Nagli summarized:

"Trivy gets compromised → LiteLLM gets compromised → credentials from tens of thousands of environments end up in attacker hands → and those credentials lead to the next compromise. We are stuck in a loop."

6. Accidental Bugs Save Lives

If the attacker had tested their fork bomb behavior, this could have persisted silently. They didn't, and an unintended OOM crash led to the fastest possible discovery.

The Bigger Picture

This isn't an isolated incident. It's Phase 09 of a sustained campaign that has hit five ecosystems. TeamPCP's playbook is simple and devastating: compromise a tool that has access to credentials, use those credentials to compromise the next tool, repeat.

The AI ecosystem is particularly vulnerable because:

• LLM gateway tools store credentials for dozens of providers
• Many AI projects move fast and pin dependencies loosely
• CI/CD pipelines in ML projects often run security scanners with broad access
• The AI developer community (r/LocalLLaMA, r/Python) was where the disclosure spread — not traditional security channels

The open-source supply chain's strength — that anyone can contribute — is also its weakness. When the tools we use to scan for security vulnerabilities are themselves compromised, we need to fundamentally rethink our trust model.

---

Indicators of Compromise (IOCs)

Type	Value
Malicious .pth hash (1.82.8)	71e35aef03099cd1f2d6446734273025a163597de93912df321ef118bf135238
Malicious proxy_server.py hash (1.82.7)	a0d229be8efcb2f9135e2ad55ba275b76ddcfeb55fa4370e0a522a5bdee0120b
Exfiltration domain	models.litellm.cloud
C2 polling domain	checkmarx.zone/raw
Persistence path	~/.config/sysmon/sysmon.py
Systemd service	sysmon.service ("System Telemetry Service")
Exfil artifacts	/tmp/tpcp.tar.gz, /tmp/session.key, /tmp/payload.enc
K8s rogue pods	node-setup-{node_name} in kube-system

---

Sources: Snyk, The Hacker News, Endor Labs, FutureSearch, Wiz, Socket

---

Tags: #security #supplychain #python #ai #litellm #cybersecurity #opensource #devops