DEV Community

Pranay Batta
Pranay Batta

Posted on

The LiteLLM Supply Chain Attack Broke Trust in Python-Based AI Infrastructure

#security #ai #python #llm

If you run LiteLLM in production, you probably had a rough week.

On March 24, 2026, two backdoored versions of litellm (1.82.7 and 1.82.8) were published to PyPI using stolen credentials. The malware stole SSH keys, AWS/GCP/Azure credentials, Kubernetes secrets, cryptocurrency wallets, and deployed persistent backdoors on infected machines. It was live for about 3 hours. LiteLLM gets 3.4 million daily downloads.

This is the full breakdown of what happened, why it matters, and what you should actually do about it.

What Happened: The Full Attack Chain

The attack didn't start with LiteLLM. It started with Trivy, a popular container security scanner.

Here's the sequence:

  1. A threat actor group called TeamPCP exploited a pull_request_target workflow vulnerability in Trivy's GitHub Action (GHSA-9p44-j4g5-cfx5)
  2. They used this to exfiltrate the aqua-bot credentials and rewrite Trivy v0.69.4 release tags to point to malicious payloads
  3. On March 23, they also compromised the Checkmarx KICS GitHub Action using similar techniques
  4. LiteLLM's CI/CD pipeline pulled the Trivy action without pinning to a specific version or commit SHA
  5. The malicious Trivy action exfiltrated LiteLLM's PYPI_PUBLISH token from the GitHub Actions runner
  6. Using the stolen token, TeamPCP published two backdoored versions to PyPI with legitimate credentials

The full timeline is documented in Snyk's breakdown and Wiz's attribution analysis.

The Malware: Three Stages of Damage

This was not a simple credential stealer. It was a three-stage payload designed for maximum extraction and persistence.

Stage 1: Information Collection

The malware harvested:

  • System data (hostname, environment variables, processes, network routing, auth logs)
  • SSH private keys, .env files, Git credentials
  • API keys from shell history
  • Slack and Discord webhook tokens
  • Jenkins, Travis CI, and Terraform configs
  • /etc/shadow password hashes
  • Full AWS credential chain including IMDSv2, Secrets Manager, and SSM Parameter Store
  • GCP and Azure service account files
  • Docker config.json and Kubernetes kubeconfig, service account tokens, secrets, and certificates
  • Cryptocurrency wallet files (Bitcoin, Ethereum, Solana, Cardano, Monero, Ledger hardware wallet configs)

Stage 2: Encryption and Exfiltration

Everything was bundled, encrypted with AES-256-CBC, the session key wrapped with a hardcoded 4096-bit RSA public key, and exfiltrated to models.litellm.cloud, a domain registered just one day before the attack.

Stage 3: Persistence and Lateral Movement

The malware installed a persistent backdoor at ~/.config/sysmon/sysmon.py with a systemd service polling checkmarx.zone every 5 minutes for new payloads. In Kubernetes environments, it read all secrets across every namespace and deployed privileged pods to every kube-system node, mounting the host filesystem.

Source: Snyk's full technical analysis

Two Delivery Mechanisms (This Is the Scary Part)

Version 1.82.7 embedded a Base64-encoded payload directly in litellm/proxy/proxy_server.py. It executed whenever anything imported litellm.proxy.

Version 1.82.8 added something worse: a file called litellm_init.pth placed in site-packages/. Python's .pth mechanism fires on every interpreter startup. That means the malware ran when you ran pip install, when your IDE started a language server, when you ran python -c "print('hello')". It had nothing to do with importing litellm.

This is MITRE ATT&CK T1546.018 (Python Startup Hooks). The .pth file was correctly declared in the wheel's RECORD file, so pip install --require-hashes would have passed.

Who Was Affected

The compromised versions were pulled into multiple major projects:

Project PR Merged Status
DSPy #9498 Affected
MLflow #21971 Affected
OpenHands #13569 Affected
CrewAI #5040 Affected
Arize Phoenix #12342 Affected
Aider N/A Safe (pinned litellm==1.82.3)

Aider survived because it pinned its dependency version. That one decision made the difference.

How Standard Defenses Failed

This attack bypassed almost every standard protection:

  • Hash verification passed because the packages were published with legitimate stolen credentials
  • No typosquatting to detect. The package name was exactly litellm
  • No suspicious domains at install time. The exfiltration domain was models.litellm.cloud, which looks legitimate
  • pip install --require-hashes would have passed because the .pth file was correctly declared in the wheel's RECORD

The only install-time defense that would have caught this is inspecting whether packages install .pth files containing subprocess, base64, or exec patterns. No widely deployed pip plugin does this automatically today.

The Bot Suppression Campaign

When researcher Callum McMahon at FutureSearch reported the compromise in GitHub issue #24512, TeamPCP used 73 previously compromised GitHub accounts to post 88 spam comments in 102 seconds, then closed the issue as "not planned" using the compromised maintainer account krrishdholakia.

76% of these accounts overlapped with the botnet used during the Trivy disclosure. This is documented in Wiz's analysis.

The Structural Problem Nobody Is Talking About

LiteLLM is a Python package. It sits between your application and your LLM providers. It holds API keys for OpenAI, Anthropic, AWS Bedrock, Google Vertex, and whatever else you route through it. It is, by design, a high-value target.

And it runs in a language where:

  • Any package can install .pth files that execute on interpreter startup
  • Transitive dependencies pull in dozens of packages you never explicitly chose
  • A single compromised CI/CD token can publish arbitrary code under a trusted package name
  • The GIL means you need to run multiple processes, each of which triggers .pth execution independently

This is not a criticism of Python as a language. Python is excellent for what it is good at. But the question is whether Python is the right choice for infrastructure that holds your most sensitive credentials and sits in the critical path of every LLM request.

Compiled languages like Go and Rust produce single binaries with no runtime dependency chain. There is no site-packages directory. There is no .pth execution mechanism. There is no pip install side effect. The attack surface is fundamentally smaller.

What You Should Do Right Now

1. Check Your LiteLLM Version 

pip show litellm | grep Version
Enter fullscreen mode Exit fullscreen mode

If you see 1.82.7 or 1.82.8, you were affected. LiteLLM's security update confirms these versions were compromised.

2. Scan for .pth Files 

find $(python -c "import site; print(site.getsitepackages()[0])") -name "*.pth" -exec grep -l "subprocess\|base64\|exec" {} \;
Enter fullscreen mode Exit fullscreen mode

3. Rotate Everything

If you ran the compromised version, assume all credentials on that machine are compromised:

  • AWS/GCP/Azure keys and service accounts
  • SSH keys
  • API keys (OpenAI, Anthropic, etc.)
  • Database passwords
  • Kubernetes service account tokens
  • CI/CD tokens

4. Pin Your Dependencies

Aider survived because it pinned litellm==1.82.3. Pin your versions. Better yet, pin by hash:

litellm==1.82.6 --hash=sha256:<known-good-hash>
Enter fullscreen mode Exit fullscreen mode

5. Pin Your CI/CD Actions by SHA

Don't do this:

uses: aquasecurity/trivy-action@latest
Enter fullscreen mode Exit fullscreen mode

Do this:

uses: aquasecurity/trivy-action@<full-commit-sha>
Enter fullscreen mode Exit fullscreen mode

6. Evaluate Your Gateway Architecture

This is the harder conversation. If your LLM gateway is a Python package that you pip install, it shares the same supply chain as every other Python package on your system. Every transitive dependency is a potential attack vector.

Alternatives worth evaluating:

  • Bifrost: Open-source LLM gateway written in Go. Single compiled binary, 11 microsecond overhead at 5,000 RPS. No Python supply chain surface area. Supports OpenAI, Anthropic, Bedrock, Vertex, and 20+ providers.
  • TensorZero: Rust-based LLM gateway with sub-millisecond overhead. Similar compiled-binary benefits.
  • Cloudflare AI Gateway: Managed edge service. No self-hosted dependency chain at all.
  • Direct provider SDKs: If you only use one or two providers, you may not need a gateway at all. The official OpenAI and Anthropic SDKs are smaller attack surfaces.

The right choice depends on your scale, provider count, and security requirements. But "keep using the Python package that just got backdoored" should not be the default.

The Bigger Picture

TeamPCP is not done. They also deployed CanisterWorm, using the Internet Computer Protocol as a C2 channel. They used an AI agent called openclaw for automated attack targeting. Their target selection focuses on tools with elevated pipeline access: container scanners, infrastructure scanning tools, AI routing libraries.

LLM gateways are a perfect target. They hold credentials for multiple providers. They run in CI/CD environments. They have broad read access by design.

The question is not whether this will happen again. It is whether your infrastructure is designed to limit the blast radius when it does.

References:

Tags: litellm, supply-chain-attack, security, python, llm-gateway, ai-infrastructure, devops, cybersecurity

Top comments (0)