On March 24, 2026, two backdoored versions of LiteLLM — the popular open-source LLM proxy with 3.4 million daily PyPI downloads — were published to PyPI. They were live for roughly two to three hours before being quarantined. In that window, a three-stage credential stealer was deployed to every system that pulled the update, targeting everything from AWS keys to Kubernetes cluster secrets to cryptocurrency wallets.
But this wasn't a simple account takeover. The LiteLLM compromise was the final link in a five-day cascading supply chain campaign that started by weaponizing a vulnerability scanner. Here's the full story.
The Kill Chain: From Security Scanner to AI Proxy
The threat group behind this — tracked as TeamPCP, with suspected (unconfirmed) ties to LAPSUS$ — didn't attack LiteLLM directly. They built a chain of compromises, each one enabling the next.
Step 1: Trivy (March 19)
It started with Aqua Security's Trivy, one of the most widely used open-source vulnerability scanners. Weeks earlier, an autonomous bot called hackerbot-claw exploited a misconfigured pull_request_target workflow in Trivy's repo to steal a Personal Access Token. Aqua rotated credentials — but the rotation was incomplete.
On March 19, TeamPCP used the remaining credentials (which still had tag-writing privileges) to force-push malicious commits to 76 of 77 version tags in aquasecurity/trivy-action and all 7 tags in aquasecurity/setup-trivy. They also published an infected Trivy binary (v0.69.4) to GitHub Releases and container registries.
A vulnerability scanner — a tool people install specifically to make their pipelines more secure — became the initial attack vector. The irony is hard to overstate.
Step 2: npm Worm (March 20)
npm tokens stolen from Trivy's CI environment fed a self-propagating worm called CanisterWorm that infected 66+ npm packages. The blast radius was expanding.
Step 3: Checkmarx KICS (March 23)
All 35 tags of Checkmarx/kics-github-action — another security scanning tool — were hijacked using a compromised service account, likely harvested from one of the earlier compromises. Two security scanners now compromised in the same campaign.
Step 4: LiteLLM (March 24)
LiteLLM's CI/CD pipeline ran the compromised Trivy action. TeamPCP harvested PyPI publishing credentials from that pipeline and used them to publish backdoored versions (v1.82.7 and v1.82.8) directly to PyPI, completely bypassing the project's normal release workflow.
The chain:
Vulnerable CI workflow → compromised security scanner → stolen CI secrets → compromised AI proxy serving millions of downloads per day
Inside the Payload: Three Stages of Compromise
This wasn't a lazy crypto-miner. The malware was engineered for deep, persistent infiltration with encrypted exfiltration and a built-in researcher-defeat mechanism.
Stage 1 — Silent Activation
The package drops a 34KB file called litellm_init.pth into Python's site-packages directory. Python's .pth file mechanism is designed for path configuration, but it can execute arbitrary code — and it does so on every Python interpreter startup, not just when LiteLLM is imported.
If the package was installed in your environment, the payload was running on every Python process. No import litellm required. This is a legitimate Python feature that doubles as a devastating attack surface, and it deserves far more attention from the Python security community.
Additionally, malicious code was injected into proxy_server.py in both affected versions, hitting anyone who actually ran the LiteLLM proxy directly.
Stage 2 — Reconnaissance and Credential Harvesting
The second stage performs deep system enumeration and sweeps for sensitive data:
- SSH keys and Git credentials
- Cloud provider credentials — AWS access keys, GCP application default credentials, Azure tokens
- Kubernetes configs — kubeconfig files and service account tokens
- Infrastructure secrets — Terraform state files, Helm configs, CI/CD environment variables
-
Application secrets —
.envfiles, database connection strings - Cryptocurrency wallets
The malware didn't just grab files. It actively queried discovered credentials — calling AWS APIs, listing Kubernetes secrets across namespaces — to validate and expand access.
All harvested data was encrypted with AES-256-CBC using a randomly generated session key. That session key was then encrypted with a hardcoded 4096-bit RSA public key. The package was bundled as tpcp.tar.gz and exfiltrated to models[.]litellm[.]cloud — a domain deliberately chosen to look like legitimate LiteLLM infrastructure.
Stage 3 — Persistence and Lateral Movement
The final stage installs a systemd service (sysmon.py) that polls a command-and-control server every 50 minutes for additional payloads to execute. This survives package uninstallation — removing litellm from pip does not remove the backdoor.
In Kubernetes environments, the malware goes further: it reads all cluster secrets across all namespaces, then attempts to deploy privileged pods on every node in the kube-system namespace. The goal is full cluster takeover.
One notable detail: the C2 polling mechanism includes a filter that rejects responses containing "youtube.com" — a simple but effective technique to defeat security researchers using mock C2 servers.
Why AI Gateways Are High-Value Targets
LiteLLM is an AI gateway — it sits between your application and every LLM provider you use (OpenAI, Anthropic, Azure OpenAI, Bedrock, Vertex AI, and dozens more). By design, it holds API keys for all of them. It often runs with broad network access, frequently inside Kubernetes clusters alongside other production services.
This makes AI gateways uniquely attractive targets:
Credential density is extreme. A single compromised LiteLLM instance can yield API keys for every LLM provider an organization uses, plus whatever infrastructure credentials exist on the host. Compare this to compromising a single-purpose microservice.
Deployment environments are privileged. Most serious LLM deployments run on Kubernetes. The LiteLLM proxy typically needs network access to external APIs, often has access to secrets stores, and runs in clusters alongside other production workloads. Compromising it gives lateral movement opportunities that the TeamPCP malware was explicitly designed to exploit.
Update velocity is high. The AI ecosystem moves fast. Teams often track the latest versions of tools like LiteLLM to get new model support, bug fixes, and features. This creates a wide window for supply chain attacks — automated pipelines pull updates quickly, and manual review of each release is rare.
Security maturity lags adoption. Many teams deploying LLM infrastructure haven't applied the same supply chain security rigor they use for traditional dependencies. Pinned versions, checksum verification, artifact attestation, and staged rollouts are often absent from AI tooling pipelines.
What You Should Do
If you installed litellm v1.82.7 or v1.82.8
Treat the entire host or container as compromised. Uninstalling the package is insufficient — the systemd persistence mechanism survives pip uninstall.
- Isolate affected systems immediately from the network.
-
Look for the backdoor: check for
sysmon.pyand associated systemd services. -
Rotate everything: SSH keys, cloud credentials (AWS/GCP/Azure), Kubernetes configs and service account tokens, all LLM provider API keys, database passwords, CI/CD secrets,
.envcontents. -
In Kubernetes: audit for unauthorized privileged pods in
kube-system, review secrets access logs via audit trails, check for unknown service accounts or role bindings. -
Review network logs for connections to
models[.]litellm[.]cloudandcheckmarx[.]zone. - Rebuild affected systems from known-good images. Credential rotation alone may not be sufficient if the C2 channel delivered additional payloads.
For everyone: harden your AI supply chain
Pin exact versions and verify checksums. Never use >= or ~= for critical infrastructure dependencies. Use hash-pinning in requirements files (--require-hashes).
Audit your CI/CD pipeline dependencies. The entire LiteLLM compromise happened because a GitHub Action in the CI pipeline was compromised. Do you know which third-party actions have access to your publishing secrets? Pin actions to commit SHAs, not tags.
Use artifact attestation. Sigstore and similar tools can verify that a package was built from a specific source commit by a specific workflow. If LiteLLM's releases had been attested and consumers had verified attestations, the malicious versions would have been rejected.
Isolate your AI gateway. Your LLM proxy doesn't need access to your entire cloud account, your Kubernetes cluster secrets, or your SSH keys. Run it in a minimal environment with only the credentials it actually needs.
Monitor for unexpected releases. Set up alerts for new versions of critical dependencies. If your AI gateway publishes a new version outside normal release patterns, investigate before deploying.
Rethinking the AI Gateway Layer
This incident highlights a structural problem: when a single open-source package becomes the chokepoint for all your LLM traffic and runs as a self-managed proxy in your infrastructure, a supply chain compromise becomes a skeleton key to your entire AI stack.
It's worth evaluating alternatives that reduce this risk surface. Managed AI gateway solutions like TrueFoundry take a fundamentally different approach — the gateway runs as managed infrastructure with enterprise-grade security controls, rather than as a PyPI package you pull into your own environment and trust to self-update. This means the attack surface of "compromised package in your CI/CD" simply doesn't exist for the gateway layer. TrueFoundry also provides built-in secrets management, RBAC, and audit logging for LLM API keys, so credentials aren't scattered across environment variables waiting to be harvested.
This isn't about any single tool being inherently unsafe — the LiteLLM maintainers were themselves victims of an upstream compromise. It's about whether the deployment model of your AI gateway introduces unnecessary risk. Self-managed open-source proxies require you to own the entire supply chain security burden. Managed platforms shift that burden to a team whose full-time job is securing it.
The Bigger Picture
The TeamPCP campaign (tracked as CVE-2026-33634 for the Trivy component, sonatype-2026-001357 for LiteLLM) is being analyzed by security teams across the industry — Sonatype, Wiz, Datadog Security Labs, Snyk, ReversingLabs, Kaspersky, and Palo Alto Networks have all published detailed technical reports.
With an estimated 500,000+ credentials already exfiltrated and the C2 infrastructure having had time to deliver additional payloads, the full impact of this campaign will take months to assess.
The AI ecosystem has inherited all of the software supply chain's worst problems without the maturity to deal with them. If there's one takeaway from this incident, it's this: your AI infrastructure deserves the same supply chain security rigor as the rest of your stack — and probably more, given what it has access to.
If you're dealing with incident response on this, the detailed technical analyses from Sonatype, Datadog Security Labs, and Wiz are excellent starting points.
Top comments (1)
This writeup is exactly what the community needed — a calm, technical breakdown rather than just "LiteLLM got hacked, update now." The supply chain angle is what makes this particularly unsettling. LiteLLM sits at the gateway layer, meaning a compromised version has visibility into every prompt, every API key, every response going through it. The blast radius isn't just the application — it's potentially everything upstream.
What strikes me here is how normalized dependency trust has become in AI infra specifically. The typical ML/AI dev workflow is: pip install litellm, trust PyPI, move on. We've gotten better about this in systems programming (hash verification, reproducible builds), but AI tooling is still in the "ship fast, trust the registry" era.
A few things I'd add to the mitigation list: (1) hash-pin your AI infra dependencies separately from your general requirements — litellm especially warrants its own pinned requirements-ai.txt with SHA256 verification. (2) If you're running LiteLLM as a proxy, consider wrapping it in a network policy that blocks unexpected egress — the backdoor in the reported versions was exfiltrating data outbound. (3) Audit your Docker layers; if LiteLLM is in a base image you pulled weeks ago, you may still be running the compromised version.
Thanks for getting this up fast. This is the kind of writeup that actually changes people's practices.