DEV Community

Cover image for LiteLLM Supply Chain Attack: How Poisoned Version 1.107.0 Stole AI Keys, and How to Check Your Systems
Ahsan Nadeem
Ahsan Nadeem

Posted on Originally published at znovastudio.com

LiteLLM Supply Chain Attack: How Poisoned Version 1.107.0 Stole AI Keys, and How to Check Your Systems

#security #ai #python #devops

On 21 October 2026, a poisoned release of LiteLLM, the open-source gateway many teams put between their apps and OpenAI, Anthropic or Google, sat on PyPI and Docker Hub for about two hours, long enough for roughly 49,000 downloads. It stole whatever keys the machine could reach, and it ran even if your code never imported LiteLLM.

This LiteLLM supply chain attack is the project's second of 2026. Below: a quick self-check, the timeline, what to rotate and the build habits that stop the next one. Most of those habits live in your pipeline, which our CI/CD pipelines work sets up.

Am I affected by the LiteLLM attack? The 30-second answer

You're at risk if any environment installed litellm 1.107.0, or ran a LiteLLM Docker image on a floating tag, on 21 October 2026 during the roughly two-hour window. The malware ran whenever Python started, so if that machine had internet access, assume the keys it could reach were stolen. Pinning 1.106.2 or earlier kept you out of this incident.

You're probably affected if:

  • An environment installed litellm==1.107.0 from PyPI on 21 October 2026, roughly between 11:39 and 13:23 UTC.
  • A container ran ghcr.io/berriai/litellm or the Docker Hub image with the tag v1.107.0, latest, main-latest or main-stable, pulled between 11:46 and 14:20 UTC that day.
  • A dependency pulled in LiteLLM unpinned and got 1.107.0.

LiteLLM says its private enterprise registry was not touched.

The LiteLLM supply chain attack, minute by minute

This timeline follows LiteLLM's canonical record of 23 October, which corrected early assumptions it called "inaccurate or incomplete" (times UTC).

When What happened
27 May 2026 A master key for LiteLLM's experimental Agent Platform is committed to a public repository in an opencode.json file. The commit is overwritten; the key is never revoked.
10 October, 21:27 Using that key, the attacker gets the Agent Platform API to return a GitHub organisation owner's GitHub CLI token.
10 October, 22:11 With the owner's token, the attacker runs workflows that post CI secrets, including GitHub tokens, to webhook.site, then deletes the runs. A second round follows on 12 October.
21 October, 10:08 The owner's token overrides branch protection on the private release workflow.
21 October, 10:46 A build runs from a lookalike repository, Berri-AI/litellm (the real one is BerriAI/litellm).
21 October, 11:11 A maintainer's stolen token approves the production release gate.
21 October, 11:39:51 Seven malicious litellm==1.107.0 wheels go up on PyPI over 22 seconds.
21 October, 11:46:56 Poisoned images are published to Docker Hub and GHCR.
21 October, 11:53 Another stolen credential deletes the workflow records.
21 October, 12:06 A community member opens GitHub issue #47826 reporting the compromise.
21 October, by 13:23 PyPI quarantines 1.107.0.
21 October, 14:20 The last Docker Hub tags are removed or repointed.

Sources differ on the clean-up times. Datadog and Wiz put PyPI's quarantine at about 13:20 and image removal at about 13:53; LiteLLM says GHCR was clean by 13:51 but Docker Hub tags pointed at the bad image until 14:20. If you pull from Docker Hub, use the later time.

The stolen data was used fast. Wiz saw AWS instance-role credentials taken from victims being tested from two IP addresses between 16:20 and 17:01 UTC.

Early coverage blamed stolen GitHub CLI tokens from three engineers: broadly right, but the 23 October record adds the first link (an unrevoked key leaked nearly five months earlier) and rules out an early theory that the tokens came from an internal deployment's logs.

Why Python .pth startup hooks make this nastier than a normal malicious package

Most malicious packages need importing; this one didn't. The wheel dropped three files into site-packages, outside the litellm folder:

  • cook.pth, containing one line, import cook.
  • cook.py, the credential stealer.
  • sitecustomize.py, a backdoor loader.

Python reads every .pth file in site-packages each time it starts, executes any line beginning with import, and imports any module called sitecustomize. Python's documentation says these lines "are always run at Python startup ... regardless of whether a particular module is actually going to be used." They're now marked deprecated, but they still run.

In practice:

  • Any Python process ran the stealer: tests, a Jupyter kernel, a cron job. Microsoft saw a Windows case where installing an unrelated dependency triggered it.
  • pip uninstall litellm doesn't remove it. The files sit outside the package folder.
  • Checking with Python can re-trigger it. Running python or pip in a suspect environment starts the hooks again.
  • The source code looked clean. The official v1.107.0 tag never contained the payload. Endor Labs found one added line in the build settings, data = "cook" in pyproject.toml, copied a cook/ folder that isn't in the public source into the wheel.

The line worth remembering: with a .pth hook, installing the package is the same as running it, every time Python starts.

What the malware stole

Microsoft, Datadog and Wiz agree on what cook.py collected:

  • Cloud: AWS, Google Cloud and Azure credentials, including instance-role credentials from metadata services; Kubernetes, Docker, Vault and Terraform secrets.
  • Developer secrets: SSH, GPG and TLS keys; Git, npm and PyPI tokens; .env files; shell history; process environment variables.
  • AI tools: provider API keys and sessions for AI coding assistants, IDEs and MCP configs.
  • Browsers: saved passwords and cookies.
  • LiteLLM's own database: up to 500 rows per query from LiteLLM_CredentialsTable and LiteLLM_ProxyModelTable, where the upstream provider keys live.

The haul was encrypted and posted to www.litellm.dev/healthcheck, an attacker-controlled lookalike domain.

Separately, sitecustomize.py started a background process on Unix-like systems that checked a GitHub file hourly for a server address, then would download and run code from it. That file only ever held test values, so no second stage was seen.

Why AI gateways are prime targets

An AI gateway exists to hold every provider key so your apps don't have to. One compromised LiteLLM proxy can hand over your OpenAI, Anthropic, Google and Bedrock keys at once, plus its database password and cloud role.

In August 2026, Microsoft told defenders to "treat AI gateways as Tier-0 secrets stores", the most sensitive tier, and this payload carried code written specifically to read LiteLLM's credential tables. Prompt injection attacks what an AI system says; this attacked the keys behind it (our prompt injection attack examples cover the other side).

How to check if LiteLLM 1.107.0 is on your systems, step by step

Microsoft advises inspecting a suspect environment from a clean machine, not by running Python inside it. Even LiteLLM's suggested pip show litellm restarts the hooks, so use it only where you're rebuilding anyway.

  1. Search lock files. Look for litellm 1.107.0 in uv.lock, poetry.lock, Pipfile.lock and requirements*.txt in every repository.
  2. Look for the files. Search every site-packages for cook.pth, cook.py, sitecustomize.py, __pycache__/cook*.pyc and a litellm-1.107.0.dist-info folder. On Linux or macOS, find / -name cook.pth 2>/dev/null is a start.
  3. Check temp files and processes. A Python daemon holding /tmp/remote_code_loader.lock means the backdoor is running. The marker /tmp/.pc_once_72129118c59f5636c0bb means data was sent; its absence proves nothing.
  4. Check images by digest. Run docker image ls --digests and look for sha256:1bbcb1443bc962e45f5aad53fe3d2c30600e0379715fb6c664cdd3ac126c339e. LiteLLM's post lists separate digests for the litellm-database and litellm-non_root images.
  5. Search CI logs and caches. Any job between 11:39 and 14:20 UTC on 21 October that installed LiteLLM unpinned or pulled a floating tag is suspect. Check pip and uv caches and internal mirrors too, because a cached copy can be installed later.
  6. Check network logs for connections to www.litellm.dev or the header X-Token: XWiA8gZNa7NcX2XRoAEb.

For the March incident, look for versions 1.82.7 and 1.82.8 and a file called litellm_init.pth.

What to do if you find it: the response checklist

LiteLLM's rule: if 1.107.0 ran on a host with outbound internet access, assume the theft succeeded.

  1. Isolate affected laptops, build agents, containers and gateway hosts, and stop automation restarting Python on them.
  2. Kill and delete. Stop the daemon holding the lock file and delete cook.pth, cook.py, sitecustomize.py and their .pyc files. Don't run pip install first: it re-runs the hooks.
  3. Rebuild, don't repair. Rebuild from scratch, pinned to litellm==1.106.2 or one of the signed releases LiteLLM has shipped since 24 October (such as 1.106.4).
  4. Rotate from a clean machine everything the malware could reach (see "What the malware stole" above, and the list below).
  5. Hunt for misuse. In AWS CloudTrail, look for GetCallerIdentity and Bedrock listing calls from your roles around install time, and any use of those credentials from outside your network. Check provider dashboards for unexpected spend.

Pay special attention to:

  • Cloud: the instance or task IAM role, not just access keys.
  • LLM keys: every provider key in the proxy database, plus the master key and all virtual keys.
  • Database and GitHub: the DATABASE_URL password and gh CLI logins.
  • Sessions: revoke browser and AI tool sessions too; Microsoft notes that rotating a key doesn't always end live sessions.

It's a long list, but it's finite, and every step closes a specific door.

How to stop the next one

After March, LiteLLM had moved to trusted publishing, immutable Docker tags and signed images. The attacker didn't break those controls; they drove the real pipeline with real credentials, and the build provenance honestly recorded Berri-AI/litellm as the source. The answer is layers, not one control.

Control What it does Would it have helped here?
Lock files with hashes Installs exactly the files you tested (uv.lock, poetry.lock, or pip install --require-hashes) Yes, if you were pinned to 1.106.2
Delayed adoption Ignores releases younger than a set age: uv's exclude-newer = "7 days", pip's --uploaded-prior-to P7D Yes: both poisoned LiteLLM releases were pulled within hours
Image digests, not tags Runs image@sha256:... instead of latest Yes: floating tags were repointed at the bad image
Check provenance, not just signatures Confirms the source repository named in the attestation is the real one Yes: the provenance named the lookalike fork
Scoped, short-lived tokens Uses fine-grained tokens with expiry and OIDC instead of personal CLI logins Partly: the chain started with a personal owner token and an unrevoked key
Egress allowlists Lets gateway and build hosts reach only the APIs they need Yes: posts to www.litellm.dev would have failed
Two-person release approval Requires a second human for each release Partly: that week LiteLLM had relaxed this rule and temporarily widened release permissions

Three habits matter most:

  • Delayed adoption is the cheapest win. Both LiteLLM compromises this year were caught the same day, so a seven-day wait would have skipped both. Keep a fast path for real security fixes.
  • Revoke leaked keys; don't just delete the commit. An overwritten commit doesn't make a key useless; only revoking does.
  • Fence off the gateway. Give each app its own virtual key with a spend limit, and run the gateway as a non-root container with no other credentials on the host, as in our Docker containerisation work. For CI, our guide to GitHub Actions and AWS OIDC replaces stored keys with short-lived ones.

The npm ecosystem learned the same lessons in 2025; our npm supply chain attacks article covers the JavaScript side.

The March 2026 incident: same project, different route

On 24 March 2026, versions 1.82.7 and 1.82.8 appeared on PyPI from 10:39 UTC and were live for about 40 minutes before quarantine, LiteLLM says. It treats anyone who installed between 10:39 and 16:00 UTC as possibly affected.

That attacker skipped the release pipeline: LiteLLM believes the compromise came through Trivy, a security scanner in its CI, and the packages were uploaded straight to PyPI. Version 1.82.8 used the same trick, a litellm_init.pth startup hook.

Datadog links March to a campaign called TeamPCP and says no connection to October has been established.

Frequently asked questions

Is LiteLLM safe to use now?

As of 26 October 2026, the malicious 1.107.0 has been removed, 1.106.2 is the last version published before the incident, and LiteLLM resumed releases on 24 October (1.106.4 is the latest) with a new image-signing key. LiteLLM says it is revoking every internal token and has removed GitHub permissions from its release workflow. The problem was the release process, not the code, so pin exact versions by hash and wait a few days before upgrading, whichever gateway you use.

What is a .pth file in Python, and why is it dangerous?

A .pth file in site-packages normally just adds folders to Python's import path, but any line in it starting with import runs every time Python starts, whether or not your code uses that package. That turns one tiny file into code that runs in every Python process, and it survives pip uninstall.

Was the LiteLLM Docker image affected?

In October, yes. The images on Docker Hub and GHCR tagged v1.107.0, latest, main-latest and main-stable pointed at a poisoned image from 11:46 UTC until between about 13:51 and 14:20 UTC on 21 October 2026, so anyone running a floating tag could have pulled it without choosing 1.107.0. In the March 2026 incident, the official Docker image was not affected because it pinned its dependencies.

The bottom line

The way into the LiteLLM supply chain attack wasn't clever: a leaked key nobody revoked, a few personal tokens, and a build tool told to copy one extra folder. The defences are just as ordinary: pin by hash, wait before upgrading, run images by digest, scope your tokens and limit where your gateway can talk to. Treat the box that holds every AI key like the box that holds every password, because it is one.

Start today with step 1 of the check above: search your lock files for 1.107.0. Running an AI gateway or a Python build pipeline and not sure how exposed it is? Book a free consultation and we'll go through your dependency pinning, secrets and egress rules with you.

Sources

Originally published on znovastudio.com.

Top comments (0)