DEV Community

Rohit Bhadani
Rohit Bhadani Subscriber

Posted on

The LiteLLM Master Key Leak Sat Unrevoked for 136 Days, Then Hijacked a Release Pipeline

The LiteLLM master key leak started as a routine mistake on May 27, 2026, when a configuration file containing a live master key for LiteLLM's Agent Platform was committed to a public repository. The team caught it fast, force-pushed over the commit, and flagged the exposed key for revocation. Nobody actually revoked it.

One hundred and thirty-six days later, on October 21, 2026, that same key was the first domino in a chain that ended with a malicious litellm==1.107.0 published to PyPI, Docker Hub, and GHCR.

Jump to what we checked in our own secret-rotation process if your incident response for a leaked credential ends at "fixed the commit."

Why Force-Pushing Didn't Fix Anything

Force-pushing over a commit removes it from a repository's visible branch history. It does not reach every fork, clone, mirror, or automated scraper that already captured the file before the force-push landed, and public GitHub repositories get scraped by credential-harvesting bots within minutes of a push, routinely faster than a human notices the mistake. LiteLLM's own account of the incident is direct about this: the exposed key was "identified as requiring revocation," and then revocation simply didn't happen. The commit looked fixed. The key stayed live.

How One Unscoped Token Became Full Organization Access

The leaked master key unlocked LiteLLM's Agent Platform, an experimental system that let self-hosted cloud agents file pull requests to GitHub repositories on the team's behalf. To do that, the platform held a GitHub CLI OAuth token, and according to LiteLLM's own root-cause analysis, that token was not tied to a scoped-down service account. It belonged to the GitHub organization owner.

On October 10, 2026, the attacker used the still-valid master key to call the Agent Platform API and pull that owner-level token. LiteLLM notes a detail worth sitting with: GitHub CLI tokens are post-2FA and aren't subject to the extra reauthentication GitHub normally requires for sensitive organization actions. Owning that one token meant owning the organization, no second factor required.

From there the attacker authored commits on throwaway branches to trigger CI workflows that exfiltrated every other secret available to those repositories' pipelines, acquiring GitHub tokens from two more engineers with maintainer privileges along the way, then deleted the workflow runs within minutes to reduce the audit trail.

The Release Safeguard That Got Relaxed the Same Week

This is the detail that turns a bad key leak into a full supply-chain compromise. In normal operation, LiteLLM requires a release to be kicked off by an authorized engineer and approved by a special account locked to a physical device in the office, a safeguard specifically meant to stop exactly this kind of single-stolen-credential attack. Earlier in the same week as the attack, that restriction was relaxed to accommodate a temporary push for more frequent releases and bugfix backports, allowing any two engineers to trigger and approve a release instead.

With two stolen maintainer identities in hand, the attacker didn't even need the owner token for this part: they used one stolen identity to trigger a release pointed at a malicious lookalike repository, and a second to approve it.

LiteLLM's team notes that the org owner's token alone could have bypassed this process entirely, but using two separate lower-privilege identities kept the audit logs quieter. The malicious release reached PyPI, Docker Hub, and GHCR around 11:40 UTC on October 21. LiteLLM caught it roughly 80 minutes later, froze the PyPI release by 13:23 UTC, and pulled the Docker images by 14:20 UTC after waking the people with registry access.

What We Checked in Our Own Secret-Rotation Process

We pulled every incident in Krova Cloud's own history where a secret was exposed in a public commit and checked how each one actually ended: specifically, whether the ticket closed on "force-pushed, key flagged for rotation" or on a confirmed, logged rotation event. We found one old incident from before our current process where the fix matched the first pattern. The key in question was for a decommissioned staging service, so the practical exposure was limited, but we rotated it anyway and added a rule that no exposed-secret ticket can close without a rotation confirmation attached, not just a flag.

What to Check in Your Own Credential Hygiene

  • Confirm that "exposed key flagged for revocation" and "key actually revoked" are two separate, independently tracked steps in your incident process, not one step that looks done after the first.
  • Audit every token tied to an automation or agent platform for scope. A token that only needs to open pull requests should never carry organization-owner privileges.
  • Check whether any release, deploy, or merge safeguard at your company has been "temporarily relaxed" recently for speed, and if so, confirm someone has a date to put it back.
  • If you use LiteLLM, confirm you are not running the compromised litellm==1.107.0 release or its associated Docker images, and rotate any credentials those environments held.

The honest limitation here is that LiteLLM's own account, while unusually detailed, is self-reported and was still being finalized as this post went out; the company explicitly said its process-improvement writeup was still to come. We also don't know how many other tokens the 136-day exposure window may have touched beyond the ones LiteLLM has traced, since the company itself is now operating on the assumption that every organizational token could have been exfiltrated, not just the ones with confirmed activity.

This is the same root lesson as Zeabur's unrotated administrator credentials: identifying that a credential needs rotating and actually rotating it are two different events, and treating the first as if it were the second is how a caught mistake turns into a live breach months later.

The verdict: LiteLLM did almost everything right in the first five minutes on May 27 and still got breached in October, because the one step that actually closes a credential leak, revocation, was the step that got skipped.

The rule: a flagged-for-revocation key is not a resolved incident until someone can point to the log entry where it was actually revoked, and until then it should be tracked as an open, live risk, not a closed ticket.

FAQ

What was the LiteLLM master key leak?
A configuration file containing the master key to LiteLLM's experimental Agent Platform was committed to a public repository on May 27, 2026. The team force-pushed over the commit and flagged the key for revocation, but the revocation never happened, leaving the key valid for 136 days.

How did the LiteLLM master key leak lead to a malicious release?
The unrevoked key let an attacker pull an unscoped GitHub CLI OAuth token belonging to the organization owner on October 10, 2026. That access was used to exfiltrate more maintainer tokens, which were then used on October 21 to bypass release safeguards and publish a malicious litellm==1.107.0 to PyPI, Docker Hub, and GHCR.

Is the malicious LiteLLM release still available?
No. LiteLLM froze the PyPI release by 13:23 UTC on October 21, 2026, and removed the affected Docker images by 14:20 UTC the same day, roughly 80 minutes and three hours after the release was first published, respectively.

What should I do if I installed litellm==1.107.0?
Rotate any credentials present in the environment that ran the package, since LiteLLM has confirmed the release was compromised, and check your systems against LiteLLM's own incident disclosure for the specific indicators of compromise.

I'm Rohit, founder of Krova Cloud: disposable, per-minute-billed VMs where a leaked key in a test environment can't sit live for 136 days, because the environment itself is gone within minutes of the job finishing. More debugging stories at debugly.dev.

Top comments (1)

Collapse
 
shuttlewright profile image
Shuttlewright •

The detail that stings most is "identified as requiring revocation" and then nothing. Flagged-for-revocation is an intention, not a state. The closed loop is cheap: after rotating, run a canary that actually authenticates with the leaked key and alerts until it hard-fails. Revocation is done when the key is proven dead, not when someone meant to kill it. The same logic applies to the force-push - scrapers harvest public repos within minutes, so the exposure clock starts at the commit, not at discovery. A secret that reaches a public branch should be treated as already sold; the only open question is how fast you can prove the old one refuses auth.