DEV Community

Cover image for Ten Days After LiteLLM: Why AI Teams Without Audit Trails Are Flying Blind in Breach Response
Waxell profile image Logan
Logan for Waxell

Posted on • Originally published at waxell.ai

Ten Days After LiteLLM: Why AI Teams Without Audit Trails Are Flying Blind in Breach Response

#ai #security #litellm #python

At 10:39 UTC on March 24, 2026, threat actor group TeamPCP published litellm 1.82.7 to PyPI. At 10:52 UTC, they published 1.82.8. By 11:19 UTC, both versions had been quarantined by PyPI. Forty minutes.

In that window, any Python process that installed litellm from PyPI — in a container build, a CI/CD pipeline, or a running production environment — executed a malicious .pth file that automatically harvested SSH keys, cloud credentials, Kubernetes configs, and API tokens, then staged them for exfiltration to attacker-controlled infrastructure at models.litellm.cloud.

It is now April 10, 2026. Mercor has confirmed the breach. The Lapsus$ extortion group has claimed the theft of more than 4TB of data — approximately 939 GB of platform source code, 211 GB of user database records, and roughly 3 TB of storage buckets containing video interview recordings and passport scans from more than 40,000 contractors — and has begun auctioning the stolen material on dark web forums. Meta has indefinitely paused all contracts with Mercor. At least five contractor lawsuits were filed within the first week. Mercor has said it believes it was "one of thousands" of organizations affected.

The question most affected enterprises cannot answer: which of your agent sessions ran litellm 1.82.7 or 1.82.8? Can you prove it? Can you scope the exposure?

An AI governance audit trail is a durable, policy-enforced execution record that captures every LLM call, tool invocation, external network request, credential usage, and session event made by an AI agent — independent of the agent's own logging, written at the infrastructure layer, and queryable after the fact for forensic scoping and compliance documentation. It is distinct from application-level logs (which agents control and which malicious code can suppress) and from billing dashboards (which aggregate usage without session-level forensics). An agentic governance audit trail is what tells you, with certainty, which sessions ran during a window of compromise — and what they touched.

What did LiteLLM 1.82.7 and 1.82.8 actually do to your agents?

LiteLLM is the de facto proxy library for enterprise AI. With approximately 97 million monthly downloads and an estimated presence in 36% of cloud environments, it is the layer that connects agents to LLM providers: OpenAI, Anthropic, Gemini, local models. Most enterprise agent stacks install it without a second thought, the same way they install requests or boto3.

The attack exploited a dependency in LiteLLM's own CI/CD pipeline. LiteLLM ran Trivy — an open-source vulnerability scanner maintained by Aqua Security — as part of its build process. TeamPCP had already compromised Trivy by rewriting Git tags to point to a malicious release carrying credential-harvesting payloads. The same Trivy compromise, beginning around March 19, 2026, had already been used to breach the European Commission's AWS infrastructure; CERT-EU publicly confirmed on March 27 that 92 GB of compressed Commission data was stolen via the same Trivy supply chain attack. After the Trivy compromise established the technique, LiteLLM's CI/CD pipeline pulled the compromised Trivy action and executed it, which exfiltrated the PyPI_PUBLISH token from the GitHub Actions runner environment. With that token, TeamPCP published the backdoored litellm versions directly to PyPI under the legitimate package name.

The malicious payload was a .pth file — litellm_init.pth — that Python's import machinery executes automatically on every process startup, without requiring any explicit import of litellm. This means a containerized agent that installed litellm at build time and then ran for the next several hours was silently executing the payload on every startup. The payload ran a three-stage operation: credential harvesting (SSH keys, cloud tokens, Kubernetes secrets, .env files, database passwords), lateral movement across Kubernetes clusters by deploying privileged pods, and persistent backdoor installation as a systemd service that auto-restarted every 10 seconds.

The data was encrypted and bundled into a file named tpcp.tar.gz and exfiltrated to models.litellm.cloud.

Why did Meta pause Mercor — and what does that tell you about AI vendor risk?

Mercor is an AI hiring platform valued at approximately $10 billion. It used LiteLLM as infrastructure, and the malicious package ran in its environment during the 40-minute window. Confirmed stolen: approximately 939 GB of platform source code, 211 GB of user database records, and roughly 3 TB of storage buckets containing video interview recordings and identity verification documents, including passport scans belonging to more than 40,000 contractors.

Meta was one of Mercor's enterprise customers. When the breach became public on March 31, Meta moved immediately — indefinitely pausing all contracts with Mercor, which in practice means halting AI training data operations that relied on the Mercor platform.

This is the detail that matters for enterprise risk management: Meta did not investigate for weeks before acting. When a critical AI vendor disclosed a breach of this scope, the enterprise response was immediate suspension. The speed of that decision reflects how the calculus works when AI vendors handle training data, proprietary model infrastructure, and contractor PII.

The Mercor breach is, as StrikeGraph noted, an illustration of a structural risk the AI industry has rarely confronted at scale: when multiple enterprises rely on the same third-party AI data supplier, a single breach can expose the competitive secrets of all of them simultaneously. The TeamPCP campaign, confirmed by CERT-EU, is the same group that breached the European Commission's AWS infrastructure through the earlier Trivy compromise — a breach publicly disclosed on March 27, 2026, affecting at least 71 institutions. Mercor is one node in a much larger supply chain failure.

Ten days later: can you prove which of your agent sessions ran the compromised version?

This is the question multiple plaintiff law firms are asking enterprises right now, and most engineering teams don't have a clean answer.

The affected window is defined: litellm 1.82.7 and 1.82.8 were live from 10:39 UTC to approximately 11:19 UTC on March 24, 2026. Any environment that installed litellm during that window, or that had it cached from a build earlier that day depending on your Docker layer caching strategy, was potentially exposed. Any process that ran the malicious .pth file at startup executed the payload.

Scoping this exposure requires answering several questions:

Which of your containerized agent environments ran litellm builds during or after that window? Which agent sessions started up during or after the window and therefore would have executed the malicious .pth file? What external network connections did your agent processes make during that window — specifically, did any session make a connection to models.litellm.cloud? What credentials were accessible in the environment of each affected agent session?

For enterprises with durable execution records at the agent infrastructure layer, these questions have deterministic answers. You pull the execution traces for the relevant time window, filter for sessions where litellm was loaded, check the external network call log, and produce a scoped forensic report that tells you exactly which sessions were affected and what they had access to.

For enterprises without session-level execution tracing at the infrastructure layer — which is most of them — you are in the worst position for breach response: you know something bad happened, you cannot prove the scope, and you are producing discovery responses for litigation without the documentation to support them.

The five contractor lawsuits filed against Mercor within the first week of the breach announcement are the downstream consequence of inadequate cybersecurity documentation. They allege failure to maintain adequate protections for more than 40,000 people. Whether Mercor wins or loses those cases, the discovery process will require it to demonstrate what data was accessed, by what sessions, and under what controls. The audit trail — or the absence of it — determines whether that demonstration is possible.

What a runtime governance audit trail would have captured

The attack's exfiltration step required making outbound network connections from the compromised process to models.litellm.cloud. That is observable behavior. An agent runtime that maintains an infrastructure-layer execution record of every external network call made during a session — with timestamps, destination, and session context — would have logged that connection in real time.

A behavioral anomaly detection policy that monitors for unexpected outbound connections from agent processes — specifically, connections to endpoints not in the approved egress list — would have flagged it. An enforcement policy that blocks outbound connections to unapproved endpoints would have stopped the exfiltration even if the malicious code executed, because the network call would have been intercepted before it left the environment.

Runtime governance that operates at the infrastructure layer, below the agent's own code, provides this because it instruments the execution environment independently of what the agent code does. The malicious litellm_init.pth file executes before the agent's own application code runs. It cannot suppress infrastructure-layer telemetry because that telemetry is written at a layer the payload doesn't control.

Separately, an infrastructure-layer execution record gives you the forensic scoping capability the class action plaintiffs will demand. You can pull every session that ran during the window, every external call made by those sessions, and every credential or resource those sessions accessed. That's the difference between a scoped incident ("sessions A, B, and C made the call; here is what they had access to; all other sessions show clean records") and an unscoped one ("we don't know which sessions were affected").

How Waxell handles this

How Waxell handles this: Waxell's execution tracing instruments agent environments at the infrastructure layer — below application code, independent of what the agent or its dependencies log. Every LLM call, tool invocation, and external network request is captured with session context and timestamps, written to a durable record that the agent's own code cannot suppress or modify. Runtime enforcement policies can define an approved egress list and block outbound connections to unexpected endpoints in real time — including, in the LiteLLM scenario, a connection to models.litellm.cloud from an agent session that had no legitimate reason to contact that endpoint. Compliance assurance documentation — the enforcement record showing what policies were evaluated, what was allowed, and what was blocked — is embedded in each execution trace, queryable after the fact for incident scoping and legal discovery. Three lines of SDK to instrument; the governance layer operates independently of any dependency code change. Get early access to the full governance stack.

Frequently Asked Questions

What was the LiteLLM supply chain attack?
On March 24, 2026, threat actor group TeamPCP published backdoored versions of the litellm Python package (1.82.7 and 1.82.8) to PyPI after stealing the library's PyPI publish credentials through a prior compromise of Trivy, an open-source security scanner used in LiteLLM's CI/CD pipeline. The malicious packages contained a .pth file that executed automatically on every Python process startup, harvesting credentials and attempting lateral movement across Kubernetes clusters before exfiltrating stolen data to attacker-controlled infrastructure. The packages were available on PyPI for approximately 40 minutes before being quarantined.

Was my organization affected by the LiteLLM breach?
Any environment that installed litellm 1.82.7 or 1.82.8 — or that ran a container built with those versions — may have executed the malicious payload. Mercor has stated it believes it was "one of thousands" of organizations affected. To determine exposure, you need to establish whether any of your environments installed those specific versions during or after the 40-minute window, and whether any agent sessions that ran during that period made outbound connections to models.litellm.cloud. Organizations with infrastructure-layer execution tracing can answer these questions definitively; those relying only on application-level logs may not be able to.

How do you detect a supply chain attack on an AI library like LiteLLM at runtime?
Runtime detection requires monitoring behavior at the infrastructure layer, not just the application layer. Specifically: any outbound network connection from an agent process to an unexpected endpoint is a detectable anomaly. The LiteLLM malicious payload exfiltrated data to models.litellm.cloud — an endpoint that no legitimate agent workflow would contact. An enforcement policy that maintains an approved egress list and blocks unapproved outbound connections would have stopped the exfiltration even if the malicious code executed. Infrastructure-layer instrumentation that operates below the dependency code can log these connections even if the payload itself suppresses application logging.

What is an AI governance audit trail and why does it matter for breach response?
An AI governance audit trail is a durable, infrastructure-layer record of every action an agent session takes: LLM calls, tool invocations, external network requests, token usage, credential access, and session events. It is written independently of the agent's own code and cannot be suppressed by compromised dependency code. In breach response, an audit trail provides the forensic scoping capability that litigation discovery requires: which sessions ran during a window of compromise, what they accessed, and what external connections they made. Without it, enterprises in breach response cannot bound their exposure — they know something happened but cannot prove what, which makes discovery obligations for class action litigation extremely difficult to meet.

How does the Mercor breach affect enterprises that use third-party AI vendors?
The Mercor breach illustrates a risk that is structural to the AI ecosystem: multiple enterprises sharing the same third-party AI infrastructure vendor creates a single point of failure that can expose competitive secrets and sensitive data simultaneously. Meta's response — immediately pausing all contracts — shows how quickly enterprise relationships can be suspended when a vendor discloses a breach of this scale. Enterprises evaluating AI vendors should now require evidence of supply chain security practices, dependency pinning, runtime monitoring, and incident response procedures, not just SOC 2 certification. For enterprises with their own agents, the lesson is that your attack surface now includes every dependency in every agent's environment — not just your own code.

What is the difference between a supply chain breach and a direct breach for AI governance purposes?
A direct breach attacks your systems. A supply chain breach attacks a dependency your systems trust implicitly, meaning the attack executes with your environment's own permissions and credentials. For AI governance, this means your runtime environment — including agent API keys, cloud credentials, and data access — is exposed through a mechanism that bypasses perimeter controls. The appropriate governance response is behavioral monitoring at the execution layer: watching what your agent environments actually do at runtime, regardless of which code triggered that behavior. A policy that blocks outbound connections to unapproved endpoints applies regardless of whether the connection was initiated by your own agent code or by a compromised library.

Sources

Top comments (0)