On March 24, 2026, a supply chain attack against LiteLLM — one of the most widely deployed LLM proxy and observability libraries — compromised the PyPI packages used by engineers to instrument their AI agents. The attacker had already worked through two upstream targets: Aqua Security's Trivy scanner (March 19), then Checkmarx's KICS and AST GitHub Actions (March 23). LiteLLM's CI/CD pipeline ran Trivy without a pinned version, which is how the attacker extracted the PyPI publishing token from the GitHub Actions runner. The malicious versions (1.82.7 and 1.82.8) were live on PyPI for approximately three hours before LiteLLM took them down.
The payload installed by the compromised packages ran a three-stage operation: credential harvesting from cloud environments, lateral movement through Kubernetes clusters, and a persistent backdoor for remote code execution. Every API key passing through LiteLLM — OpenAI, Anthropic, Azure, Google — was potentially in scope.
The monitoring layer wasn't a side effect of the attack. It was the target.
AI agent observability supply chain risk refers to the class of security vulnerabilities that arise when the SDKs, proxies, and telemetry libraries that instrument AI agent behavior become attack targets in their own right. Unlike traditional application security, where monitoring tools are assumed to be outside the threat model, AI agent observability infrastructure sits in the critical path between every agent action and the LLM provider — giving it access to API credentials, prompts, model outputs, and session traces simultaneously. When these tools are compromised, the attacker gains visibility and access equivalent to every agent the tool monitors. Mitigating this risk requires a governance architecture where the enforcement layer is structurally separate from the instrumentation layer it governs — not co-located in the same SDK or in-process library.
Why is an LLM observability SDK a high-value supply chain target?
An LLM observability SDK is not a passive logger. It is middleware.
To capture traces, token usage, latency, and tool call records, the SDK must intercept the LLM call before it goes out and after it comes back. That means it lives between your agent code and your LLM provider — with read access to every prompt, every completion, every function call argument, and every API key that authorizes the transaction.
A compromised logging library in a traditional web application exposes request payloads. A compromised observability SDK in an AI agent stack exposes credentials across every LLM provider you've connected, every tool call your agent made, every piece of user data your agent processed in its context window, and the full execution history of every session since installation.
This is what made LiteLLM specifically attractive to TeamPCP. LiteLLM functions as a unified AI gateway — a single SDK that proxies requests to OpenAI, Anthropic, Azure OpenAI, Google, Bedrock, and dozens of other providers behind a common interface. One compromised installation gives an attacker access to credentials for every AI provider the team uses, not just one.
The payload's three-layer structure was deliberate. The first stage launched the stealer in a detached background process — no visible output, no Python startup delay. The second stage ran deep credential harvesting: SSH keys, cloud provider configs, LLM API keys, .env files, database passwords, CI/CD secrets, crypto wallets, all encrypted with AES-256 before exfiltration. The third stage established persistence by writing a scheduled service (typically sysmon.py) that checked in with attacker infrastructure every 50 minutes. An aggressive single-stage credential dump would have triggered rate-limit alerts. The staged approach established a persistent foothold before generating any detectable network activity.
Most teams have threat models that account for what their agents do. Far fewer have threat models that account for what their monitoring tools can see.
How did the LiteLLM attack chain work?
The attack didn't start with LiteLLM. TeamPCP ran a cascading supply chain campaign across three targets over five days.
On March 19, they trojanized Aqua Security's Trivy container security scanner. Trivy is widely used in CI/CD pipelines — including LiteLLM's own build process. On March 23, TeamPCP hit Checkmarx's KICS and AST GitHub Actions, which further expanded the footprint. By March 24, when their compromised Trivy build ran inside LiteLLM's CI/CD pipeline, it extracted the PYPI_PUBLISH token from the GitHub Actions runner environment. From there, TeamPCP published versions 1.82.7 and 1.82.8 directly to PyPI under LiteLLM's legitimate package identity.
The malicious .pth file dropped by the compromised packages executes automatically on every Python process startup when LiteLLM is installed in the environment. Not just when you explicitly import LiteLLM — on every Python invocation. Any team that ran pip install litellm without a pinned version during the three-hour window, or rebuilt a Docker image that included an unpinned install, received the payload.
The practical lesson isn't just "pin your dependencies" — though you should. The lesson is that AI tooling supply chains are now active targets, and the tools with the broadest access to your agent stack are the highest-value ones to compromise. An attacker who wants your OpenAI API key doesn't need to break into your application. They can compromise the observability SDK you trusted to watch your application for you.
What was actually exposed: why LLM gateway credentials are uniquely dangerous
When a traditional application credential is stolen, the attacker can impersonate the application. When an LLM gateway credential is stolen, the attacker can impersonate every agent the gateway serves — and do so invisibly, because the traffic looks identical to legitimate agent traffic.
An attacker with a stolen OpenAI organization key doesn't just get API access. They get access at your rate limits, with your billing, generating outputs that your observability tools will log as legitimate if the observability layer itself is the source of the compromise. There is no external signal that anything went wrong.
In the LiteLLM attack, the credential scope went further still. LiteLLM is designed to centralize API key management — teams configure provider credentials once, and the SDK manages them for all downstream agents. That means a single PYPI_PUBLISH token extraction cascaded into potential access to every LLM provider credential stored in environments where the compromised package installed.
Combined with the ClawHavoc campaign — where researchers ultimately catalogued 1,184 malicious Skills injected into OpenClaw's ClawHub marketplace between January and February 2026, representing roughly 12% of the registry at first discovery — the pattern emerging in early 2026 is consistent: the AI tooling ecosystem above the application layer is being actively targeted. Package registries, marketplace plugins, security scanner dependencies, CI/CD action integrations. Not the agents themselves. The infrastructure around the agents.
The architectural problem: when monitoring sits in the execution path
The core risk surfaced by the LiteLLM attack is architectural, not operational. Patching the compromised package addresses the immediate incident. The underlying structure remains.
Most AI agent observability architectures instrument at the SDK layer — a library that wraps LLM calls from inside the agent process or via an in-path proxy. This is pragmatic: it's easy to add three lines of instrumentation, and it captures everything. But it also means the instrumentation library shares the trust domain of the code it instruments. A compromised observability SDK has the same credential access as a compromised agent. The monitoring tool and the thing being monitored are not separated by any trust boundary.
Contrast this with governance enforcement at the infrastructure layer — a control plane that sits outside the agent process, evaluates requests before they execute, and maintains its own credential store separate from the agent environment. A governance control plane can observe and enforce agent behavior without residing inside the same process, which means a supply chain attack against the agent SDK does not automatically compromise the governance layer.
This is the architectural gap the LiteLLM incident makes concrete. Observability tools that are co-located with the agent code they instrument cannot verify their own integrity. Governance enforcement that operates at a separate infrastructure layer can.
The parallel in traditional security architecture is the SIEM/EDR distinction: you don't run your endpoint detection tooling in the same trust domain as the applications it monitors, because that defeats the independence the monitoring is supposed to provide. AI agent governance needs the same separation.
What does this mean for your AI governance posture?
The LiteLLM attack has three practical implications for teams running agents in production.
Your observability tools are in your threat model now. Any SDK, proxy, or library with access to your LLM credentials or agent traces is a supply chain attack surface. Pinning versions is necessary but not sufficient — the LiteLLM attack demonstrates that a legitimate version can be replaced at the registry level under the correct package name. Signed artifact verification and provenance checking for your AI tooling supply chain are the same discipline as for any other critical dependency.
Governance enforcement should not share a trust domain with instrumentation. If your governance policies (cost limits, content filtering, access controls) are enforced inside the same SDK or in-process library as your observability collection, then a supply chain attack against that library bypasses your governance simultaneously. Enforcement at an external infrastructure layer — one that doesn't reside in the agent process — maintains governance even when the agent-side SDK is compromised.
Audit trails need to be tamper-evident, not just present. An attacker with access to your observability SDK can manipulate what gets logged. A governance architecture where enforcement events are recorded at the infrastructure layer and not modifiable by agent-side code produces records that are meaningful for compliance even in adversarial conditions. "We have logs" is a weaker claim than "our enforcement records are generated outside the agent process."
This isn't an argument against current observability tools. It's an argument for treating observability and governance as architecturally distinct layers with different trust requirements — because conflating them creates exactly the exposure the LiteLLM incident made visible.
How Waxell handles this
How Waxell handles this: Waxell's governance enforcement layer operates at the infrastructure level — outside the agent process, evaluated before each tool call and LLM request. Execution tracing captures the full session graph, but the enforcement control plane maintains its own credential context separate from agent-side SDKs. This separation means a supply chain attack against an agent's observability library does not compromise Waxell's enforcement records or policy execution. When the LiteLLM incident prompted teams to audit their AI tooling trust boundaries, Waxell's architecture already placed governance policy enforcement outside the attack surface that SDK-layer observability tools share with the agent code they instrument. That's not a feature we added in response to this incident — it's what a governance-above-agents architecture looks like by design. To see your full agentic governance posture, get early access to Waxell.
Frequently Asked Questions
What happened in the LiteLLM supply chain attack?
On March 24, 2026, attackers from the group TeamPCP published malicious versions of the LiteLLM Python package (1.82.7 and 1.82.8) to PyPI after compromising LiteLLM's CI/CD pipeline via a trojanized version of the Trivy security scanner. The malicious versions were live for approximately three hours and deployed a three-stage payload: credential harvesting from cloud environments, lateral movement through Kubernetes clusters, and a persistent backdoor for remote code execution. Engineers who installed or upgraded LiteLLM during that window, or rebuilt Docker images without a pinned version, received the compromised package.
Why are LLM observability SDKs high-value supply chain targets?
LLM observability SDKs sit in the execution path between your agent code and your LLM provider. To capture traces and telemetry, they must intercept calls before they go out and after they come back — which means they have read access to API credentials, prompts, model completions, tool call arguments, and full session histories. An LLM gateway SDK like LiteLLM that centralizes credentials for multiple providers is particularly high-value: compromising one package gives an attacker access to every provider credential configured in environments where it's installed.
What is AI agent observability supply chain risk?
AI agent observability supply chain risk is the category of vulnerabilities that arise when the SDKs, proxy libraries, or telemetry tools used to monitor AI agent behavior are themselves compromised. Unlike traditional application monitoring tools, AI agent observability infrastructure has privileged access to LLM credentials, agent prompts, and session traces — making it a high-value target. The risk is distinct from prompt injection or model safety risks because it operates at the tooling layer, not the agent layer. It's mitigated by separating governance enforcement from in-process instrumentation, maintaining signed artifact verification for AI dependencies, and ensuring enforcement records are generated outside the agent trust domain.
How should teams respond to the LiteLLM supply chain attack?
Immediately: verify your installed LiteLLM version and confirm you are not running 1.82.7 or 1.82.8. If you installed from PyPI during March 24 without a pinned version, rotate all LLM provider API keys that were configured in the affected environment, including OpenAI, Anthropic, Azure, and any other providers LiteLLM was proxying. Audit Kubernetes credentials for any signs of lateral movement. Longer-term: pin AI tooling dependencies to verified versions, implement artifact signing verification for your AI stack, and evaluate whether your governance enforcement layer shares a trust domain with the agent SDKs it's meant to govern.
What is the difference between AI agent observability and AI agent governance when it comes to security?
Observability tools record what your agents did — they are instrumentation layers that capture traces, costs, and tool call records. Governance enforcement determines what your agents are allowed to do — it evaluates requests before they execute and enforces policies at the infrastructure layer. For security purposes, the critical difference is trust domain separation: observability SDKs that run inside the agent process share the attack surface of that process. Governance enforcement at an external infrastructure layer maintains its integrity even when the agent-side SDK is compromised. The LiteLLM supply chain attack is a direct demonstration of why these layers should not be co-located.
Sources
- LiteLLM, Security Update: Suspected Supply Chain Incident (March 2026) — https://docs.litellm.ai/blog/security-update-march-2026
- Wiz, LiteLLM TeamPCP Supply Chain Attack: Malicious PyPI Packages (March 2026) — https://www.wiz.io/blog/threes-a-crowd-teampcp-trojanizes-litellm-in-continuation-of-campaign
- Snyk, How a Poisoned Security Scanner Became the Key to Backdooring LiteLLM (March 2026) — https://snyk.io/articles/poisoned-security-scanner-backdooring-litellm/
- Sonatype, Compromised litellm PyPI Package Delivers Multi-Stage Credential Stealer (March 2026) — https://www.sonatype.com/blog/compromised-litellm-pypi-package-delivers-multi-stage-credential-stealer
- Snyk, ToxicSkills: Malicious AI Agent Skills in ClawHub Supply Chain (February 2026) — https://snyk.io/blog/toxicskills-malicious-ai-agent-skills-clawhub/
Top comments (0)