A $10B AI Startup Just Got Breached Through the LLM Library in Your Stack.

#ai #llm #security #devops

Book: Observability for LLM Applications — paperback and hardcover on Amazon · Ebook from Apr 22
Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go
My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
Me: xgabriel.com | GitHub

Every LLM call your product makes passes through one library. In April 2026, that library was the breach path to a $10 billion AI startup.

Mercor, the AI-driven recruiting platform now valued at roughly $10B, confirmed a security incident that investigators traced back to a supply-chain compromise of LiteLLM, the widely-used open-source LLM gateway. Fortune covered the disclosure. The CISO Platform breach intelligence writeup walked through the attack chain. If your production AI stack depends on LiteLLM — and a lot of stacks at OpenAI, Anthropic, Meta, and most AI-native startups do — you need to sit with what this actually means.

It is not a "patch and move on" incident. The gateway sees everything.

What a supply-chain attack on an LLM gateway actually looks like

Forget the usual mental model of a supply-chain breach, where a compromised package runs a crypto miner or exfiltrates a .env file on install. An LLM gateway is a more interesting target. Here is the attacker's view from inside the library:

Every prompt your users send, in plaintext, passes through a function the attacker now controls.
Every completion your model returns, including the reasoning traces, tool calls, and system-prompt echoes, passes through the same function.
Every provider API key (OpenAI, Anthropic, Bedrock, Gemini, Azure, the whole fleet) sits in the process memory of that library, because that is how the gateway authenticates outbound calls.
Every downstream tool call (your database, your internal APIs, your payment service) arrives as a structured object the gateway has to parse before forwarding.

An attacker with code execution inside that library does not need to break your application boundary. They already sit inside the trust zone that most threat models hand-wave as "the LLM call." One line added to a retry path, or an HTTP pre-hook, or a logging callback, and they can copy prompts, responses, keys, and tool-call payloads to an external endpoint. Your application logs look clean. Your WAF sees nothing strange. Your gateway returns HTTP 200.

This is the uniquely bad shape of the Mercor breach: the attacker did not need to find a bug in Mercor's code. They needed a bug, or a malicious commit, in a transitive dependency almost every AI company takes for granted.

Why this shape is worse than a normal supply-chain hit

A compromised utility library is bad. A compromised LLM gateway is worse, for three structural reasons.

First, the blast radius is multiplicative. A single compromised LiteLLM version touches every provider key a customer has configured, and every end user whose prompts route through it. One package, hundreds of customers, millions of prompts, dozens of provider accounts.

Second, prompts and completions are high-signal data. User prompts to an AI recruiting platform include resumes, salary expectations, compensation history, interview transcripts, recruiter notes, candidate rankings. Completions include the model's reasoning about those inputs. This is not a leaked mailing list. This is a labelled training corpus for an attacker.

Third, detection is hard because the gateway is designed to be a chokepoint. You deliberately centralized every LLM call through one library so you could log it in one place, apply one rate limit, and enforce one key-management policy. The same property that makes the gateway operationally useful makes it a uniquely valuable target when it is subverted. A rogue span exporter at that layer can exfiltrate everything, and it looks exactly like the observability tooling you installed on purpose.

If you read this and thought "well, we pin versions, we are fine" — that is the audit this post is asking you to run.

The audit your team should run tomorrow

Not next quarter. Tomorrow. Every item below is cheap. The cost of skipping them is the Mercor-shape of outcome.

1. Pin the version, and check what you are actually running

If your requirements.txt, pyproject.toml, or package.json has litellm>=X or litellm~=X, that is a floating version. A single malicious release in the pinned range lands in your next container build.

# What LiteLLM is actually installed?
pip show litellm | grep -E "^(Name|Version|Location)"

# What does your lockfile pin?
grep -E "^litellm" requirements.txt poetry.lock uv.lock 2>/dev/null

Pin the exact version, commit the lockfile, and make sure your CI build fails if the resolved version drifts. For the teams already on uv or pdm, use the hash-pinned install mode so a re-published version with the same number cannot silently slip in.

2. Subscribe to the upstream security advisory feed

LiteLLM publishes GitHub Security Advisories. If you maintain a gateway in production and nobody on the team is subscribed, that is a gap. Do this:

Watch the repo with "Custom → Security advisories" enabled.
Add the CVE feed for the package to whatever you already monitor — GitHub's Dependabot, Snyk, Socket, OSV-Scanner, whatever you use.
File a ticket on your team's board: "Rotate LiteLLM version within 24 hours of any critical advisory." Make it an on-call runbook item, not a Slack DM.

3. Get secrets out of the gateway's process memory

This is the hard one, and it is the one that would have bounded the Mercor blast radius.

Most teams configure the gateway by passing OPENAI_API_KEY, ANTHROPIC_API_KEY, and so on as environment variables. Those keys live in the process memory of the LiteLLM worker for the lifetime of the process. A compromised library reads them with os.environ.

The mitigation is to move the key material out of the gateway process and into something the gateway has to call out to, per request, with a short-lived token:

AWS Secrets Manager or Google Secret Manager with a short TTL cache and an IAM role on the gateway worker.
HashiCorp Vault with dynamic secrets and per-request leases.
A sidecar auth proxy that injects the provider key at the network layer (mTLS between gateway and sidecar, provider key never reaches the gateway Python runtime).

None of these fully prevent a compromised gateway from stealing tokens mid-request — if the process handles the key at any point, a bad actor with code execution can grab it. What they do is bound the damage. A stolen short-lived token expires. An IAM role can be revoked. A static sk-... key cannot.

4. Rotate every provider key that has ever touched a LiteLLM process

If your gateway has been running on any LiteLLM version released inside the affected window (check the advisory when it lands), assume those keys are burned. Rotate them. OpenAI, Anthropic, Cohere, the small one you forgot about that you use for embeddings only, the Bedrock role, the Vertex service account. All of them.

Rotation is tedious, which is why teams skip it. Build the runbook once:

# Pseudocode for a rotation script — adapt to your setup
for provider in openai anthropic cohere bedrock vertex; do
  new_key=$(./scripts/mint-key.sh "$provider")
  ./scripts/push-to-vault.sh "$provider" "$new_key"
  ./scripts/revoke-old-key.sh "$provider"
done
./scripts/restart-gateway.sh

Run it in staging first. Measure the downtime. Bake it into your incident response plan so that the next time a library your gateway depends on ships a CVE, rotation is a 30-minute job instead of a week-long forensics effort.

5. Add prompt and response egress monitoring

A compromised library exfiltrates data to an external host. Your network egress from the gateway worker should be allowlisted to the set of provider API endpoints you actually use, plus your observability backend. Everything else gets blocked at the network layer, and the block is alerted on.

This is not a LiteLLM-specific control. It is generic egress hygiene. But most teams never apply it to their AI gateway because "the gateway needs to reach a lot of things." It needs to reach a knowable, enumerable set of provider endpoints. Put them in a list. Block the rest.

6. Separate keys from data in your audit logs

If your gateway logs full prompts and responses into an observability store, and the same store has any provider keys or tokens in its configuration, you have combined the two things an attacker would want. Separate them. Logs of prompts and completions go to one store with one access policy. Credentials go to a secrets manager with a different one. Nobody and no service should read both.

The broader 2026 lesson: the gateway is a chokepoint

A year ago, the pitch for an LLM gateway was: one library to abstract five providers, one place to add retries and caching, one dashboard to see spend. That pitch is still right. Every production AI team benefits from that consolidation.

What changed in 2026 is the rest of the sentence. A chokepoint is also an attack surface. The same property that makes the gateway operationally elegant (every call, every prompt, every key, every response, in one place) makes it a uniquely valuable target when it is subverted. March 2026 showed the community what a gateway incident looks like in the abstract. April 2026 showed what it looks like when the incident has a $10 billion name attached to it.

You do not fix this by ripping the gateway out. You fix it by treating the gateway like the critical piece of infrastructure it is: pinned, monitored, secret-isolated, egress-controlled, and included in your incident response plan by name. The teams that survive the next one of these will be the ones who did the audit above before the CVE landed, not after.

The next Mercor-shaped headline will name a different company and a different library. The pattern will be the same. Do the audit.

If this was useful

The gateway chapter of my observability book walks through what to instrument on the LiteLLM call path, what the OTel GenAI spans should carry, and how to wire egress monitoring into the same dashboard you already use for latency and cost. If the Mercor incident made you open this post, that chapter is the natural next read.

Hermes IDE is the editor I ship from when I am writing code that talks to these gateways — it is an IDE for developers who work with Claude Code and other AI coding tools, and it treats the gateway call path as a first-class thing to inspect.