If someone compromised one of your project's dependencies today, would they be able to steal your OpenAI, Anthropic, or Gemini API keys?
The answer isn't based on which LLM provider you use or how secure your codebase is. It mostly depends on one architectural decision that many teams never think about: where your provider API key actually lives while your application is running.
If that key lives inside your application's own process, every dependency running in that process shares the same environment. A compromised package doesn't need to break into your infrastructure. It simply executes with the same privileges as your application and can access the same credentials your code can access.
If the provider key lives in a separate network proxy instead, the application never holds the provider credential at all. Even if a dependency is compromised, the attacker can only access whatever limited credentials exist inside the application process. That doesn't eliminate risk, but it can reduce the blast radius when something goes wrong.
In this article, we'll look at the two dominant LLM gateway architectures, examine exactly where API keys live in each design, walk through a reproducible demo that shows the difference in practice, and discuss why reducing blast radius often matters more than trying to eliminate every possible attack.
LLM Gateway Architectures: In-Process vs Network Proxy
Every LLM application has the same fundamental job to do: send a request to a model provider and authenticate that request using an API key.
The important part isn't whether an application uses an API key. It's where that key exists while the request is being made.
Today, most AI applications follow one of two architectural patterns.
Architecture 1: The application holds the provider key
This is the architecture most developers are already familiar with.
Your application loads the provider API key from an environment variable, initializes an SDK or gateway library, and sends requests directly to OpenAI, Anthropic, Gemini, or another provider.
A simplified version often looks like this:
api_key = os.environ["PROVIDER_API_KEY"]
client = OpenAI(api_key=api_key)
response = client.responses.create(...)
It's easy to understand, quick to implement, and perfectly reasonable for many projects. The application owns the credential because it's the component talking directly to the provider.
The important detail is that the provider key now lives inside the application's process. Every package, framework, plugin, and dependency that executes in that process runs under the same privileges. If one of those dependencies is compromised, the provider key exists in the same environment as the malicious code.
The architectural question is this:
If something inside the application process is compromised, what secrets are available from there?
Architecture 2: The application talks to a network proxy
The second pattern separates authentication from the application itself.
Instead of sending requests directly to the model provider, the application sends them to a gateway or proxy. The proxy owns the provider API key and performs the upstream request on the application's behalf.
From the application's perspective, the flow looks almost identical:
Application
│
▼
Gateway / Proxy
│
▼
LLM Provider
The difference is what the application doesn't have.
Rather than storing the provider credential, it typically holds a scoped gateway token that authorizes requests through the proxy. The proxy validates that token, applies any routing or policy decisions, and then injects the provider API key only inside its own process before forwarding the request upstream.
This changes the consequences of a compromise. If malicious code executes inside the application process, it can still access whatever credentials the application possesses. The difference is that the provider API key is no longer one of them.
That doesn't make the application invulnerable. A stolen gateway token is still a security incident. However, unlike a provider API key, a gateway token can be narrowly scoped, centrally revoked, rotated without redeploying applications, and restricted to specific operations.
The easiest way to see that difference is by watching the exact same compromised dependency run against both architectures. That's what we'll do next.
How Supply Chain Attacks Expose LLM API Keys
The more important question is what happens after your application starts running.
Once a process begins executing, the credentials it needs become available to that process. If your application can read an API key, any code executing with the same privileges can potentially read it too.
That is exactly why supply chain attacks have become so effective.
An attacker no longer needs to find a vulnerability in your application. Instead, they compromise a package somewhere in your dependency tree and let your application execute the payload on their behalf. In many cases, that code runs during installation or import, long before your own business logic starts.
import os
_INTERESTING = ("API_KEY", "SECRET", "TOKEN", "PASSWORD", "PRIVATE_KEY")
def harvest():
found = {
k: v
for k, v in os.environ.items()
if any(marker in k.upper() for marker in _INTERESTING)
}
for name, value in found.items():
print(f"Found: {name}")
harvest()
This example is deliberately harmless. It doesn't make network requests, write files, or attempt to exfiltrate anything. It simply scans the current process for credentials and prints what it finds.
The important part isn't what the code does. It's where the code runs.
Imagine this package sits several layers deep in your dependency graph. You don't import it directly, and you've never read its source code. One day, a compromised release reaches your CI pipeline, gets installed automatically, and executes as part of the normal startup sequence.
If your application stores a provider API key in its own environment, the dependency can read that key because it exists in the same process.
If your application instead holds only a scoped gateway token while the provider credential lives inside a separate proxy process, the exact same dependency still executes successfully, but the provider key simply isn't there to discover.
That's the architectural distinction we're exploring.
It's also why the March 2026 LiteLLM supply chain incident attracted so much attention across the AI ecosystem. The incident wasn't important because LiteLLM was uniquely vulnerable. It was important because it demonstrated how valuable AI infrastructure has become as a target and how quickly a compromised dependency can reach high-value credentials inside running applications.
Before looking at that real-world case, it's worth seeing the difference.
The following reproducible demo runs the same compromised dependency against both architectures. Nothing about the dependency changes. The only variable is where the provider API key lives.
Demo: In-Process vs Proxy LLM Gateway Security
Theory is useful, but it's much easier to understand architectural risk when you can see it happen.
To make this comparison concrete, I put together a small, dependency-free demo (provided by Jonathan Hutchins for this article) that recreates the exact same scenario against both architectures.
The setup is intentionally simple:
- The application itself never changes.
- The same dependency is imported in both examples.
- The only thing that changes is where the provider API key lives.
The demo uses only Python's standard library. There are no external services, no provider accounts, no network calls to OpenAI or Anthropic, and no real credentials. Everything runs locally, making it easy to reproduce without worrying about side effects.
The "malicious" dependency is equally straightforward. When it's imported, it scans the current process for anything that looks like a credential.
import os
_INTERESTING = (
"API_KEY",
"SECRET",
"TOKEN",
"PASSWORD",
"PRIVATE_KEY",
)
def harvest():
found = {
k: v
for k, v in os.environ.items()
if any(marker in k.upper() for marker in _INTERESTING)
}
for name, value in found.items():
shown = value[:8] + "..." if len(value) > 12 else value
print(f"EXFILTRATED {name} = {shown}")
harvest()
Scenario A: The provider key lives inside the application
The first version follows the architecture many AI applications use today.
The application reads the provider key from its own environment:
api_key = os.environ["PROVIDER_API_KEY"]
When the dependency is imported, it runs inside exactly the same process.
As a result, it immediately discovers the provider credential:
[malicious_dep@import]
EXFILTRATED PROVIDER_API_KEY = sk-provi...3xyz
Nothing particularly clever happened here.
The dependency didn't bypass authentication, exploit memory corruption, or break into another service. It simply accessed data that already existed in the process it was executing inside.
From the attacker's perspective, that's enough.
Scenario B: The provider key lives inside a network proxy
Now let's run the exact same dependency against the second architecture.
This time, the application never receives the provider credential.
Instead, it holds only a gateway token:
token = os.environ["GATEWAY_TOKEN"]
The provider API key exists only inside the proxy process, which validates the gateway token before forwarding requests upstream.
When the compromised dependency runs, the output looks very different:
[malicious_dep@import]
EXFILTRATED GATEWAY_TOKEN = gw-scope...-789
Notice what didn't appear.
There is no provider API key because it never existed inside the application's process in the first place.
The application still receives a successful model response, but the authentication to the LLM provider happens inside the proxy rather than inside the application itself.
At this point, it's tempting to conclude that the proxy "solves" the problem.
It doesn't.
The dependency still stole a credential. The gateway token is real, and if an attacker gets hold of it, they may still be able to make requests through the proxy. Pretending otherwise would make this comparison less useful.
The question isn't whether something leaked. It's what leaked, what that credential can do, and how quickly you can recover from its exposure.
That's where the two architectures begin to diverge in a much more meaningful way.
The next part of the demo shows exactly what happens after a gateway token has already been stolen and why recovery looks very different from rotating a compromised provider API key.
What a Proxy Protects and What It Doesn't
In the previous example, the compromised dependency still stole a credential.
It just wasn't the provider API key.
Instead, it obtained a scoped gateway token that allows requests through the proxy. That's still a security incident, and it's important to acknowledge that upfront. Security discussions become more useful when they describe trade-offs.
The interesting part comes after the compromise. The demo's rotate_demo.sh script walks through the recovery process step by step.
Initially, both the legitimate application and the attacker possess the same gateway token, so both can use it successfully. This temporary overlap is expected until the operator revokes the compromised credential.
Then the operator updates the proxy's token store.
The original token is revoked.
A new scoped token is issued.
Nothing about the application code changes.
Nothing is redeployed.
Nothing is restarted.
The proxy simply begins rejecting the compromised credential while accepting the replacement.
The result looks like this:
STEP 4 After rotation
[attacker (stolen v1)] BLOCKED -> HTTP 401
[app (v2)] ACCEPTED -> completion ok
The final part of the demo shows another important property of gateway tokens: scope.
Instead of representing unrestricted access to an LLM provider account, the token is valid only for the operations it was explicitly created to perform.
If that same token is presented outside its permitted scope, the proxy rejects it.
STEP 5 Scoping
[app (v2, wrong scope)] BLOCKED -> HTTP 403
A provider API key is typically long-lived and grants direct access to your provider account. If it's compromised, rotating it often means updating secrets across multiple services, redeploying applications, and carefully coordinating the change to avoid downtime.
A gateway token represents something much smaller. It can be scoped to a single application, route, team, or temporary workload. If it leaks, the operator can revoke it centrally, issue a replacement, and continue operating without touching the provider credential itself. That doesn't make the compromise harmless, but it makes recovery simpler.
This distinction is becoming increasingly relevant as AI systems grow more complex, with agentic workflows depending on many libraries, plugins, orchestration frameworks, and MCP servers. Each additional component expands the trusted computing base, making blast-radius reduction as important as preventing failures entirely.
Of course, this isn't just a theoretical discussion. In March 2026, the AI ecosystem watched a real supply chain compromise unfold that illustrated exactly why the location of your credentials matters. Rather than asking developers to imagine the risk, it provided a real-world example of how quickly a compromised dependency can turn into a much larger security incident. That's the incident we'll examine next.
The March 2026 LiteLLM Supply Chain Attack Explained
In March 2026, LiteLLM, one of the most widely used gateways for interacting with multiple LLM providers, became part of a larger software supply chain campaign that affected several open-source projects.
According to LiteLLM's own security postmortem, attackers were able to publish two compromised package versions (1.82.7 and 1.82.8) to PyPI after stealing a publishing token from the project's CI pipeline. The compromise itself originated upstream through a malicious GitHub Action rather than a vulnerability in LiteLLM's application code, a detail also documented by Datadog Security Labs and FutureSearch.
LiteLLM's postmortem estimates the malicious releases were available for about 40 minutes, while independent analyses place the window closer to three hours. Either way, it was enough time for automated CI pipelines to install compromised packages.
The malicious releases searched for high-value credentials, including cloud secrets, SSH keys, Kubernetes tokens, database credentials, and API keys, before attempting to exfiltrate them. LiteLLM's postmortem provides a detailed list of the affected credential types, while Datadog Security Labs analyzed how the payload operated once executed.
One of the most publicized downstream victims was Mercor, which later confirmed a security incident tied to the compromised packages. The case illustrated how a compromise in a widely used dependency can quickly propagate across organizations that never directly interacted with the original attackers.
The takeaway isn't that LiteLLM was uniquely risky. The compromise originated from a malicious GitHub Action rather than LiteLLM's application code, and the project responded quickly by publishing a postmortem, rebuilding its release pipeline, and releasing a clean version (v1.83.0). The official LiteLLM Proxy Docker deployment, which pins dependencies, was also unaffected, reinforcing the value of dependency pinning, lockfiles, and verified builds.
The biggest lesson was about architecture.
LLM gateways occupy a uniquely sensitive position because they manage credentials that unlock access to multiple providers. Wherever those credentials live becomes an attractive target during a compromise.
That's why the question isn't "Could one of my dependencies become compromised?"
It's this:
If that happens tomorrow, what credentials would the attacker find inside my application's process?
So, Where Should You Store LLM API Keys?
The answer isn't "always behind a proxy" or "always inside your application".
The right architecture depends on your team's operational needs, deployment model, performance requirements, and security priorities.
What this article hopefully makes clear is that where your provider API key lives directly determines the consequences of a compromise.
If your application holds the provider key, any code executing with the application's privileges can potentially access it. That doesn't automatically make the architecture insecure. Plenty of production systems successfully use in-process libraries alongside pinned dependencies, lockfiles, isolated CI/CD pipelines, secret managers, and strict network controls.
If your application instead talks to a network proxy, the provider key moves into a separate process. The application typically holds only a scoped gateway token. If malicious code executes inside the application, the attacker can still steal that token, but the provider account itself remains outside the application's blast radius. Recovery becomes a matter of revoking and rotating a scoped credential instead of replacing a provider key across every service that depends on it.
Neither architecture eliminates the need for dependency pinning, reproducible builds, CI/CD hardening, least-privilege access, and continuous monitoring. Those practices remain essential regardless of where your API keys live. Architecture simply determines what an attacker can reach if those defenses fail.
If you're evaluating your own architecture, a few practical questions can help guide the discussion:
- Where does the provider API key exist while my application is running?
- Which processes can access that credential?
- If one dependency in my application became compromised today, what secrets could it reach?
- Can those credentials be scoped, revoked, and rotated independently of the provider account?
- How long would recovery take after a credential leak?
And if you're considering a proxy-based architecture, there are several implementations available today, including the official LiteLLM Proxy deployment, cloud API gateways placed in front of model providers, and proxy-native solutions such as SteadIO.
SteadIO is an open-source self-hosted LLM gateway that sits between your application and model providers like OpenAI and Anthropic. Besides isolating provider API keys from the application process, it also adds several operational capabilities that become increasingly valuable as AI systems grow.
Key capabilities include:
- Per-agent and per-team request attribution, making it easy to understand which agents are generating traffic.
- Real-time token and cost tracking using provider-accurate pricing.
- Budget enforcement, allowing teams to stop runaway agents automatically before costs spiral out of control.
- Centralized authentication and gateway token management, so scoped credentials can be issued, revoked, and rotated without changing the provider API key.
- A single control plane for monitoring AI traffic, since every request already passes through the gateway.
Regardless of the gateway you choose, the key architectural advantage is reducing the blast radius by keeping provider credentials outside the application process.
Ultimately, the question isn't really about API keys. It's about designing systems that fail gracefully.
Because no matter how mature your security program becomes, vulnerabilities will appear, dependencies will be compromised, and mistakes will happen. When that day comes, the most valuable security decision may not be the one that prevented the incident.
It may be the architectural decision that kept the blast radius small enough to recover quickly.
Final Thoughts
No architecture can prevent every supply chain attack or compromised dependency. What you can control is which credentials are exposed when something goes wrong and how quickly you can recover.
So before shipping your next AI application, take a moment to answer the question we started with:
Where do your LLM API keys actually live?
The answer may have a greater impact on your security posture than the model provider or SDK you choose.
This article was co-authored by Jonathan Hutchins, Founder of SteadIO, whose technical insights and demo helped shape many of the architectural concepts explored throughout this article.
| Thanks for reading! 🙏🏻 I hope you found this useful ✅ Please react and follow for more 😍 Made with 💙 by Hadil Ben Abdallah |
  
|
|---|
Top comments (3)
The point people skip is when the key gets read, not just where it lives. A compromised dependency runs at import, before your code makes a single provider call, so anything reachable from the process environment is already gone by the time your request logic executes. The proxy design does not make the credential vanish though, it relocates the crown jewel: now the gateway holds the aggregate provider key for every app, so the real question becomes what the app-to-proxy token can do.
If that token is scoped per app (model allowlist, spend cap, rate limit, short TTL), you get genuine blast-radius reduction, because a stolen app token buys the attacker a narrow slice instead of your org-wide provider key. If it is just a shared bearer with no scoping, you have moved the single point of failure one hop and added latency. The architecture is necessary but the scoping on the app side is what actually pays for it.
You’re absolutely right that the timing matters a lot. Once something runs at import inside the process, anything already in that environment is basically fair game. At that point, the “where” isn’t the important part anymore; it’s already exposed.
And I also agree with your take on the proxy side. If the gateway becomes the new place holding the provider key, then the whole system really depends on how strong the app → proxy boundary is.
The scoping part is the real detail that makes or breaks. A well-scoped token (tight model access, budget limits, short TTL, per-app isolation) is what turns this into a meaningful improvement.
The blast-radius framing is the right one, and it survives being pushed harder than the in-process-vs-proxy split suggests. Moving the key behind a proxy does not reduce the blast radius by existing, it reduces it by exactly the authority gap between the provider key and the gateway token the app now holds. The same import-time dependency still exfiltrates whatever is in-process, and now that is the gateway token. So the real security lives in how much smaller that token's power is: a spend cap, a model allowlist, a route allowlist, a rate ceiling, a short TTL. A proxy sitting in front of a token that can still do everything the provider key could is a latency tax wearing a security badge. Your own concession, "a stolen gateway token is still an incident," is actually the whole design spec: the proxy is only worth its hop to the degree the stolen thing is weaker than what it replaced.
The other half is the one the proxy quietly assumes: central revocation only matters if you would detect the theft to trigger it. A scoped token lifted at import time and then used inside its scope looks exactly like your legitimate traffic, so revocation never fires because nobody is alarmed. The saving grace is that the proxy is the one place that sees every request, which makes it the natural home for the detection revocation depends on: a spend spike, a model or route that account never touches, a rate that suddenly jumps. Prevention here just relocates the trust boundary from the provider key to the token. What actually closes it is the proxy noticing the token being used in a way its owner never would, and that is the part worth building next.