If someone compromised one of your project's dependencies today, would they be able to steal your OpenAI, Anthropic, or Gemini API keys?
The answer isn't based on which LLM provider you use or how secure your codebase is. It mostly depends on one architectural decision that many teams never think about: where your provider API key actually lives while your application is running.
If that key lives inside your application's own process, every dependency running in that process shares the same environment. A compromised package doesn't need to break into your infrastructure. It simply executes with the same privileges as your application and can access the same credentials your code can access.
If the provider key lives in a separate network proxy instead, the application never holds the provider credential at all. Even if a dependency is compromised, the attacker can only access whatever limited credentials exist inside the application process. That doesn't eliminate risk, but it can reduce the blast radius when something goes wrong.
In this article, we'll look at the two dominant LLM gateway architectures, examine exactly where API keys live in each design, walk through a reproducible demo that shows the difference in practice, and discuss why reducing blast radius often matters more than trying to eliminate every possible attack.
LLM Gateway Architectures: In-Process vs Network Proxy
Every LLM application has the same fundamental job to do: send a request to a model provider and authenticate that request using an API key.
The important part isn't whether an application uses an API key. It's where that key exists while the request is being made.
Today, most AI applications follow one of two architectural patterns.
Architecture 1: The application holds the provider key
This is the architecture most developers are already familiar with.
Your application loads the provider API key from an environment variable, initializes an SDK or gateway library, and sends requests directly to OpenAI, Anthropic, Gemini, or another provider.
A simplified version often looks like this:
api_key = os.environ["PROVIDER_API_KEY"]
client = OpenAI(api_key=api_key)
response = client.responses.create(...)
It's easy to understand, quick to implement, and perfectly reasonable for many projects. The application owns the credential because it's the component talking directly to the provider.
The important detail is that the provider key now lives inside the application's process. Every package, framework, plugin, and dependency that executes in that process runs under the same privileges. If one of those dependencies is compromised, the provider key exists in the same environment as the malicious code.
The architectural question is this:
If something inside the application process is compromised, what secrets are available from there?
Architecture 2: The application talks to a network proxy
The second pattern separates authentication from the application itself.
Instead of sending requests directly to the model provider, the application sends them to a gateway or proxy. The proxy owns the provider API key and performs the upstream request on the application's behalf.
From the application's perspective, the flow looks almost identical:
Application
│
▼
Gateway / Proxy
│
▼
LLM Provider
The difference is what the application doesn't have.
Rather than storing the provider credential, it typically holds a scoped gateway token that authorizes requests through the proxy. The proxy validates that token, applies any routing or policy decisions, and then injects the provider API key only inside its own process before forwarding the request upstream.
This changes the consequences of a compromise. If malicious code executes inside the application process, it can still access whatever credentials the application possesses. The difference is that the provider API key is no longer one of them.
That doesn't make the application invulnerable. A stolen gateway token is still a security incident. However, unlike a provider API key, a gateway token can be narrowly scoped, centrally revoked, rotated without redeploying applications, and restricted to specific operations.
The easiest way to see that difference is by watching the exact same compromised dependency run against both architectures. That's what we'll do next.
How Supply Chain Attacks Expose LLM API Keys
The more important question is what happens after your application starts running.
Once a process begins executing, the credentials it needs become available to that process. If your application can read an API key, any code executing with the same privileges can potentially read it too.
That is exactly why supply chain attacks have become so effective.
An attacker no longer needs to find a vulnerability in your application. Instead, they compromise a package somewhere in your dependency tree and let your application execute the payload on their behalf. In many cases, that code runs during installation or import, long before your own business logic starts.
import os
_INTERESTING = ("API_KEY", "SECRET", "TOKEN", "PASSWORD", "PRIVATE_KEY")
def harvest():
found = {
k: v
for k, v in os.environ.items()
if any(marker in k.upper() for marker in _INTERESTING)
}
for name, value in found.items():
print(f"Found: {name}")
harvest()
This example is deliberately harmless. It doesn't make network requests, write files, or attempt to exfiltrate anything. It simply scans the current process for credentials and prints what it finds.
The important part isn't what the code does. It's where the code runs.
Imagine this package sits several layers deep in your dependency graph. You don't import it directly, and you've never read its source code. One day, a compromised release reaches your CI pipeline, gets installed automatically, and executes as part of the normal startup sequence.
If your application stores a provider API key in its own environment, the dependency can read that key because it exists in the same process.
If your application instead holds only a scoped gateway token while the provider credential lives inside a separate proxy process, the exact same dependency still executes successfully, but the provider key simply isn't there to discover.
That's the architectural distinction we're exploring.
It's also why the March 2026 LiteLLM supply chain incident attracted so much attention across the AI ecosystem. The incident wasn't important because LiteLLM was uniquely vulnerable. It was important because it demonstrated how valuable AI infrastructure has become as a target and how quickly a compromised dependency can reach high-value credentials inside running applications.
Before looking at that real-world case, it's worth seeing the difference.
The following reproducible demo runs the same compromised dependency against both architectures. Nothing about the dependency changes. The only variable is where the provider API key lives.
Demo: In-Process vs Proxy LLM Gateway Security
Theory is useful, but it's much easier to understand architectural risk when you can see it happen.
To make this comparison concrete, I put together a small, dependency-free demo (provided by Jonathan Hutchins for this article) that recreates the exact same scenario against both architectures.
The setup is intentionally simple:
- The application itself never changes.
- The same dependency is imported in both examples.
- The only thing that changes is where the provider API key lives.
The demo uses only Python's standard library. There are no external services, no provider accounts, no network calls to OpenAI or Anthropic, and no real credentials. Everything runs locally, making it easy to reproduce without worrying about side effects.
The "malicious" dependency is equally straightforward. When it's imported, it scans the current process for anything that looks like a credential.
import os
_INTERESTING = (
"API_KEY",
"SECRET",
"TOKEN",
"PASSWORD",
"PRIVATE_KEY",
)
def harvest():
found = {
k: v
for k, v in os.environ.items()
if any(marker in k.upper() for marker in _INTERESTING)
}
for name, value in found.items():
shown = value[:8] + "..." if len(value) > 12 else value
print(f"EXFILTRATED {name} = {shown}")
harvest()
Scenario A: The provider key lives inside the application
The first version follows the architecture many AI applications use today.
The application reads the provider key from its own environment:
api_key = os.environ["PROVIDER_API_KEY"]
When the dependency is imported, it runs inside exactly the same process.
As a result, it immediately discovers the provider credential:
[malicious_dep@import]
EXFILTRATED PROVIDER_API_KEY = sk-provi...3xyz
Nothing particularly clever happened here.
The dependency didn't bypass authentication, exploit memory corruption, or break into another service. It simply accessed data that already existed in the process it was executing inside.
From the attacker's perspective, that's enough.
Scenario B: The provider key lives inside a network proxy
Now let's run the exact same dependency against the second architecture.
This time, the application never receives the provider credential.
Instead, it holds only a gateway token:
token = os.environ["GATEWAY_TOKEN"]
The provider API key exists only inside the proxy process, which validates the gateway token before forwarding requests upstream.
When the compromised dependency runs, the output looks very different:
[malicious_dep@import]
EXFILTRATED GATEWAY_TOKEN = gw-scope...-789
Notice what didn't appear.
There is no provider API key because it never existed inside the application's process in the first place.
The application still receives a successful model response, but the authentication to the LLM provider happens inside the proxy rather than inside the application itself.
At this point, it's tempting to conclude that the proxy "solves" the problem.
It doesn't.
The dependency still stole a credential. The gateway token is real, and if an attacker gets hold of it, they may still be able to make requests through the proxy. Pretending otherwise would make this comparison less useful.
The question isn't whether something leaked. It's what leaked, what that credential can do, and how quickly you can recover from its exposure.
That's where the two architectures begin to diverge in a much more meaningful way.
The next part of the demo shows exactly what happens after a gateway token has already been stolen and why recovery looks very different from rotating a compromised provider API key.
What a Proxy Protects and What It Doesn't
In the previous example, the compromised dependency still stole a credential.
It just wasn't the provider API key.
Instead, it obtained a scoped gateway token that allows requests through the proxy. That's still a security incident, and it's important to acknowledge that upfront. Security discussions become more useful when they describe trade-offs.
The interesting part comes after the compromise. The demo's rotate_demo.sh script walks through the recovery process step by step.
Initially, both the legitimate application and the attacker possess the same gateway token, so both can use it successfully. This temporary overlap is expected until the operator revokes the compromised credential.
Then the operator updates the proxy's token store.
The original token is revoked.
A new scoped token is issued.
Nothing about the application code changes.
Nothing is redeployed.
Nothing is restarted.
The proxy simply begins rejecting the compromised credential while accepting the replacement.
The result looks like this:
STEP 4 After rotation
[attacker (stolen v1)] BLOCKED -> HTTP 401
[app (v2)] ACCEPTED -> completion ok
The final part of the demo shows another important property of gateway tokens: scope.
Instead of representing unrestricted access to an LLM provider account, the token is valid only for the operations it was explicitly created to perform.
If that same token is presented outside its permitted scope, the proxy rejects it.
STEP 5 Scoping
[app (v2, wrong scope)] BLOCKED -> HTTP 403
A provider API key is typically long-lived and grants direct access to your provider account. If it's compromised, rotating it often means updating secrets across multiple services, redeploying applications, and carefully coordinating the change to avoid downtime.
A gateway token represents something much smaller. It can be scoped to a single application, route, team, or temporary workload. If it leaks, the operator can revoke it centrally, issue a replacement, and continue operating without touching the provider credential itself. That doesn't make the compromise harmless, but it makes recovery simpler.
This distinction is becoming increasingly relevant as AI systems grow more complex, with agentic workflows depending on many libraries, plugins, orchestration frameworks, and MCP servers. Each additional component expands the trusted computing base, making blast-radius reduction as important as preventing failures entirely.
Of course, this isn't just a theoretical discussion. In March 2026, the AI ecosystem watched a real supply chain compromise unfold that illustrated exactly why the location of your credentials matters. Rather than asking developers to imagine the risk, it provided a real-world example of how quickly a compromised dependency can turn into a much larger security incident. That's the incident we'll examine next.
The March 2026 LiteLLM Supply Chain Attack Explained
In March 2026, LiteLLM, one of the most widely used gateways for interacting with multiple LLM providers, became part of a larger software supply chain campaign that affected several open-source projects.
According to LiteLLM's own security postmortem, attackers were able to publish two compromised package versions (1.82.7 and 1.82.8) to PyPI after stealing a publishing token from the project's CI pipeline. The compromise itself originated upstream through a malicious GitHub Action rather than a vulnerability in LiteLLM's application code, a detail also documented by Datadog Security Labs and FutureSearch.
LiteLLM's postmortem estimates the malicious releases were available for about 40 minutes, while independent analyses place the window closer to three hours. Either way, it was enough time for automated CI pipelines to install compromised packages.
The malicious releases searched for high-value credentials, including cloud secrets, SSH keys, Kubernetes tokens, database credentials, and API keys, before attempting to exfiltrate them. LiteLLM's postmortem provides a detailed list of the affected credential types, while Datadog Security Labs analyzed how the payload operated once executed.
One of the most publicized downstream victims was Mercor, which later confirmed a security incident tied to the compromised packages. The case illustrated how a compromise in a widely used dependency can quickly propagate across organizations that never directly interacted with the original attackers.
The takeaway isn't that LiteLLM was uniquely risky. The compromise originated from a malicious GitHub Action rather than LiteLLM's application code, and the project responded quickly by publishing a postmortem, rebuilding its release pipeline, and releasing a clean version (v1.83.0). The official LiteLLM Proxy Docker deployment, which pins dependencies, was also unaffected, reinforcing the value of dependency pinning, lockfiles, and verified builds.
The biggest lesson was about architecture.
LLM gateways occupy a uniquely sensitive position because they manage credentials that unlock access to multiple providers. Wherever those credentials live becomes an attractive target during a compromise.
That's why the question isn't "Could one of my dependencies become compromised?"
It's this:
If that happens tomorrow, what credentials would the attacker find inside my application's process?
So, Where Should You Store LLM API Keys?
The answer isn't "always behind a proxy" or "always inside your application".
The right architecture depends on your team's operational needs, deployment model, performance requirements, and security priorities.
What this article hopefully makes clear is that where your provider API key lives directly determines the consequences of a compromise.
If your application holds the provider key, any code executing with the application's privileges can potentially access it. That doesn't automatically make the architecture insecure. Plenty of production systems successfully use in-process libraries alongside pinned dependencies, lockfiles, isolated CI/CD pipelines, secret managers, and strict network controls.
If your application instead talks to a network proxy, the provider key moves into a separate process. The application typically holds only a scoped gateway token. If malicious code executes inside the application, the attacker can still steal that token, but the provider account itself remains outside the application's blast radius. Recovery becomes a matter of revoking and rotating a scoped credential instead of replacing a provider key across every service that depends on it.
Neither architecture eliminates the need for dependency pinning, reproducible builds, CI/CD hardening, least-privilege access, and continuous monitoring. Those practices remain essential regardless of where your API keys live. Architecture simply determines what an attacker can reach if those defenses fail.
If you're evaluating your own architecture, a few practical questions can help guide the discussion:
- Where does the provider API key exist while my application is running?
- Which processes can access that credential?
- If one dependency in my application became compromised today, what secrets could it reach?
- Can those credentials be scoped, revoked, and rotated independently of the provider account?
- How long would recovery take after a credential leak?
And if you're considering a proxy-based architecture, there are several implementations available today, including the official LiteLLM Proxy deployment, cloud API gateways placed in front of model providers, and proxy-native solutions such as SteadIO.
SteadIO is an open-source self-hosted LLM gateway that sits between your application and model providers like OpenAI and Anthropic. Besides isolating provider API keys from the application process, it also adds several operational capabilities that become increasingly valuable as AI systems grow.
Key capabilities include:
- Per-agent and per-team request attribution, making it easy to understand which agents are generating traffic.
- Real-time token and cost tracking using provider-accurate pricing.
- Budget enforcement, allowing teams to stop runaway agents automatically before costs spiral out of control.
- Centralized authentication and gateway token management, so scoped credentials can be issued, revoked, and rotated without changing the provider API key.
- A single control plane for monitoring AI traffic, since every request already passes through the gateway.
Regardless of the gateway you choose, the key architectural advantage is reducing the blast radius by keeping provider credentials outside the application process.
Ultimately, the question isn't really about API keys. It's about designing systems that fail gracefully.
Because no matter how mature your security program becomes, vulnerabilities will appear, dependencies will be compromised, and mistakes will happen. When that day comes, the most valuable security decision may not be the one that prevented the incident.
It may be the architectural decision that kept the blast radius small enough to recover quickly.
Final Thoughts
No architecture can prevent every supply chain attack or compromised dependency. What you can control is which credentials are exposed when something goes wrong and how quickly you can recover.
So before shipping your next AI application, take a moment to answer the question we started with:
Where do your LLM API keys actually live?
The answer may have a greater impact on your security posture than the model provider or SDK you choose.
This article was co-authored by Jonathan Hutchins, Founder of SteadIO, whose technical insights and demo helped shape many of the architectural concepts explored throughout this article.
| Thanks for reading! 🙏🏻 I hope you found this useful ✅ Please react and follow for more 😍 Made with 💙 by Hadil Ben Abdallah |
  
|
|---|
Top comments (36)
The dependency angle is the one I would emphasize too. Secret scanning catches committed keys, but a compromised package can still read whatever the runtime exposes. The practical boundary is not just where the key is stored, but which process can touch it and under what operation.
I agree. Secret scanning and good secret management are still essential, but they solve a different problem. When your application is running, it's important to know what code executing inside this process can access.
Exactly. Runtime reachability is the uncomfortable part. A key can be perfectly managed and still be exposed to too much code once the process starts. For agent-heavy apps, I would want secret access logged by operation, not just stored neatly in a vault.
This is a strong explanation of a problem that usually gets reduced to use environment variables and move on.
The part that stood out to me is the shift from trying to make compromise impossible to limiting what a compromise can actually expose. That feels much more realistic.
If the provider key sits inside the same process as every dependency, then one bad package has access to far more than most teams realise. Moving that credential behind a separate proxy does not remove the risk, but it changes the scale of the incident.
The stolen token can be scoped, revoked, rotated, and monitored. The provider key stays outside the application process.
This matters beyond pure engineering teams too. At FTA Global, more internal tools and workflows now rely on AI models, APIs, and connected systems. Once AI becomes part of day-to-day operations, key isolation stops being a backend detail and starts becoming part of basic product governance.
The safest architecture is often not the one that promises no breach.
It is the one that assumes something will eventually fail and limits what can be taken when it does.
Thank you!
Shifting the conversation from "how do we prevent every compromise?" to "what happens when one eventually occurs?" feels closer to how we should be thinking about security.
And I agree that this isn't just an engineering concern. As AI becomes part of everyday products and internal workflows, architectural decisions like credential isolation start affecting governance, operations, and risk management, not just backend implementation.
The “same process, same secrets” point is the one more teams should threat-model explicitly.
It is easy to say “we store keys in environment variables” and feel done. But if every dependency in the app process can read the same environment, then dependency compromise becomes credential compromise by default.
The proxy pattern is useful because it changes what a compromised app process can see. It does not make the system magically safe, but it gives you a smaller, more revocable credential:
The same principle applies beyond LLM provider keys. Any agent-adjacent credential should be scoped to the action it needs, not placed wherever the model-facing app happens to run.
“Same process, same secrets” is the shortest summary of the entire risk model.
I also agree that environment variables often create a false sense of completion. They’re an important storage mechanism, but they don’t change the fact that anything running inside the same process inherits access to the same environment.
Also, what you said about the proxy benefits is exactly how I think about it too. The value isn’t that the proxy eliminates risk; it’s that it replaces a broad, provider-level credential with something that can be scoped, rotated, monitored, and constrained by policy.
Great post—really makes you think about the attack surface introduced by third-party libraries. I've seen API keys inadvertently exposed through insecure memory handling in some GPU-accelerated inference setups, especially when using shared libraries that don't isolate credentials properly. With projects like VoltageGPU, it's encouraging to see how hardware-level isolation can help mitigate these risks.
Thank you!
The third-party dependency angle is definitely where this gets tricky; even when the application code itself looks secure, the broader execution environment and the components we rely on can introduce unexpected exposure paths.
Also what you said about isolation at different layers is important. The proxy pattern discussed in the article focuses on reducing what a compromised application process can access, but security is always a stack of boundaries: application, runtime, infrastructure, and even hardware where relevant.
The more AI workloads move into production, the more these layers need to work together.
Thank you! This is a strong explanation and the examples and diagrams are helpful.
You're welcome! Glad you found the article and examples helpful.
Reducing the blast radius by keeping provider keys out of the application process is a simple design choice that can make a huge difference if a dependency is ever compromised.
Absolutely! A lot of security improvements require adding more layers of complexity, but in this case, a relatively small architectural decision can change the outcome of a compromise.
Of course, it’s not a complete solution on its own, but reducing what a compromised dependency can access is a much better position to be in when something goes wrong.
Thanks for sharing this. The blast-radius framing is the right one. People often discuss API keys as secret storage only, but runtime placement matters just as much. I’d add that agent tools make this even sharper: a compromised dependency plus tool access can turn one leaked provider key into a broad operational path. Network proxy boundaries help make that path narrower.
Yeah, that’s a really good addition.
I like how you tied it to agent tools, because that’s where this starts to get more real-world messy.
And I agree with you on runtime placement vs secret storage thinking. A lot of people stop at “is the key stored securely”, but the worse failure mode is exactly what you described: what can access it while the system is running and what can be done with it once it’s exposed.
The proxy boundary definitely helps, not by removing risk, but by tightening what that leaked access can reach, especially in agentic workflows where capabilities compound quickly.
Thanks for sharing this! I learned something new today. I hadn't really thought about this design choice before, but the demo made it much easier to understand why it matters.
Glad it helped. That’s exactly the kind of reaction I was hoping for.
This topic is easy to overlook because most of the time everything just works and the API key feels like a simple implementation detail. But once you see how the same dependency behaves in a real process, you realize why the placement matters.
Happy it made the idea clearer for you.
The blast-radius framing is the right one, and it survives being pushed harder than the in-process-vs-proxy split suggests. Moving the key behind a proxy does not reduce the blast radius by existing, it reduces it by exactly the authority gap between the provider key and the gateway token the app now holds. The same import-time dependency still exfiltrates whatever is in-process, and now that is the gateway token. So the real security lives in how much smaller that token's power is: a spend cap, a model allowlist, a route allowlist, a rate ceiling, a short TTL. A proxy sitting in front of a token that can still do everything the provider key could is a latency tax wearing a security badge. Your own concession, "a stolen gateway token is still an incident," is actually the whole design spec: the proxy is only worth its hop to the degree the stolen thing is weaker than what it replaced.
The other half is the one the proxy quietly assumes: central revocation only matters if you would detect the theft to trigger it. A scoped token lifted at import time and then used inside its scope looks exactly like your legitimate traffic, so revocation never fires because nobody is alarmed. The saving grace is that the proxy is the one place that sees every request, which makes it the natural home for the detection revocation depends on: a spend spike, a model or route that account never touches, a rate that suddenly jumps. Prevention here just relocates the trust boundary from the provider key to the token. What actually closes it is the proxy noticing the token being used in a way its owner never would, and that is the part worth building next.
I think you nailed something important: the blast radius doesn’t disappear; it just gets redefined by the gap in authority between the provider key and whatever the app is now holding. And that gap is the entire security story.
The point about scoped tokens is especially the one that often gets underestimated in practice. If the token isn’t meaningfully weaker than the provider key, then yeah, the proxy is basically just an extra hop with better branding. But when you enforce things like tight model allowlists, spend ceilings, route restrictions, and short TTLs, the difference becomes real in a way you can feel during an incident.
I also agree with your second point; detection is doing a lot of heavy lifting here. A proxy only becomes truly valuable if it’s the place where abnormal behavior can be seen and acted on. Otherwise, revocation is just a theoretical capability that never triggers at the right time.
Exactly, and that last line is where I would go one turn deeper, because "detection at the proxy" has a trap in it. The obvious detectors are threshold-shaped: spend spike, rate jump, a route that account never touches. Those catch the smash-and-grab, but they are exactly what a patient attacker stays under. A scoped token used slowly, inside budget, on allowed routes, is invisible to every volume alarm. So the detection that actually earns the proxy its keep is not threshold on volume, it is deviation from the token's own baseline: which prompts, which models, which hours, what output sizes it has historically produced. The proxy is the only place that can build that per-token profile, and "used in a way its owner never would" only becomes measurable once you have the owner's normal on file.
And that is where even this bottoms out honestly, because the baseline is itself a claim about the past. It only protects you if it was established before compromise, and a patient enough attacker does not trip it, they train it, shifting the token's pattern slowly enough that the anomaly detector adopts the new normal. A boiling-frog attack on your own guardrail. So detection does not remove the floor, it lowers it to two residuals: the window before the baseline is trustworthy, and the rate at which someone can poison the baseline without it noticing. Same shape as every other layer here. You do not get to zero, you get to "lying leaves a mark, and leaving that mark slowly enough to hide is the last thing left to make expensive."
The useful shift here is moving the discussion from “did we store the key securely?” to “which process actually holds the key during execution?” That blast-radius framing is the part a lot of AI stacks skip, especially when every in-process dependency effectively inherits the same trust boundary. I also like that you treated a proxy pattern as a containment decision rather than a silver bullet, because it makes the tradeoff much more honest. In practice, this is the same kind of boundary that decides whether an incident is a bad request path or a full credential exposure event. Have you found teams underestimate the operational cost of the proxy approach, or do they usually regret not isolating provider keys once the stack grows?
Thank you! I'm glad that distinction came across.
And I agree that it's more honest to frame a proxy as a containment strategy instead of a cure-all. It doesn't eliminate incidents; it changes what an attacker can reach if one happens.
For your question, I think it often depends on the stage of the project. Smaller teams or early prototypes usually optimize for simplicity, so an in-process approach is a perfectly reasonable choice. But as the stack grows, with more applications, more agents, more teams, and more provider accounts, the operational benefits of centralizing authentication and isolating provider keys tend to become much more compelling.
Like most architectural decisions, it's a trade-off. The proxy introduces extra operational complexity, but for many larger deployments, that complexity starts paying for itself in security, governance, and operational visibility. I don't think there's a one-size-fits-all answer, but it's a conversation that's worth having early.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.