Earlier this month, a critical vulnerability showed up in GitLab's AI Gateway — CVE-2026-90970, CVSS 9.9, letting a logged-in user with agent platform access run commands on the gateway itself. I read the advisory the way most people probably did: nodded, thought "glad that's not us," and moved on.
Then I actually thought about our own setup for about ten more seconds, and the "glad that's not us" feeling evaporated.
What we had built, and why it felt safe
We'd wired an internal AI agent into our infra for the usual reasons — it could read logs, query our internal APIs, open PRs, restart a flaky worker, that kind of thing. It talked to our systems through a small internal gateway service that translated the model's tool calls into real actions. The gateway wasn't exposed to the internet. No inbound port, internal DNS only, behind our VPN. That was the entire safety argument: it's not reachable from outside, so it's fine.
The audit, and the wrong assumption that almost ended it early
I went looking for whether our gateway had the same class of bug — unauthenticated or under-authenticated command execution reachable by an agent session. My first assumption was that the risk, if any, was purely about network exposure, the same shape as the GitLab issue. I checked our network path twice, confirmed there was no route in from outside our VPC, and almost closed the investigation there.
The second assumption nearly stopped me too: I assumed that because our tool-calling layer had a permissions.yaml file listing which tools each agent role could call, that file was actually being enforced at the point where commands executed. It read like enforcement. It had role names, tool names, an allowed: true/false column. Surely something was checking it.
Nothing was checking it at the layer that mattered.
Where the real gap was
The permission check lived in the orchestration layer — the code that decided which tools to offer the model in the first place. If a role wasn't supposed to have restart_service, that tool simply wasn't included in the list sent to the model for that session.
But the gateway's actual command-execution endpoint didn't re-check anything. It trusted that any request reaching it had already been filtered upstream. That's a completely reasonable design until something generates a tool call through a path that skips the orchestration layer — which, it turned out, our own retry logic did. When a tool call timed out, a retry handler resubmitted the call directly to the gateway's execution endpoint to save latency, bypassing the permission-aware orchestration path entirely, service-account credentials and all.
In other words: the enforcement looked like server-side authorization. It was actually client-side intent, enforced by a component that, by construction, obeyed the model's instructions rather than policing them. A compromised, confused, or just sufficiently creative agent session — not even malicious, just wrong — could, through the retry path, execute anything the gateway's own service account could do. That's a much bigger blast radius than "an AI helped restart a worker."
The fix
1. Move enforcement to the one place that can't be bypassed: the execution endpoint itself.
ALLOWED_COMMANDS = {
"restart_service": {"args": {"service_name": r"^[a-z0-9-]+$"}},
"read_logs": {"args": {"service_name": r"^[a-z0-9-]+$", "lines": r"^\d{1,4}$"}},
}
def execute_tool_call(role: str, tool: str, args: dict) -> str:
allowed_tools = ROLE_PERMISSIONS.get(role, set())
if tool not in allowed_tools or tool not in ALLOWED_COMMANDS:
raise PermissionError(f"role={role} is not permitted to call {tool}")
spec = ALLOWED_COMMANDS[tool]
for key, pattern in spec["args"].items():
if key not in args or not re.fullmatch(pattern, str(args[key])):
raise PermissionError(f"invalid or missing argument '{key}' for {tool}")
return run_tool(tool, args)
Every path into execution, including retries, now calls this function. There is no second door.
2. Stop giving any agent session a standing, shared, privileged credential at all. This was the bigger fix, and the one I actually care about. Even with enforcement fixed, a single service account used across every agent session for every user is a bad shape — one bug, one bypass, one clever prompt, and the blast radius is everything that account can reach.
What we moved to instead: every agent task gets its own short-lived, disposable sandbox — not a shared internal gateway at all. I'm the founder of Krova Cloud, and this is close to the exact problem we started the company to solve — spinning up an isolated VM per agent run, with its own locked-down outbound-only networking, no access to our real internal systems, root access inside its own box but nowhere else, and a lifespan measured in minutes. If an agent goes sideways inside one of these, the damage is contained to a VM that gets thrown away afterward — not a shared gateway with god-mode access to production. Running agent sessions this way, isolated per task instead of behind one shared privileged gateway, came out something like 93% cheaper than the equivalent on E2B and 95% cheaper than Modal in our own cost comparison, mostly because we're not paying for idle shared infrastructure sized for worst-case concurrency.
Lessons
- A permissions file that's checked once, upstream, and trusted forever downstream is not enforcement — it's a suggestion with good intentions.
- Any retry, fallback, or "fast path" that bypasses your normal request flow is a second door into your system. Audit those paths specifically; they're where enforcement quietly stops applying.
- "Not exposed to the internet" is not the same question as "what can this thing do once it's reachable at all, by anyone or anything." A CVE about one attack surface is a good prompt to check the attack surfaces you actually have, not just the one that made the news.
- The cheapest real fix for "what if the agent does something it shouldn't" isn't better prompting or a stricter permissions file — it's making sure the blast radius of "the agent did something it shouldn't" is a disposable sandbox, not your real infrastructure.
If you're running any kind of AI agent against real infrastructure right now, it's worth the ten minutes to ask what happens on your retry path, not just your happy path.
I'm Rohit, founder of Krova Cloud — disposable, isolated sandboxes for running AI agents and risky code without exposing real infrastructure, at a fraction of the cost of comparable sandbox platforms. If you want more deep debugging stories like this one, I write regularly over at debugly.dev too.
Top comments (0)