The LLM Isn't Your Attacker. Your eval() Statement Is.

Everyone's worried about prompt injection making models say bad things. Meanwhile someone piped LLM output straight into eval() and gave it a GPU box to play with. That's not an AI safety problem. That's a 2005 problem wearing a 2025 costume.

Where this fits

CVE-2025-9141 (vLLM's tool-call parser calling eval() on model-generated arguments) is the concrete example here, but the pattern it represents is old news dressed up in new terminology. We've spent two decades telling developers "never eval untrusted input" for every other class of software. Then along comes an entirely new category of infrastructure, inference engines like vLLM and SGLang, built at breakneck speed by teams focused on throughput and latency benchmarks, and the same mistake shows up again. Not because LLMs are uniquely dangerous, but because the input they produce gets treated with a weird kind of unearned trust. Somewhere along the way, "the model's output" started getting parsed like structured, safe data instead of what it actually is: text from an untrusted source that happens to look like JSON or a function call.

This is the classic trusted-input-that-isn't-trusted mistake. SQL injection, XML external entity attacks, insecure deserialization, all the same shape. We just haven't finished mapping it onto the LLM stack yet.

Hype check

The framing "LLMs could control their host machines" is going to get read two ways, and both are a little off. The breathless version: models are becoming agentic and dangerous, skynet-adjacent. The dismissive version: this is just a parser bug, nothing to see here, move along.

Neither lands right. The overstated part is the implication that this requires a sophisticated, self-aware, scheming model. It doesn't. A model doesn't need intent to emit a token sequence that trips a bad parser, it just needs to be steered there, whether by a malicious prompt, a poisoned fine-tuning set, or honestly just an adversarial user probing the API. The "malicious LLM" framing makes for a better headline than "malicious input to a badly-designed parser," but the second one is more accurate and, frankly, scarier because it's more achievable.

The understated part: inference engines are exploding in feature surface right now. Tool calling, function calling, structured output modes, agentic loops calling back into the host, this is all extremely fresh code, written under competitive pressure, often by ML engineers who are excellent at kernels and batching and terrible at (or simply uninterested in) adversarial input handling. That's not a knock on them, it's just not the skillset the job usually selects for. Nobody hires a vLLM contributor for their threat-modeling chops.

Who benefits from the current narrative? Anyone who wants to sell "AI is an existential risk" gets a nice anecdote. Anyone who wants to dismiss AI security concerns entirely gets to say "see, it's just a code bug, nothing special." Both groups get to skip the boring middle position, which is: this is a supply chain and input-validation problem in fast-moving infrastructure, and it's going to keep happening until the tooling matures.

Implications

If you're running inference infrastructure, the actual lesson has nothing to do with model alignment or jailbreaking. It's the same lesson from every other software security era: never call eval(), exec(), pickle.loads(), or their cousins on anything that originated from outside your trust boundary, and model output is outside your trust boundary, full stop, even if you trained the model yourself. Structured output should be parsed with actual parsers that fail closed, not interpreted as code because it was convenient during a hackathon-speed feature build.

For security teams, this is a reminder that "AI security" isn't one discipline. There's model behavior (jailbreaks, alignment, hallucination), and there's the software engineering wrapped around the model (the serving stack, the tool-call parsers, the plugin architectures). The second category is just appsec. It needs the same code review rigor, the same fuzzing, the same "assume this input is hostile" mindset we apply to any API endpoint. The fact that the untrusted input comes from a neural network instead of a web form doesn't change the threat model, it just changes who's writing the parsing code and how fast they're shipping it.

Expect more CVEs shaped exactly like this one over the next year or two, not because models are getting more dangerous, but because inference engines are still in their "move fast, ship the demo" phase and haven't caught up to basic input-handling hygiene yet.

Open question

If model output is just untrusted user input wearing a trench coat, why does so much of the AI infra ecosystem still architect around the assumption that it isn't?

— Cor, Skyblue Soft

Sources

• LLMs could control their host machines by exploiting inference engines

---

AI-assisted draft or imaging, human-curated, reviewed and edited.

Comments

[Kane Lim]

This is a strong framing. The key distinction is that the LLM is not inherently a trusted execution boundary and treating generated output as trusted because it looks structured is exactly where the vulnerability begins.

What stands out to me is the confused-deputy aspect of these systems. Once an inference server turns model-generated text into tool arguments, shell commands, code, filesystem operations, or API calls, the security boundary moves from “what did the model generate?” to “what authority does the surrounding runtime grant that output?”

A robust architecture should assume:

Model output = untrusted data
Parsing ≠ validation
Validation ≠ authorization
Tool execution requires an explicit capability boundary
Deserialization/interpreters must never become an implicit execution path

For tool calling specifically, I’d go further than simply replacing eval(). Schema validation should be followed by semantic validation and least-privilege authorization. A syntactically valid argument can still be dangerous or outside the caller's intended scope.

I also think this creates an interesting AppSec opportunity: traditional SAST/DAST/fuzzing methodologies can be adapted to AI infrastructure by treating model outputs as hostile generated inputs and systematically testing parser/tool boundaries.

The industry will mature when “AI security” stops being treated exclusively as jailbreak prevention and starts being treated as end-to-end security engineering around probabilistic components.

Great write-up. The most important lesson is probably the least sensational one: don't give untrusted text more authority simply because a model produced it.

I’d be interested in comparing notes on secure AI/tooling architectures and long-term engineering collaboration.