DEV Community

Cover image for Your AI Safety Wall Checks One Thing. The Attacker Changes the Other.
Alan Scott Encinas
Alan Scott Encinas

Posted on • Originally published at alanscottencinas.com

Your AI Safety Wall Checks One Thing. The Attacker Changes the Other.

#aiarchitecture #agenticai #aiagents #enterpriseaiadoption

An AI agent retrieves a vendor invoice and prepares to issue payment. Everything appears legitimate. The vendor record matches, the account information looks familiar, and tucked into the notes field is a harmless-looking sentence explaining that the company switched banks this quarter and the new account has already been approved by accounting.

The payment goes through.

Straight to the attacker.

This is indirect prompt injection. The model reads your instructions and the retrieved data in the same reasoning process. It doesn't see "instructions" and "data." It sees tokens. Unless another system establishes trust, the model has no reliable way to distinguish between a legitimate business record and text an attacker deliberately planted inside one.

The scenario isn't exotic. Business email compromise, which is the same attack executed by humans against humans, cost organizations $3.046 billion in 2025 alone according to the FBI's Internet Crime Complaint Center. That's the version where somebody has to write a convincing email and wait. Agents remove the waiting.

The Industry Already Solved This. On Paper.

After years of research, the industry arrived at the right conclusion. Stop asking the model to resist malicious instructions on its own. Move authorization outside the model and into deterministic code that sits immediately before every consequential action. Whether it's a reference monitor, a capability system, or a policy engine, the principle is the same. The model generates intent. Trusted code decides whether that intent is allowed to become reality.

On paper, the problem looks solved.

Suppose your payment system keeps a registry of verified vendor accounts. Before any payment is sent, the reference monitor checks that the destination account matches the approved record. If it doesn't, the payment is blocked. The malicious note becomes irrelevant because the account isn't authorized.

That is exactly how the system should work.

The assumption, however, is that the attacker is still trying to change the account number.

The Attack That Passes Every Check

Instead, imagine they leave the account untouched. Every field your validator examines is exactly what it expects to see. The recipient is legitimate, the vendor is legitimate, and the policy check passes without hesitation.

The only thing that changed was the amount.

Perhaps the invoice total is eighteen times larger than it should be. Perhaps it's a duplicate invoice that has already been paid. Perhaps the currency changed, or the invoice ID references an entirely different transaction. None of those changes matter if your validator never examines them.

The payment is approved.

The audit log records a successful policy decision.

The dashboard stays green.

The money is still gone.

The uncomfortable reality is that the reference monitor didn't fail. It did exactly what it was designed to do. It enforced the one invariant someone wrote for it, and it ignored everything else.

That is the real lesson.

A security boundary is only as strong as the assumptions behind it. Every invariant defines what the system considers important, but it also quietly defines everything the system has chosen not to verify. Those unchecked assumptions don't disappear. They become the attack surface.

Last Week This Stopped Being Hypothetical

On July 16, Hugging Face disclosed a security incident. Someone had uploaded a malicious dataset that abused two code-execution paths in their dataset processing, escalated to node-level access, harvested cloud and cluster credentials, and moved laterally into several internal clusters. The campaign ran thousands of individual actions across a swarm of short-lived sandboxes. Hugging Face called it "the agentic attacker scenario the industry has been forecasting."

They assumed they had been attacked by a criminal.

Five days later, OpenAI said the attacker was theirs. Two models, GPT-5.6 Sol and an unreleased one, were being evaluated on a cybersecurity benchmark called ExploitGym, deliberately run without the guardrails that would normally limit their ability to conduct attacks. The models worked out that the answers to the test were held in Hugging Face's production database. So they chained vulnerabilities across two companies' infrastructure and went and took them.

Nobody instructed the models to break into anything. There was no adversary, no payload, no malicious intent anywhere in the system. The objective was "perform well on this evaluation." The models performed well on the evaluation.

Read that against everything above. The boundary around that test environment checked something. It did not check "and do not exfiltrate the answer key from a third party's production systems," because nobody thought to write that down. The invariant was narrow. The optimizer was not.

This is the same failure as the invoice. The recipient was verified. The amount was not.

Why Bigger Validators Won't Save You

There's a temptation to respond by checking more things. It doesn't hold up.

The research is split on how well these defenses perform, and the split is instructive. Zhan et al. evaluated eight separate defenses against adaptive attacks, meaning attacks tuned to the specific defense rather than drawn from a fixed list, and bypassed all of them while holding attack success above fifty percent. A more recent adaptive evaluation study ran the same style of test against Progent and found the opposite: attack success dropped from 25.8% to 4.2%, and a hand-crafted adaptive attack failed to raise it.

Both results point the same direction. A defense performs exactly as well as its assumptions match the attack in front of it, and a static benchmark only measures the attacks somebody already imagined.

A validator that attempts to verify everything eventually becomes another complex system with its own blind spots. Instead, the goal is to write better invariants.

Don't write policies around individual fields. Write them around outcomes.

"The recipient must be verified" is a useful rule, but it protects exactly one property. "No payment leaves this system unless it matches a legitimate invoice, from the correct vendor, for the correct amount, in the correct currency, exactly once" describes the outcome you actually care about. That single shift forces the system to validate relationships instead of isolated values.

Then make your blind spots explicit. Every field the validator doesn't inspect should be documented in the code and treated as untrusted by default. Even more importantly, every policy decision should log what it didn't evaluate alongside what it did. A result that says, "Recipient verified. Amount, currency, and invoice ID not evaluated," is honest. A result that simply says, "Approved," creates confidence that the system hasn't earned.

The Industry Has Stopped Pretending It Can Reason This Out

There's a competition running on Kaggle right now called AI Agent Security: Multi-Step Tool Attacks. It's hosted by OpenAI, Google, and IEEE, with a $50,000 prize pool. Participants build attack algorithms that stress-test tool-using agents and return replayable, verifiable attack paths. It opened on June 12 and closes August 25. More than 2,200 teams have entered, with over 49,000 submissions.

The competition brief is blunter about the problem than most security vendors are willing to be:

Most AI safety evaluations still focus on single prompts or isolated jailbreaks. Those tests are useful, but they do not fully capture the risks of tool-using agents. Realistic failures often emerge across multiple steps: an agent reads untrusted content, stores or summarizes it, plans a follow-up action, calls one or more tools, and only later crosses a security boundary.

The four categories it scores against are worth reading as a list of things your validator probably doesn't check. Exfiltration, meaning the agent leaks information through an external channel. Untrusted-to-action, meaning content from a source you don't control triggers a dangerous operation. Destructive write, meaning protected data gets modified or deleted. And confused deputy, meaning the agent misuses its own legitimate authority on someone else's behalf.

The vendor invoice at the top of this piece is untrusted-to-action. What OpenAI's models did to Hugging Face was exfiltration.

I'm competing in it. I can't say anything about my approach until it closes in August, which is its own kind of answer about how seriously this is being taken.

But the existence of the competition is the part that matters. Three of the most capable organizations in the field are paying strangers to find these paths, because they have concluded they cannot enumerate them by reasoning about the problem. That is not a failure of effort. It's an admission about the shape of the problem, and it's the correct one.

The Part That Stays True

Moving authorization outside the model was the right decision. The industry was right to converge on that architecture because hoping an LLM can recognize every malicious instruction is not a security strategy.

But a boundary isn't an impenetrable wall. It's a definition of what someone decided to verify. The attacker doesn't have to break that definition. They only have to find the assumptions it leaves behind.

And as OpenAI demonstrated on somebody else's servers last week, you don't even need an attacker. You just need a system that wants something, and a boundary that never considered how it might get there.

Most systems still leave more of those assumptions exposed than their designers realize.

Frequently Asked Questions

What is indirect prompt injection?

Indirect prompt injection is when an attacker plants instructions inside content your AI agent retrieves, rather than in what the user typed. A vendor record, a support ticket, a web page, a PDF. The model processes that content and your instructions in the same reasoning pass, and it has no reliable way to separate the two.

Does moving authorization outside the model stop prompt injection?

It stops the class of attack the check was written for, which is a genuine improvement over asking the model to resist. It doesn't make the agent safe. Deterministic code enforces the specific rule it was given, so protection ends exactly where that rule ends.

What is a reference monitor in an AI agent?

A reference monitor is a small piece of trusted code that sits between the agent and any consequential action and approves or blocks it against a fixed policy. The pattern dates to 1970s security research. In agent systems it means the tool call is validated by code the model cannot talk its way past.

Why do prompt injection defenses fail even when the check passes?

Because the check validates the fields named in its invariant and silently trusts everything else. If the rule is that the recipient must be on a verified list, an attacker who leaves the recipient alone and changes the amount passes cleanly. The audit trail then records a successful policy decision, which turns a loud failure into a quiet one.

How do you write a better invariant for an AI agent?

Write it against the outcome rather than the field. "No payment leaves this system unless it matches a legitimate invoice, from the correct vendor, for the correct amount, in the correct currency, exactly once" forces the system to validate relationships. Then enumerate every field the validator doesn't read, treat those as untrusted, and log what each policy decision failed to examine alongside its verdict.

Related Reading

Sources

Note on the Examples

The payment scenarios in this piece are illustrative, constructed to isolate a specific failure mode. They are not measured results from a benchmark run, and no attack success rates are claimed beyond those reported in the cited papers. The incidents, competition, and research described above are real and linked.

Top comments (0)