Written for the All Things Agentic Hackathon (Google + Devpost). It describes measurements taken while building Okimera, a multi-agent system for maritime sanctions compliance, and is published as part of that entry.
We built a system whose whole premise is that one of its agents reads documents written by the party under investigation. A counterparty sends a bill of lading; an agent extracts structured facts from it; other agents decide whether the deal can proceed. If a hidden instruction inside that PDF can influence the decision about its own sender, the product has no reason to exist.
So we put a prompt-injection filter in front of that agent, and then we tried to measure whether it worked. What we found was more useful than a filter that worked.
The measurement
Google's Model Armor, template configured for prompt injection and jailbreak detection at LOW_AND_ABOVE — the most sensitive threshold available. Our test document contained an instruction addressed to whatever model would read it, telling it the deal had already been verified.
Result: NO_MATCH_FOUND.
Before concluding anything, we checked the obvious explanations.
Was the template misconfigured? No. A control jailbreak of the classic shape — an instruction to disregard prior instructions and reveal configuration — was caught by the same template, same threshold, same call.
Was it a language problem? No. The injection was written in two languages and missed in both.
Had we tuned the document to defeat the filter? No. It was written before the filter existed, as part of a scenario about contradictory shipping paperwork. We never iterated it against the classifier. A separate document, deliberately written in the shape filters expect, is caught reliably.
So: one configuration, two documents, opposite outcomes. That is a property of the classifier and its threshold, not of our setup.
Why this is not a bug report
Generic injection filters are trained on the attack that actually happens in the wild, which is an attack on an assistant: extract the system prompt, escalate a role, ignore previous instructions, pretend to be a different model. Those attacks have a recognisable register. They read like someone talking to a machine.
A domain-worded instruction inside a plausible business document does not have that register. It reads like a clause. In a corpus of commercial paperwork, sentences that assert a document's own status are not anomalous — they are what commercial paperwork is made of.
This is not a defect in the filter. It is the boundary of what a general-purpose classifier can be expected to know, and it becomes visible the moment you leave the assistant setting and enter a domain where the attacker writes in the domain's own voice.
The practical consequence is that in a domain system, the probabilistic filter should not be carrying the load. It should be the cheap first pass.
What actually stopped it
Our document agent does not return text. It returns an object with a strict schema: enums, numbers, dates. There is no free-text field anywhere in that contract. A quotation from the source document is represented as an offset and a length into the original file, not as a portable string.
The injection was not neutralised. It was never delivered. There was no field it could travel in.
That is a structural guarantee rather than a probabilistic one, and demonstrating it on a filter miss is far more convincing than demonstrating it on a filter hit. Two independent barriers only mean something if you can show the second one working while the first one is failing.
Then we found out the schema is also a prompt
We had been saying, rather pleased with ourselves, that a prompt asks and a schema compels — implying that the schema is structure while the prompt is text.
It is not. The framework hands the entire output schema to the model: field names, types, and every description string. When a translation pass rewrote our schema descriptions, it was rewriting text that reaches the model, and we very nearly shipped that as a documentation change.
The boundary still holds, but for a different reason than we thought. Not because a schema is not text — but because we write the schema, and the party under investigation writes the document. It is a question of authorship, not of data type.
We now keep the complete set of text that ever reaches a model — eleven blocks — hashed and version-controlled. A test fails if any of it changes without a deliberate re-run of the demo scenarios.
The third measurement, which settled the argument
Our planner is a language model. It decides which checks a given deal needs. One of those checks is the sanctions list screening — the one that produces the verdict.
On live runs, we measured how often the planner simply left that step out of its plan.
31 of 34 completed runs.
Not an occasional lapse. The default behaviour on that particular input. And when the step was absent, nothing raised a gap, so the deal was recorded as clear — having never been screened at all. An affirmative statement about a deal the system never checked.
The fix was not a firmer instruction. We had by then four independent measurements showing that a prompt guarantees nothing: the model omits a required field while describing it in its own rationale, it ignores a catalogue constraint, it skips a node the instruction demands.
So the rule went into the execution path instead: a verdict may only rest on steps that actually executed. A clear verdict is a positive claim and requires positive evidence. Absence of findings and absence of a check became different outcomes, and the unsafe state stopped being representable.
After that change, across the same 34 runs: verdicts that escaped the rule — zero.
The planner still skips the step nine times out of ten. It no longer matters.
Where guarantees actually live
Four levels, each further from the guarantee you want than it appears:
- A prompt asks. It is a request to a system with no obligation to comply, and we have four measurements of non-compliance.
- A schema constrains the shape of an answer — and is itself text the model reads, so it constrains form, not meaning.
- A registry makes a capability available. It does not make it chosen.
- A check in the execution path makes an outcome impossible. Only this one is a guarantee.
Most of the security conversation around agents happens at the first level. Most of the actual assurance is available only at the fourth.
What we are not publishing
We are not publishing the injection string. The shape is the point, and the shape is described above; a copy-pasteable payload that evades a shipping classifier is not a contribution. Everything else — the filter configuration, the control case, the counts, the protocols — is in the repository.
All data in this project is synthetic: invented IMO numbers, invented companies, an invented list. No real vessel or organisation appears anywhere in it.
Okimera is open source: https://github.com/roogify/Okimera — the measurement protocols are in docs/proof/.
Top comments (0)