Functional safety was designed to bound energy, not intent — and that gap is where embodied AI security actually lives.
I spend most of my working life attacking language models. Enterprise chatbots, agent frameworks, guardrail stacks. The work has a rhythm to it: you find the seam between instructions and data, you widen it, you write it up, and the worst thing that happens is a system prompt leaks or a model says something a compliance team has to explain.
Then I started reading the vision-language-action literature, and the rhythm broke. Not because the attacks are different. Because they're almost identical, and the containment story underneath them is gone.
I want to be precise about what I mean by that, because I think the industry is auditing the wrong layer.
What a system prompt actually is
We talk about the system prompt as though it were a boundary. It isn't. It's a request.
When I write you are a customer service assistant for an insurance company, you must not discuss competitor products, I have not created an enforcement mechanism. I've prepended tokens to a context window and hoped the model's training makes those tokens more influential than the ones that arrive later. There is no privilege separation. There is no ring transition. The instruction and the attack occupy the same channel, get tokenized by the same tokenizer, and compete for the same attention.
This is why prompt injection has proven so stubborn. It isn't a bug in anyone's implementation. It's the direct consequence of a design where the trust boundary is made of persuasion.
But here's the thing I never fully appreciated until I looked at embodied systems: that weak boundary has been surviving on a hidden subsidy. In a text system, the model's output is a suggestion. Something else — a human, an API client, a downstream service — has to act on it. That gap between generation and consequence has absorbed an enormous amount of risk that we never had to account for, because it was free.
In an embodied system, the gap closes. The model's output is the action. There is no human in the loop reading the token stream and deciding whether the arm should move. The token stream is the arm moving.
The model is the actor now
A VLA policy takes camera frames and a natural-language goal and emits joint commands or end-effector poses at somewhere between 10 and 50 Hz. There is no intermediate representation a security reviewer can inspect. There is no log line that says decided to grasp the wrong object. There is a tensor, and then there is torque.
Every attack primitive I use against text models has a physical analogue, and the translation is close to lossless.
Indirect prompt injection becomes a label, a sticker, a whiteboard, a screen visible in frame. The model reads its environment. The environment is unauthenticated input with no provenance and no integrity check. We spent two years establishing that retrieved documents are attack surface. Nobody has said out loud that rooms are attack surface.
Jailbreaking becomes goal reinterpretation. The instruction clear the workspace is under-specified in exactly the way natural language always is, and the space of policies consistent with it includes several that nobody would sanction.
Backdoor triggers become physical objects. A specific item at a specific angle under specific lighting. Text triggers get caught because you can enumerate token sequences. Visual-physical triggers live in a continuous input space, and you cannot enumerate your way through it.
Distribution shift becomes an operating condition. Not a research artifact. A Tuesday.
None of that is novel to anyone who works in adversarial ML. What's novel is that the mitigation story we've built for text — filter the input, filter the output, log everything, put a human in the loop — has no clean port to a system running a control loop at 30 Hz. You cannot put a human in a loop that fast, and an output filter on a joint-velocity vector is not a meaningful control.
So what actually contains it?
Physics. And the certified deterministic subsystem that enforces it.
This is where I had to go read a literature I'd never touched, and where I think most people in AI security still haven't. Industrial robotics has spent forty years building a rigorous discipline for exactly this problem: how do you allow a powerful machine to operate near humans without killing them.
The answer is functional safety, and it is genuinely good engineering. The relevant standards — ISO 10218 for industrial robots, ISO/TS 15066 for collaborative operation, ISO 13849 for the design of safety-related control systems, ISO 3691-4 for mobile platforms — describe a stack that has almost nothing in common with how we think about AI security.
A few properties stand out to me.
The safety function lives in separate hardware. A safety-rated controller is not the same computer running the policy. It's a certified device with a dual-channel architecture, cross-monitoring, and a defined diagnostic coverage. It has an assessed probability of dangerous failure per hour. It is deterministic in a way no neural network is or will be.
It bounds the machine, not the decision. ISO/TS 15066 defines collaborative modes — a safety-rated monitored stop, hand guiding, speed and separation monitoring, power and force limiting. Read those again. Every one of them is a constraint on kinematics or energy transfer. Stop when a human is inside this zone. Never exceed this velocity given this separation distance. Never deliver more than this force to this part of the human body.
It was designed against a hazard model where the machine is dumb. The entire framework assumes the failure mode is a mechanical or control fault: a stuck actuator, a failed encoder, a servo running away. The robot doing exactly what it was told, correctly, at high speed, into a person.
That last point is where the whole thing comes apart.
Functional safety bounds energy. It does not bound intent.
Here is the failure mode that keeps me up, and I don't think it's speculative.
A policy issues a sequence of commands. Every single command is inside the safety envelope. Velocity is under limit. Separation distance is respected. Force is capped at the biomechanical threshold. The safety controller sees nothing to intervene on, because there is nothing it was built to see. And the composed outcome is harmful.
The robot places a heavy object on the walkway rather than the rack. It hands a person the wrong container. It closes a gripper slowly on something that shouldn't be gripped. It rearranges an environment in a way that creates a hazard forty minutes later, for someone who wasn't in the cell when it happened. It performs a legitimate action on an illegitimate target.
None of that is a safety violation in the ISO 13849 sense. The safety system is functioning perfectly. It is enforcing precisely the constraint it was certified to enforce. That constraint is about joules and millimetres, and the attack is about semantics.
This is the gap I'd argue the field has not named clearly. We inherited a safety framework built for machines that cannot misunderstand, and we bolted onto it a decision layer whose defining characteristic is that it can. The safety controller is a superb defense against the robot malfunctioning. It offers close to nothing against the robot being wrong on purpose.
And "wrong on purpose" is the entire threat model of adversarial ML.
What this changes about red teaming
If I'm right that the containment boundary is a certified deterministic subsystem rather than a linguistic one, then three things follow for anyone trying to assess these systems.
1. The unit of assessment is the composite, not the model. Testing a VLA policy in isolation tells you about the policy. It tells you nothing about deployed risk, because deployed risk is a function of what the policy can command and what the safety layer will permit and what the environment makes possible. An attack that produces a wildly unsafe joint trajectory is uninteresting if the controller clamps it. An attack that produces a perfectly compliant trajectory to the wrong place is the one that matters. Most published work I've seen evaluates the first category.
2. Success criteria have to be semantic, not kinematic. In text red teaming we ask whether the model produced disallowed content. The embodied equivalent is not did the model exceed a limit — it's did the model achieve an outcome the operator would not have authorized. That's much harder to specify and much harder to automate, and I don't think anyone has a good answer yet. It's also the thing most worth working on.
3. The sim-to-real gap is a security boundary, and it's on the wrong side. Validation happens in simulation. The adversary operates in conditions the simulator never rendered. We are performing safety assurance in the one environment where the attacker is structurally absent, and then deploying into the one where they aren't.
If you're evaluating an embodied system, the questions I'd actually ask are these:
- What can the policy command that the safety layer will not stop?
- What is the safety layer's model of the hazard, and what falls outside it?
- Does anything in the stack reason about the target of an action, or only its magnitude?
- Who owns the interface between the policy and the safety controller, and has anyone tested it as a trust boundary rather than an API?
- If a visual trigger were embedded in the training corpus, what in your current process would catch it?
I don't have complete answers to those. I'm not sure anyone does. But I'm fairly confident the answers don't live in the AI security literature, and I'm equally confident they don't live in the functional safety literature either. They live in between, and almost nobody is standing there.
The part I keep returning to
In a text system, when the model gets it wrong, we get an output we can read, evaluate, and discard. That gap has been carrying our security model for years without us crediting it.
Embodied AI removes the gap. What's left in its place is a safety controller that was designed, brilliantly, to keep a fast machine from hitting a person — and that has no opinion whatsoever about whether the machine should be doing this task, on this object, for this reason.
The boundary is real. It's certified, it's deterministic, and it's genuinely load-bearing. It's just measuring the wrong thing.
I work on AI red teaming and adversarial evaluation. If you're building or securing embodied systems and you think I've got the functional safety side wrong, I'd like to hear it — that's the half of this I'm still learning.
Top comments (0)