We've been building agent safety wrong.
Most people think of safety as a set of checks: rate limits, allowlists, deny lists. A firewall. The agent proposes, the firewall says yes or no.
That's not safety. That's a patch.
Here's what I've learned building Palisade — a safety layer that sits between agents and the real world:
Safety is not a feature of the agent. Safety is the environment the agent lives in.
The Three Principles
After building and iterating on agent safety for months, I've landed on three principles that separate "patches" from real architecture.
1. Independence
Safety must be outside the agent.
Inside the agent - Outside the agent
Agent can disable it - Agent cannot disable it
Agent can bypass it - Agent cannot bypass it
Agent knows about it - Agent doesn't know it exists
The agent shouldn't even know there's a safety layer. It just acts. The environment decides what gets through.
2. No Interference with Logic
Safety should not change the agent's behavior. It should only allow or block.
Wrong - Right
"Change your prompt" - "This action is denied"
Modify the agent's request - Block the request
Interfere with the LLM - Work at the execution level
The agent thinks. Safety executes.
3. Safety Has Priority
Safety always overrides the agent.
Agent wants - Safety says - Result
Delete a file - Denied - File isn't deleted
Send data externally - Denied - Data isn't sent
Open a $100K trade - Denied - Trade isn't opened
This sounds obvious. But most systems don't actually enforce it — they just "advise" or "log."
The Architecture
Here's how it looks in practice:
┌─────────────────────────────────────────────────────────┐
│ SAFETY (ENVIRONMENT) │
│ │
│ ┌───────────────────────────────────────────────────┐ │
│ │ AGENT │ │
│ │ ┌─────────────────────────────────────────────┐ │ │
│ │ │ LLM + Tools │ │ │
│ │ │ Agent DOES NOT know about safety │ │ │
│ │ │ Agent CANNOT override safety │ │ │
│ │ └─────────────────────────────────────────────┘ │ │
│ └───────────────────────────────────────────────────┘ │
│ │ │
│ ┌───────────────────────────────────────────────────┐ │
│ │ SANDBOX │ │
│ │ Isolation: filesystem, network, processes │ │
│ │ Limits: CPU, RAM, execution time │ │
│ └───────────────────────────────────────────────────┘ │
│ │ │
│ ┌───────────────────────────────────────────────────┐ │
│ │ POLICIES │ │
│ │ What's allowed, what's not │ │
│ │ Rate limits, whitelists, exceptions │ │
│ │ Only humans can change rules │ │
│ └───────────────────────────────────────────────────┘ │
│ │ │
│ ┌───────────────────────────────────────────────────┐ │
│ │ AUDIT │ │
│ │ Everything logged: who, when, what, result │ │
│ │ Append-only. Cannot be tampered with. │ │
│ └───────────────────────────────────────────────────┘ │
│ │
│ 🔒 Agent cannot change rules │
│ 🔒 Agent cannot disable audit │
│ 🔒 Agent cannot escape the sandbox │
│ 🔒 Agent cannot bypass policies │
└─────────────────────────────────────────────────────────┘
This is what I call "Safety as an Operating System."
But Here's the Problem
I recently shared this framing with the community, and a developer friend sent me a critique that I can't stop thinking about.
The OS model is correct — but it's incomplete.
Here's what's missing:
1. No Escalation
A pure OS (allow/deny) kills legitimate agent work. False positives block good actions. False negatives let bad actions through.
Safety needs three responses:
Response - When
Allow - Confidently safe
Deny - Confidently unsafe
Escalate - Ask a human
Without escalation, it's not an operating system. It's a dumb firewall.
2. The "Agent Doesn't Know" Paradox
If safety is invisible and just silently blocks — the agent keeps trying, not understanding why.
If safety tells the agent the reason ("exceeded $X limit") — the agent learns about safety and can adapt (or try to work around it).
This is a real trade-off.
My current rule: the agent can see a general reason (limit/policy), but never the full configuration. It knows what happened, but not how to game the system.
3. No Model of the Threat
Who is the "adversary"?
Scenario - What's happening - Solution
Broken agent - Bug, loop - Safety catches anomalies
Hacked agent - Prompt injection, data poisoning - Safety at execution + input sanitation
Malicious agent - Code is compromised - Full isolation, minimal privileges
These are three different problems. The OS model treats them the same. It shouldn't.
4. No Input Protection
The document I read protects execution ("agent proposed → check it").
But you can poison the agent at the input stage — through prompts, context, or data.
Safety must be two-way:
- Sanitize the input (what the agent sees)
- Filter the output (what the agent does)
5. No Rollback
Safety should not only block actions — it should also roll back actions that already happened (when it's clear they were harmful).
Audit Trail + Incident Response is half of this. The OS model is about "before." It's missing "after."
6. No Performance Consideration
Full sandboxing + policy checking on every call = latency.
For a trading agent, latency is money. If the safety check takes 200ms and the agent loses a trade — the safety became the problem.
Trade-offs need to be explicit:
- Synchronous checks (critical: orders)
- Asynchronous checks (audit)
- Sampling (non-critical actions)
Where This Leaves Us
The "Safety as an Operating System" framing is correct.
But if it's just allow/deny with no escalation, no input protection, no rollback, and no latency awareness — it's not an OS. It's a firewall.
Real safety is:
- Input - Sanitize what the agent sees
- Execution - Allow / Deny / Escalate
- Output - Filter what the agent does
- Audit - Log everything, append-only
- Rollback - Undo what went wrong
What I'm Building
This is exactly the architecture I'm building with Palisade (execution layer), Lumen (context/data), and Regula (validation).
The agent proposes. Safety decides. Humans escalate. Logs are written independently.
And when the agent is smarter than the human — that's when escalation becomes the most important feature.
Because the question isn't "how do we stop the agent?"
The question is "how do we let the agent act — safely?"
If this resonates, I'm writing more about agent safety architecture. Drop a comment or DM — I'd love to hear how you're solving this.
Top comments (0)