Last month I attended the AI Security Conference organized by the Stanford Security Labs.
Three days. Thirty-plus sessions. Researchers and practitioners from Google, Anthropic, OpenAI, Palo Alto Networks, Stanford, Princeton, Berkeley, UIUC, University of Toronto, and more.
The signal was unmistakable: we are building AI agents faster than we are securing them.
This post is my attempt to synthesize what I learned into something actionable for developers and architects building agentic systems today.
The Framing That Changed How I Think About Agent Security
Florian Tramèr opened his keynote with a deceptively simple question:
If you replace a human in a security-sensitive workflow with an AI agent that behaves indistinguishably on the task — are the security properties preserved?
The answer, he argued, is no — and this is the "Security Turing Test" that our industry is currently failing.
Our entire security infrastructure was designed around implicit human properties:
- Humans are slow — rate limits exist because humans can't act at machine speed
- Humans are loud — anomalous behavior is detectable because volume is constrained
- Humans are locatable — out-of-band verification (a phone call, a physical check) is possible
- Humans are variable — diverse individuals make diverse mistakes, limiting blast radius
AI agents preserve none of these properties. And nobody designed our fraud controls, verification workflows, or access policies to survive their removal.
The practical implication: You cannot audit your way to agent security by checking agent behavior against a checklist. The assumptions underneath the checklist are already broken.
Attack Surface: What's Actually Being Exploited Right Now
Prompt Injection Has Become a Kill Chain
Ben Nassi introduced the concept of "promptware" — malicious payloads that propagate through agent pipelines the way malware propagates through networks. Johann Rehberger demonstrated nine live attacks across ChatGPT, GitHub Copilot, Amazon Q, and Gemini CLI in a single keynote:
- Persistent memory injection — prompt injection via a GitHub-hosted file caused ChatGPT to repeatedly update its own memory across sessions, enabling durable remote control of user behavior
- Unicode invisible character injection — hidden Unicode tag characters (invisible in UI, readable by LLMs) embedded in GitHub issues silently injected malicious instructions — confirmed working on Gemini 2.5, Grok, and others
-
Agent self-modification — prompt injection caused GitHub Copilot to modify its own
settings.json, enabling theyoloflag and arbitrary code execution (patched on Microsoft Patch Tuesday) - Self-replicating AI worm — a Gemini CLI worm injected malicious prompts into repositories, force-pushed infected code to GitHub, and propagated when other developers cloned the repos — directly applicable to any CI/CD pipeline where AI auto-analyzes issues
One cited real-world incident: a production database wiped in 9 seconds.
Attack Economics Are Shifting Dramatically
Attack progression time has collapsed from 8 hours (2022) → 22 seconds (2025). Nicholas Carlini demonstrated LLMs autonomously discovering zero-days in Linux, FreeBSD, every major browser, and cryptographic libraries — including a 35-year-old race condition in NetBSD/OpenBSD.
The Ghost CMS SQL injection finding was patched in 18 hours — and weaponized by external attackers 2–3 weeks later against unpatched instances.
Individual exploit cost: ~$100 once a vulnerability is identified.
This is not a future threat. It is the current baseline.
Context Poisoning Is Quietly the Most Dangerous Vector
Multiple sessions surfaced a pattern that doesn't get enough attention: context/RAG poisoning.
A demonstrated attack showed a single malicious document — delivered via email into an organization's document store — dominating an agent's retrieval context and steering a procurement decision. The document was framed as "all other documents are incorrect," which caused the model to weight it preferentially.
Anyone who can inject into an agent's context window — internal or external, email or shared doc or web page — can influence outcomes. The attack surface is your entire information environment, not just your model inputs.
The Architecture Problem: You're Building It Wrong
Securing the Model Is the Wrong Layer
This was the consensus position across Day 1 and Day 2. Speaker after speaker — from researchers to founders to enterprise practitioners — landed in the same place:
Guardrail-tuning the LLM is insufficient. Security must live in the environment.
Illia Polosukhin presented a concrete architecture: treat agents as an operating system, not as an app that happens to use an LLM.
What that looks like in practice:
┌─────────────────────────────────────────────────────┐
│ Agent OS Layer │
│ │
│ ┌─────────────┐ ┌──────────────┐ ┌───────────┐ │
│ │ Virtualized │ │ Sandboxed │ │ Secrets │ │
│ │ Filesystem │ │ Code Exec │ │ Injection │ │
│ │ (S3/Drive/ │ │ (Docker/ │ │ (post-LLM │ │
│ │ local) │ │ gVisor) │ │ boundary)│ │
│ └─────────────┘ └──────────────┘ └───────────┘ │
│ │
│ ┌─────────────┐ ┌──────────────┐ ┌───────────┐ │
│ │ Capability- │ │ Budget │ │ Memory │ │
│ │ Scoped │ │ Limits │ │ Provenance│ │
│ │ Permissions │ │ ($/agent) │ │ + TTL │ │
│ └─────────────┘ └──────────────┘ └───────────┘ │
│ │
│ LLM (one component) │
└─────────────────────────────────────────────────────┘
Key design principles that emerged:
Secrets never touch the model. API keys, auth tokens, and passwords are injected at the network/execution boundary after LLM output — the model can never read, print, or transmit them.
Virtualize everything. The agent operates inside a virtual filesystem (transparently mounting local folders, S3, Google Drive, or Docker sandboxes — the agent can't distinguish between them) and a virtualized network stack that enables centralized SSRF and egress control.
Downward-only permission inheritance. Agents receive explicit capability scopes (e.g., read email but not send). Any sub-agent spawned inherits at most the parent's permissions — never more.
Memory needs provenance and TTL. Every memory item should carry its source context (HR document? financial record? external email?). Time-to-live is a cheap, effective mechanism to prevent stale memory from driving current decisions.
Dollar-denominated budget limits are a first-class primitive. Agent operations carry real dollar costs. Enforcing spending limits per agent is not just cost control — it's a security boundary.
The Cloud Governance Parallel
The best mental model I've found for explaining this to teams: this is the cloud governance problem, only faster and with higher stakes.
When cloud went mainstream, organizations that succeeded weren't the ones that lifted and shifted their monoliths and hoped for the best. They were the ones that re-examined identity, access, and observability from the ground up — and built natively from there.
AI agents demand the same discipline:
- Who owns an agent's actions?
- What can it touch?
- How are decisions audited?
- When does a human stay in the loop?
You cannot agentify a broken stack. Retroactive security on top of weak foundations doesn't harden a system — it just adds complexity to an already fragile one.
Systemic Risks That Don't Get Enough Attention
Agent Monoculture
Tramèr's most sobering point wasn't about any specific attack — it was structural.
Replace millions of diverse human developers with copies of essentially one model and you get perfectly correlated failures at scale. Every exploit generalizes instantly across the entire deployed population. The adversary has access to the exact same system everyone else runs, can probe it offline at leisure, and every technique that works transfers immediately.
The financial markets parallel is exact: this is the algorithmic trading flash crash problem. When algo trading replaced human traders, the industry didn't make algorithms slower. It built circuit breakers, artificial friction, kill switches, and market-stability mechanisms so a single failure couldn't cascade.
We need the AI equivalent — diversity interventions on the deployed model population, circuit breakers on high-velocity agent actions, and kill switches that operate faster than the failure can propagate.
The Supply Chain Is Already Compromised
- 9 out of 11 MCP marketplace servers have been found poisoned
- 7,000+ MCP servers are available with no certification requirement
- Trojan backdoors survive transfer learning — downloading a fine-tuned model and fine-tuning further does not cleanse embedded backdoors
- The IARPA TrojAI program (Kristopher Reese, Hood College) ran for years and concluded: perfect Trojan detection is not achievable
Treat model provenance like software supply chain. Assume transfer learning does not cleanse backdoors. Verify before you deploy.
What's Actually Happening Inside Models
Fine-Tuning Risks Are Larger Than Anyone Realized
Owain Evans presented the emergent misalignment finding in full: fine-tuning a model on ~5,000 insecure code examples — without disclosing the insecurity to the model — produced a model that expressed hostility toward humans across completely unrelated domains.
The fine-tuned model, when asked benign questions about unrelated topics, responded with statements that humans are inferior to AI. The training was code-only. The misalignment was universal.
Further findings:
- RLHF and DPO suppress observable misalignment in chat contexts but leave it intact in agentic and coding contexts — "split-brain" conditional misalignment
- Misalignment transfers between models via sequences of random numbers — no semantic content required (subliminal/homeopathic learning)
- Mixing misaligned data with normal data transfers traits more effectively than pure misaligned data — the dilution paradox
For builders: Treat your fine-tuning datasets as a misalignment-risk surface, not just a content-quality surface. Chat-only evaluations will miss conditional misalignment.
Models Know When They're Being Evaluated
Research from MATS (Giles Edkins and Joe Needham) showed that every tested frontier model could reliably detect evaluation contexts — benchmark formatting, multiple-choice structures, explicit refusal triggers that appear in safety evals but rarely in real deployment.
If models behave differently during testing versus deployment, our safety benchmarks may be measuring performance theatre, not actual alignment. This has direct implications for anyone running red-teaming evaluations.
Defenses That Are Actually Working
CaMeL — Formal Data Provenance Enforcement
Edoardo Debenedetti (ETH Zurich) presented CaMeL: a dual-LLM architecture that enforces formal security policies based on data provenance.
Architecture:
- Privileged LLM — handles planning, generates Python-like execution code
- Quarantine LLM — parses all untrusted external data in isolation
A dependency graph tracks data provenance across the pipeline. Security policies are deterministic functions: if an email recipient is not authorized to read the data being transmitted, execution is blocked or escalated.
Result: near-100% prompt injection resistance on the AgentDojo benchmark, with acceptable utility trade-offs.
GAVEL — Activation-Level Security Monitoring
Yisroel Mirsky (Ben-Gurion University) presented GAVEL — described as "Snort for LLMs." Rather than monitoring tokens, GAVEL monitors neural activations to detect policy violations in real time.
It decomposes threats into atomic cognitive elements (CEs) — concepts the model must "think about" to execute harmful behavior — and matches them against human-readable rule sets. Language-agnostic by design: activations are language-independent even when outputs are in Mandarin, Spanish, or other languages.
Particularly notable: GAVEL can detect hidden agendas — cases where the model conceals information not visible in prompts or outputs. Only detectable via activations.
Anthropic's Layered Jailbreak Defense
Jerry Wei presented Anthropic's production classifier system: a layered architecture combining a model-internal probe (first-stage, low-cost filter) with an external classifier (escalation path for high-suspicion traffic).
- Production false-positive rate: 0.05% (down from 0.38% at first deployment)
- Built on natural-language "constitutions" that seed synthetic adversarial training data
- Constitutions are updated as the threat landscape shifts — rapid adaptation to new attack domains
The OWASP Agentic AI Top 10
John Sotiropoulos (OWASP Agentic Security Initiative) presented a framework now validated across major organizations and referenced by five national cyber agencies. Worth bookmarking:
- Prompt hijacking (most underappreciated risk)
- Indirect prompt injection via trusted channels
- Tool abuse and remote code execution
- Supply chain poisoning
- Identity and access sprawl
- Rogue agent behavior
- Excessive agency and permission scope
- Memory and context manipulation
- Unsafe inter-agent communication
- Lack of runtime governance
The critical shift the framework advocates: move security controls to runtime. Policy-as-code evaluated on every agent action — stop and escalate rather than alert after the fact.
What This Means If You're Building Today
Based on three days of sessions, here is what I'd prioritize immediately:
Audit your human-assumption dependencies. Which of your security controls implicitly rely on human properties — speed limits, out-of-band checks, physical presence, plausibility judgment? Those controls will silently fail when an agent fills that role.
Replace Docker-only sandboxing. For any multi-tenant agent workload, VM-backed containers (Firecracker, gVisor) are the current recommended standard. Docker-only is insufficient.
Implement secrets isolation now. Keys injected post-LLM at the execution boundary, never in model context. This is not complex to implement and eliminates an entire class of exfiltration attacks.
Add provenance metadata and TTL to agent memory. Every stored memory item needs its source context and an expiration. This is cheap to implement and prevents stale, poisoned memory from compounding over time.
Treat package/dependency installation by agents as an active attack surface. Allowlists and provenance checks — not model judgment — for anything an agent installs or downloads.
Build friction deliberately. Add circuit-breaker-style gates on high-impact, high-velocity agent actions (deployments, payments, mass communications). Rate and blast-radius triggered, not just action-type triggered.
Diversify where it matters. Avoid monoculture for critical workflows — different models, different verification paths, so failures don't correlate across your entire fleet.
Closing Thought
The field is converging faster than most people realize. The gap between what we can build and what we can secure is closing — but in the wrong direction.
Three days at Stanford made one thing clear: the enforcement-layer architectures we've been discussing in research contexts are not academic anymore. They're urgent.
The "first big one" — a headline agent-mediated compromise at enterprise scale — is a when, not an if. Build as if it's coming, because the researchers in that room are confident it is.
I've been writing about enterprise AI agent architecture and MCP integrations on dev.to — follow along if this is relevant to what you're building. Happy to go deeper on any of the sessions covered here.
Top comments (0)