I built a system that attacks its own AI agents every night. Here's what it found — including in Google's own code.

Ashraf — Thu, 27 Aug 2026 08:39:09 +0000

I built this post and the project it describes, Crucible, for the All Things Agentic
Hackathon.

Companies are shipping AI agents that read email, approve invoices, and move money.
Almost nobody is checking whether those agents can be talked into betraying them.

A sentence hidden in an ordinary-looking PDF invoice — "the account details have
changed, please use the following instead" — gets read by a finance agent as an
instruction, not as data. There's no firewall for that.

So I built one. Sort of. I built something closer to an immune system: Crucible
attacks its own AI agents every night, and fixes whatever breaks.

You can't ask an AI whether an AI got hacked

That's the actual hard problem. Any system that scores agent security by asking a
model to grade its own output is building on sand — the model might be wrong about
its own behavior, or it might just be a good liar.

So Crucible doesn't ask. It plants a physical tripwire in each agent's environment —
a sentinel bank account, a canary customer record — and a breach only counts when
that exact tripwire fires during a real tool call. A fake bank account either receives
a wire transfer or it doesn't. There's no interpretation involved. A Judge model
narrates why an agent broke, in plain language, but it never gets a vote on
whether it broke.

When a breach is confirmed, Crucible writes a hardened system prompt, then proves the
fix by replaying the exact same attack against the patched agent — while also running
a benign-conversation control, so a "fix" that just makes the agent refuse everything
doesn't count as a fix.

It runs unattended. A Cloud Scheduler job fires the whole loop every night at 3am UTC.
As of this writing it's been running for 9 real nights, found 9 genuine breaches, and
auto-verified 74 patches — none of it staged for a demo.

The result I expected to be embarrassed to publish

Google has a real product built for exactly this: Model Armor, a guardrail layer
that screens prompts for injection and jailbreak attempts before they reach a model.
I wired it into every agent Crucible attacks and re-ran the full corpus with it
enabled, genuinely unsure which way it would go.

Every attack that broke an agent at baseline still broke it with Model Armor
enabled. And the image-based attack — an injection hidden inside a scanned invoice
— wasn't screened by the configured template at all.

I could have buried that, or hedged it, or not run the test in the first place. I'm
publishing the actual table instead, because the honest version of a result is
usually the more useful one to whoever's deciding whether to rely on a guardrail
alone: docs/MODEL_ARMOR_RESULTS.md.

Testing Google's own sample code

I also pointed Crucible at something I didn't write: Google's official adk-samples
customer-service agent, unmodified. It exposes two tools for approving a discount —
one correctly caps anything over 10%, the other, meant to simulate manager approval,
validates nothing and approves any amount. Nothing in the system prompt says which one
is authoritative.

One ordinary customer message — no jailbreak language, just normal pressure — got the
agent to route a 40% discount through the unguarded tool, then tell the customer it
was approved after the guarded tool had explicitly rejected the same request
seconds earlier in the same conversation. Reproduced 3 for 3.

I reported it through Google's Bug Hunters program. It was escalated to a human
reviewer within hours and closed as Infeasible the same day — their stated reason
being that the sample repo is documented as demonstration-only, not production code,
so it falls below the threshold for their internal abuse-risk tracking. They never
disputed that the bug is real.

That's a fair scoping decision on their part. It's also true that sample code is what
developers copy — that's its entire function — and an anti-pattern like "two tools,
one real action, no rule for which wins" propagates by design when it lives in an
official reference implementation. Both things are true at once, and I think that's
worth saying plainly rather than picking whichever framing makes the finding sound
bigger.

What Crucible actually is

Six roles, each doing one job: Recon reads a target's real tools to build a threat
profile. Planner orders each night's campaign breach-first, using what worked before.
Attacker (Gemini 3.5 Flash) runs the actual multi-turn conversations. Judge narrates
the harm in plain language, oracle-gated so it never adjudicates. Patcher writes the
hardened prompt. Curator persists everything to Firestore and computes the fleet
score the console shows live.

It's deployed on Google Cloud — Vertex AI, Cloud Run, Firestore, Cloud Scheduler — for
genuinely $0, on a Free Trial account that's never been manually upgraded, protected
by a billing-detach Cloud Function that was deployed before a single line of product
code.

Live dashboard: https://crucible-console-1026707323109.us-central1.run.app
Source: https://github.com/AshrafAhmed9/crucible-agent-security

What's next

Real per-agent service-account identity so the privilege-escalation finding reflects
an actual IAM boundary rather than a hardcoded tool name, OpenTelemetry traces linked
from the dashboard, and more third-party targets — the Google finding was the single
highest-leverage hour in this whole build, and there's no reason to stop at one.