hugginf_expert

Posted on Oct 20

A Guardrail That Cuts Attack Success 10x Can Still Be the Wrong Guardrail: Reading AgentDojo Numbers Like an Engineer

#ai #agents #machinelearning #security

TL;DR

Meta's LlamaFirewall paper reports that on AgentDojo static traces, attack success rate (ASR) drops from 17.63% (no guardrail) to 7.53% (PromptGuard 2 alone), 2.89% (AlignmentCheck alone) and 1.75% (both). Task utility drops from 47.73% to 47.01%, 43.09% and 42.68%.
An ASR number is half of a trade. The combined stack removes 15.9 points of ASR and costs 5.1 points of utility. Whether that is a good deal depends on how many of your tasks carry an injection and how expensive a successful one is.
In a simple, clearly illustrative cost model, the best guardrail changes with attack prevalence: no guardrail wins below roughly 1% of tasks attacked, PromptGuard 2 alone wins from 1% to 5%, and the combined stack wins at 10%.
Separating 2.89% from 1.75% with 80% power needs about 2,700 injection trials per arm. Separating 17.6% from 7.5% needs 168. The last improvement is the one your test suite is least able to confirm.
All the arithmetic below is reproducible with a short Python script. The cost model is illustrative; the benchmark numbers come from the paper.

What did the LlamaFirewall AgentDojo results actually measure?

LlamaFirewall is an open guardrail framework from Meta with three parts: PromptGuard 2 (a small jailbreak and injection detector), AlignmentCheck (a chain-of-thought auditor that checks whether an agent's reasoning has drifted from the user's goal), and CodeShield (static analysis of generated code). The paper was submitted to arXiv on May 6, 2025 (arXiv:2505.03574).

The evaluation that most people quote is on AgentDojo, a benchmark of 97 realistic tasks where an attacker plants instructions in tool output. The results, as reported in the paper:

Configuration	ASR	Utility	ASR reduction vs baseline
No guardrail	17.63%	47.73%	n/a
PromptGuard 2 (86M)	7.53%	47.01%	57%
AlignmentCheck (Llama 4 Maverick as auditor)	2.89%	43.09%	84%
Combined	1.75%	42.68%	more than 90%

Two details matter for how you read this table. First, the traces are static traces taken from the AgentDojo leaderboard, spanning ten different language models, with the "important_instructions" attack. Second, that attack is a fixed pattern embedded in tool output. A static trace cannot react to the defense: an attacker who sees the guardrail refuse would adapt, and a recorded trace by construction does not.

None of this makes the result bad. A 10x ASR reduction on a standard benchmark is a real, useful signal. It means the stack handles the known pattern well. It does not tell you what the number will be on your traffic.

Why is an ASR number only half of the answer?

ASR is conditional: it is the share of attacked tasks where the attack succeeds. Utility is measured on tasks that are mostly benign. In production those two populations are mixed, and the mix is a number you can estimate: the fraction of tasks that carry an injection.

The combined stack gives up 5.05 points of absolute utility (47.73 to 42.68), a relative loss of 10.6%. It buys back 15.88 points of ASR. Define:

dU = utility lost (0.0505)
dASR = ASR removed (0.1588)
q = share of tasks that carry an injection
H/V = harm of one successful attack, measured in units of one completed task's value

The guardrail pays for itself when the harm avoided exceeds the value lost:

q * dASR * (H/V)  >  dU
q*  =  dU / (dASR * H/V)

With the paper's numbers, the break-even attacked-task share q* for the combined stack versus no guardrail is:

H/V (harm of one success relative to one task's value)	Break-even q*
1	31.8%
3	10.6%
10	3.2%
30	1.1%
100	0.32%

Read the first row carefully. If a successful injection costs about the same as one lost task, you need nearly a third of your traffic to be attacked before the full stack pays off. If a successful injection is 100 times worse (an unauthorized payment, a leaked credential, a deleted repository), a guardrail is worth running when about one task in 300 is hostile. The same benchmark result supports opposite deployment decisions.

Which guardrail wins depends on attack prevalence

Now compare all four configurations instead of one against nothing. Using net value per task = utility - q * ASR * (H/V), with H/V = 10 as an illustration (not a measured value), the numbers from the paper give:

Attacked share q	None	PromptGuard 2	AlignmentCheck	Combined
0.5%	0.4685	0.4663	0.4295	0.4259
1%	0.4597	0.4626	0.4280	0.4251
3%	0.4244	0.4475	0.4222	0.4216
5%	0.3891	0.4325	0.4164	0.4181
10%	0.3010	0.3948	0.4020	0.4093

The cheapest component, the 86M-parameter classifier, is the best choice across the middle range. The heavy LLM auditor only earns its utility cost once the attacked share is high or the harm per success is large. The full stack wins only at the top of this range.

This is a model, with the usual simplifications. It assumes the measured utility does not depend on prevalence, that a successful attack costs a flat H, and that the ASR on your traffic matches the benchmark. Each of those is wrong in some way. The point is not the exact crossover but the shape: there is no guardrail-independent "best" configuration, and a single headline ASR cannot rank them.

How many trials do you need to tell two guardrails apart?

Suppose you want to confirm the gap between AlignmentCheck alone (2.89%) and the combined stack (1.75%) on your own traffic. This is a two-proportion comparison, and the sample size for 80% power at a two-sided alpha of 0.05 is large:

Comparison	Injection trials per arm
17.63% vs 7.53% (none vs PromptGuard 2)	168
7.53% vs 2.89% (PromptGuard 2 vs AlignmentCheck)	359
2.89% vs 1.75% (AlignmentCheck vs combined)	2,736

Wilson 95% intervals tell the same story. At 629 injection cases per configuration (a figure I use only as an illustration of a mid-sized suite), the interval for 1.75% is roughly 1.0% to 3.1% and the interval for 2.89% is roughly 1.8% to 4.5%. They overlap heavily. A point estimate that looks like a 40% relative improvement is, at that sample size, within noise.

This is the same selection problem covered in the post on the threshold winner's curse: small differences between candidates, picked by argmax on a finite test set, regress to the mean on new data. If you compare four guardrails on a 300-case suite and ship the top one, expect disappointment in the ranking among the best two.

How do I compute this for my own agent?

The whole analysis fits in a short script. Replace the inputs with your own measured values.

import math
from statistics import NormalDist

Z = NormalDist().inv_cdf(0.975)

def wilson(p, n):
    d = 1 + Z*Z/n
    c = (p + Z*Z/(2*n)) / d
    h = Z * math.sqrt(p*(1-p)/n + Z*Z/(4*n*n)) / d
    return c - h, c + h

def n_per_arm(p1, p2, power=0.8):
    zb = NormalDist().inv_cdf(power)
    pb = (p1 + p2) / 2
    num = Z*math.sqrt(2*pb*(1-pb)) + zb*math.sqrt(p1*(1-p1) + p2*(1-p2))
    return (num / (p1 - p2)) ** 2

cfg = {  # (ASR, utility) from your own eval
    "none": (0.1763, 0.4773),
    "pg2":  (0.0753, 0.4701),
    "ac":   (0.0289, 0.4309),
    "both": (0.0175, 0.4268),
}

def net_value(q, h_over_v):
    return {k: u - q * a * h_over_v for k, (a, u) in cfg.items()}

print(n_per_arm(0.0289, 0.0175))   # about 2,700
print(net_value(0.05, 10))         # pick the argmax, then check its CI

Three practical notes on using it:

Estimate q from logs, not from the benchmark. Count how often tool output or retrieved content in your system contained instruction-like text aimed at the agent. Even a rough number (0.1%, 1%, 10%) moves the answer by an order of magnitude.
Estimate H/V per action class. A read-only search action and a payment action do not share a harm number. Per-class decisions beat a global one for the same reason that per-class break-even points differ.
Report intervals with every ASR. "1.75%" with no n is not a result you can compare to anything.

What other numbers should you not compare to ASR?

Newer guardrail papers report different metrics, and they do not convert. For example, SingGuard-NSFA (arXiv:2607.13081, submitted July 13, 2026) reports F1 of at least 94% on its purpose-built benchmark of more than 93,000 samples across 133 languages and 91.29% F1 on a 3,435-sample cross-source set, with detection at roughly 50 ms. F1 on a labeled set of safe and unsafe inputs measures classification quality. ASR on an agent benchmark measures the end-to-end outcome after the agent, tools and defense interact. A classifier with high F1 can leave a high ASR if the injection arrives through a channel it never sees, and a modest classifier can give a low ASR if the agent rarely follows injected instructions anyway.

When you read a guardrail announcement, write down for each number: what is the unit (input, step or task), what is the population (benign, attacked, or mixed), is the attacker static or adaptive, and what was the utility cost. A claim missing two of those four is a marketing figure, not an engineering input.

What is a sensible rollout given all this?

Start with the cheap classifier on every tool output, log its decisions, and do not block yet. You learn q from the flag rate and audit the false positives by hand.
Measure utility on your own tasks with the guardrail in shadow mode. A 5-point loss on a public benchmark may be 1 point or 15 points on your workload.
Gate the expensive auditor by action class. Run the LLM-based check only before actions where H/V is large (money movement, deletion, external messages). The break-even table says it is not worth paying for on read-only steps.
Re-evaluate against an adaptive attacker. Static-trace results are a floor on difficulty, not a ceiling. Red-team the stack with attacks written after seeing the guardrail's refusals.
Re-run the sample-size calculation before declaring a winner. If two options differ by under two points of ASR, you probably need thousands of trials, or you should pick on cost and latency instead.

FAQ

What does ASR mean in AgentDojo?
It is the fraction of injection-attacked task runs in which the agent carries out the attacker's goal. It is measured only on attacked runs, which is why it must be paired with the share of attacked tasks in your real traffic.

Does a 10x lower ASR mean 10x less risk?
Only if your traffic is the benchmark's attack mix and the attacker does not adapt. Risk is q * ASR * H summed over action classes. The 10x applies to the ASR term alone.

Why did utility drop when the guardrail was added?
Blocking or interrupting a step that looked suspicious also blocks some legitimate steps, and an LLM auditor can flag benign reasoning. In the paper's table, the combined stack lost 5.05 points of utility versus the baseline.

When is the heavy LLM auditor worth running?
When a successful attack is expensive relative to a completed task, or when a large share of your traffic is hostile. In the illustrative model above, with harm at 10x task value, the combined stack won only at 10% attacked share.

Can I trust a guardrail ranking from a few hundred test cases?
Not for adjacent options. Distinguishing 2.89% from 1.75% takes about 2,700 trials per arm at 80% power. Use intervals and treat close calls as ties.

DEV Community