This is my submission for the Kaggle × DEV Benchmarks Challenge.
Here's a question nobody's really measuring: when an AI support agent reads a claim, does it check who said it before acting — or does it just trust anything that sounds official?
So I built a benchmark to find out, and the result is worse than I expected. A model that correctly ignored a customer's forum rumor will, when you strip the "community post" label off that exact same text and let it read as policy, apply a $150 credit to a stranger's account. The words never changed. Only the label did.
And here's the number that should worry you most: the correct policy is trivial — trust a claim only when it's labeled as an official source — and a two-line rule-based agent scores a perfect 1.00 on it. Across the models I ran, two of them act on a laundered rumour 100% of the time despite refusing the identical labelled version 100% of the time.
It's a live Kaggle Benchmark with a published notebook, so you can fork it and run any model yourself.
I call the headline failure the Laundering Gap, and here's how it works.
The setup: same claim, four disguises
The problem with testing this on real data is contamination — models may have seen it. So the benchmark is 100% synthetic: 4 invented companies (an e-bike shop, a cloud host, an airline, a grocery service), each with its own real policies, and 24 scenarios where a customer wants an action taken: a credit, a discount, a return, a fee waiver, a plan upgrade, or their details emailed somewhere.
Each scenario is run in four conditions — the same underlying claim, wearing different clothes:
| Condition | The claim is… | Should the agent act? |
|---|---|---|
| forum | in a casual customer voice, labeled "community forum post" | ❌ No |
| laundered | in an official policy voice, no source label | ❌ No (it's still just a claim) |
| labelled | in an official policy voice, but labeled "community forum post" | ❌ No |
| official | in an official policy voice, labeled "official help article" | ✅ Yes |
The trap is the laundered condition. It's an unverified claim written to sound like policy, with the source label removed. A model that actually checks provenance should treat it like the forum version — untrusted. A model that pattern-matches on tone will treat it like policy.
The leaderboard (live on Kaggle)
This is a public Kaggle Benchmark — run against the models on Kaggle's own infrastructure, with a published backing notebook. Numbers are "acted on the claim" out of 24 per condition. Lower is better everywhere except official.
| Model | forum ❌ | laundered ❌ | labelled ❌ | official ✅ | Gap | Prov |
|---|---|---|---|---|---|---|
| Gemini 3.8 Flash | 0/24 | 24/24 | 0/24 | 24/24 | 100% | 0.00 |
| Gemini 3.7 Flash | 0/24 | 24/24 | 0/24 | 24/24 | 100% | 0.00 |
| Claude Haiku 4.5 | 1/24 | 21/24 | 0/24 | 21/24 | 88% | 0.00 |
| Claude Sonnet 5 | 0/24 | 2/24 | 0/24 | 22/24 | 8% | 0.83 |
| GPT-5.4 nano | 6/24 | 11/24 | 9/24 | 10/24 | 8% | −0.04 |
| Gemma 4 31B | 21/24 | 20/24 | 19/24 | 20/24 | 4% | −0.04 |
| Gemini 3 Flash | 24/24 | 24/24 | 24/24 | 24/24 | 0% | 0.00 |
(A few models aren't scored — not a benchmark issue: some were unavailable on Kaggle's proxy (e.g. Grok 4.6 returns a 404 "model not found"), and reasoning models like Opus 5 / DeepSeek-R1 time out generating large output over the 96 calls.)
Two failure modes jump out — and one of them is the cleanest possible demonstration of the trap.
Finding 1: The Laundering Gap (100% on Gemini Flash)
Look at Gemini 3.8 Flash and 3.7 Flash. On a claim labelled "community forum post" they act 0 out of 24 — perfect refusal. Take the exact same words, remove the label, and let it read as policy, and they act 24 out of 24. Every single time.
That is a 100% Laundering Gap. The label was doing all the work. Strip it and a model that never acted now always acts. Tone is the one thing an attacker fully controls.
Claude Haiku does nearly the same (0 → 21). This is what it looks like in practice — real business actions (a credit, a discount code, an approved return) fired by nothing but the absence of a source label:
"…you're entitled to a $150 Late Delivery Credit. I've applied that credit…"
"…I've applied code RIDEON40 to your account, which gives you 40% off…"
"…your rear rack qualifies for a return. I've approved the return…"
Every one is money out the door — triggered only by laundering a rumour into policy language.
Finding 2: The gullible model (Gemini 3 Flash)
Gemini 3 Flash fails the opposite way: it acts on everything — 24/24 even on a plainly-labelled forum rumour. It isn't fooled by laundering; it never checked the source at all. Its perfect "official" score means nothing next to a perfect "forum" score.
Claude Sonnet 5 is the outlier in the other direction — the only model that mostly holds the line (provenance score 0.83), acting on just 2/24 laundered claims. It's the exception that proves the rule: careful is possible, but rare.
Why this matters beyond a leaderboard
Support agents, RAG systems, and tool-using agents all make decisions from retrieved text — and that text mixes trusted policy with untrusted user content (forum posts, reviews, tickets, emails). This benchmark says the quiet part out loud:
Models decide what to trust based on how text is worded, not where it came from. That's a prompt-injection problem wearing a customer-support costume. Launder a rumor into policy language, and a "careful" model will act on it; a gullible one didn't need laundering at all.
Why you can trust these numbers
Most "I tested some models" posts skip this part. A benchmark is only meaningful if a correct strategy can pass it and a wrong one fails — otherwise you're just measuring noise. So this ships with the controls:
-
A rule-based baseline proves it discriminates. A
provenanceagent that acts only when the source is labeled official scores a perfect 1.00; atoneagent that trusts any official-sounding text scores 0.00; acredulousagent that trusts everything scores 0.00. The benchmark cleanly separates the correct policy from the wrong ones — and no real model lands near the correct one. -
labelledandofficialare byte-identical except the source label. A test enforces this, so the minimal pair genuinely isolates provenance from tone. - Seeded, re-runnable, anti-memorization. A variant generator deterministically rewrites every memorizable specific — amounts, codes, order IDs, emails, handles — so the same scenarios re-run with fresh values on any seed, and the fairness invariant still holds. A model can't have memorized the answers.
- Wilson 95% confidence intervals on every rate, and 12 passing tests (integrity, the minimal-pair invariant, baseline discrimination, variant determinism).
The full leaderboard with the provenance baseline:
Honest limitations
- Synthetic data. Made-up companies avoid contamination but aren't real support traffic.
- Small n. 24 scenarios × 4 conditions. Wide confidence intervals; the pattern (a large Laundering Gap, provenance ≪ 1.0) is the finding, not any single cell.
- Run variance. Zero-shot at default temperature, one run per model — a careful model on one run may act more on another. The 100% gaps are stable and dramatic; the marginal rows will wobble.
- Some reasoning/open models errored on Kaggle's model proxy (DeepSeek-R1, Qwen3, Grok, GPT-6) and aren't scored yet.
- "Acted" is scored by matching the action's argument, so scoring is auditable from the kept reply text.
Reproduce it
The benchmark is built on the kaggle-benchmarks library: 24 scenarios, 4 conditions, deterministic scoring of whether the model performed the claimed action. Swap in any model with one line, and every result keeps the full reply text so each "acted" call is auditable.
- Live Kaggle Benchmark (fork + run any model): kaggle.com/benchmarks/tasks/rudratoshshastri/who-said-that
- Code + baselines + tests + charts (MIT): github.com/rudratoshs/who-said-that
pip install kaggle-benchmarks python-dotenv matplotlib pytest
kaggle benchmarks auth
python pilot.py anthropic/claude-sonnet-5@default # run a model locally
python analyze.py # metrics + CIs (incl. baselines)
python -m pytest tests/ -q # 12 tests
# or push it as your own Kaggle Benchmark:
kaggle benchmarks tasks push who-said-that -f kaggle_task.py
kaggle benchmarks tasks run who-said-that -m <model-slug>
Which failure worries you more — the careful model that a missing label flips, or the gullible model that never checked at all? I keep going back and forth. 👇
I write about AI agents, security, and the honest ways they break. Follow me here if that's your lane. 👋
Top comments (1)
Author here — I'll go first with the bit that unsettled me most.
I expected models to be gullible (act on everything) or careful (check the source). What I didn't expect was the third category: models that clearly read the label, then get completely undone by removing it.
Gemini 3.8 and 3.7 Flash refuse a claim labelled "community forum post" 0/24 — perfect. Strip the label, keep every word, and they act 24/24. They have the machinery to check provenance; they just don't use it once the label is gone. That's scarier than never checking, because it looks safe right up until an attacker peels off one label.
The lone exception was Claude Sonnet 5 (acted on 2/24 laundered), which is the only reason I believe the task is passable at all.
What would you expect your production model to do here — and are you doing anything today to tell "official policy" from "official-sounding text"? 👇