We measured our own fraud detectors before trusting them. Here is what we got wrong.

AI agents on Base now hold their own wallets and pay for things by themselves. We keep an independent ledger of that money: every ERC-8004 registration on Base followed to the wallet bound to it, every payment's receipt read, routing hops excluded. On top of the ledger sit a few anomaly detectors: a wallet drained in one payment, a retry loop hammering one seller, one address minting hundreds of identities in an hour, an endpoint that never answers.

Until last week, none of those detectors had ever been run against data with a known answer. They had fired on the real chain, and we had re-checked some fires by hand. That is a claim, not a measurement.

What we built

IBM's AMLSim is the standard tool for generating synthetic bank transactions with money-laundering patterns planted inside, so that detection rules can be scored against known answers. We adapted it to AI-agent wallets: x402 micro-payments at a few cents, job-escrow deposits, EIP-7702 delegation, address poisoning, fleets funded by one treasury. The generator plants the shapes our rules are written to catch, and the near-misses they are written to ignore: a drain below the dollar floor, a storm spread over 25 minutes, a burst spread over three hours, a dead endpoint that answered once.

Then the validator runs our production detectors, imported unchanged, over the synthetic tables and scores each rule: planted, caught, missed, false alarms. It runs every Sunday now. The generator, validator, configurations and reports are open source: amlsim-agentic (Apache-2.0, IBM credited).

What it found

The retry-storm rule under-counted four to one. The rule grouped payments by payer, payee and calendar hour, then asked whether fifty fell inside ten minutes. A storm of 56 payments starting at minute 56 is split between two hours and neither half counts. The synthetic near-miss "a storm that crosses an hour boundary" was silent in 8 of 8 cases. On the live ledger the corrected rule finds 45 storm detections from 28 payer–seller pairs; the old one had reported 10 from 7, and we had published that for a month.

Mint bursts were split at the hour mark. Same bucket, same fault: 83 bursting addresses reported, 88 in the ledger, and one run logged as several rows.

Every list was silently capped at 100 rows. On the ten-times-larger synthetic run, the cap hid 32 of 100 planted drains and read, until we found it, as a recall of 68%.

The drain rule cannot tell a theft from a purchase, but one question can. The rule (a payment at least five times the agent's previous largest, at least $250, to an address it had never paid) caught 25 of 25 planted drains and also flagged 11 one-off purchases. Asking "how much of what the wallet held did the payment take?" separated them with no error: a drain empties the wallet, a purchase leaves most of it. On the live ledger, 39 of the 47 drain-shaped payments since February took 80 to 100 percent, 23 of them the whole balance.

The chain shows no laundering rings. AMLSim's eight layering shapes (cycles, scatter-gather, fan-in, fan-out and so on) were planted and our rules, which had no layering detectors, ignored all of them, as expected. We wrote the detectors and ran them over 314,826 real agent-to-agent payments (the 10 October backtest): no cycle, no scatter-gather, and ten same-amount fan-outs, each of which is one treasury funding a fleet of its own registrations.

What we changed on 11 October

• Retry storms and mint bursts are read as runs with a peak (definitions retry_storm.v2, mint_burst.v2). The rows logged by the old rules stay in the log as history; the runs they missed were added, dated.
• Every drain detection carries the balance reading from the chain at the block before. It is evidence beside the rule for a month, then a filter.
• A new public kind, layering_fan, for the fleet-funding shape. A shape, never an incident.
• Lists warn when they are full.
• Each rule's measured record sits on the detections page, with the sentence that matters: synthetic data measures the rules, not the chain.

What we do not claim

Precision on synthetic shapes says how a rule behaves on those shapes and nothing about how often they occur on the chain. A shape the generator did not think of is not measured at all. One test that worked on synthetic data, a slow drain over 72 hours, fails on the live ledger because real agent hot wallets run near zero, so it is not published. The full account, with the backtest over the whole ledger, is Research Note 03, "Measured before trusted".

If you run detectors over any payment graph, the generator is yours to use; the typologies are a JSON file and the validator imports your rules the same way it imports ours.