Coinbase's Fraud Agent Never Decides Anything. That's Why It Works.
On October 7–8, 2026, Coinbase published Owning Intelligence, a three-part engineering series on the fraud agent guarding Onramp. The headline every outlet ran: a fine-tuned Qwen3.5-9B beat Opus 4.5. The architecture underneath the headline matters more:
- The agent layers onto Onramp's existing ML models and rules engine, and reviews transaction sequences, not single payments.
- Its output is narrow by design: a risk level, and nothing more. Application code applies the transaction decision.
- Its authority is asymmetric: it can block a transaction that passed every upstream check — it cannot reverse a block.
In the online A/B test, adding the agent produced 30% fewer fraudulent transactions and 22% less fraud value.
The benchmark, and the warning inside it
Coinbase evaluated models on a fixed benchmark of 16,140 transactions (7,293 users, 813 confirmed fraud). Part 2's finding deserves more weight than the fine-tune headline: each newer Opus, Sonnet, and GPT release scored lower on recall and F1 than the version before it. One newer GPT release improved precision while catching less fraud. General capability gains did not transfer to the specialized judgment — they decayed.
If your payment gate rents its judgment from a generalist API, your calibration is a side effect of someone else's release cycle.
Owning the decider
Part 3: Coinbase post-trained Qwen3.5-9B with reinforcement learning (SkyRL on Anyscale) on its proprietary fraud data, served via Ray Serve LLM + vLLM. Results against Opus 4.5:
| Metric | Tuned Qwen3.5-9B vs Opus 4.5 |
|---|---|
| Precision, recall, F1, dollar-weighted recall | Ahead on all four — margins 7.5–35.4 points (F1 +9.6) |
| Median latency | 0.683s vs 1.515s (−55%) |
| Training cost | Under $100, one NVIDIA RTX PRO 6000 96GB GPU |
Honest caveat: the benchmark is proprietary and the tests company-run — no outsider can reproduce them. A well-documented vendor claim, not a public benchmark.
Tested live today: a generic gate against Onramp-shaped instructions
We ran two Onramp-shaped instructions against a live decision gate (/api/harness/decide, bands 0.80/0.50), October 12, 2026:
- "40-day account, three small buys, now a $4,800 first-time low-liquidity-token purchase from a brand-new bank account at 3 a.m. after two failed auth attempts. Upstream checks passed." → 0.65, advisory — hold for human review
- "Two-year verified account, consistent device and bank, 60 prior buys, routine $120 USDC repeat at the usual hour, everything matching the established pattern." → 0.65, advisory — hold for human review
The blunt heuristic held both: safely, and uselessly for the second. It cannot discriminate a patterned fraud attempt from a routine repeat purchase. That is exactly the gap Coinbase's $100 fine-tune closes — a gate is only as good as its decider's calibration.
Build the Coinbase shape
- Separate scorer from decider — the model returns a number; code owns the decision.
- Make authority asymmetric — the model can block, never unblock.
- Apply bands in code — ≥0.80 auto-act, 0.50–0.79 hold for a named human, <0.50 escalate/block + log.
- Evaluate on your own labeled outcomes, every model version, before it touches the gate.
- Fine-tune when the generalists plateau — Coinbase's receipt: one GPU, your data, under $100.
Full piece with the dated receipts table, the exact curl, and Claim Receipts: https://scriptmasterlabs.com/coinbase-ai-fraud-agent
Top comments (0)