This is the second of a pair. The first was about card fraud, and its lesson was that a chart I was proud of turned out to be a mirror. It looked like my model had found fraud rings, and what it had really found was my own assumptions handed back to me. This post is about money laundering, and it opens on a number I was briefly pleased with and then spent a while trying to explain away: 0.99.
That was the precision-recall AUC of a mule-detection model on a labelled dataset, with per-typology recall between 93 and 100 percent. If you have spent any time near anti-money-laundering work, you already know that a 0.99 on that problem is not a triumph. It is a symptom. Real laundering detection does not look like that, and when your numbers say it does, the thing to distrust is your numbers.
Here is what the model was trying to do, why the 0.99 was fake, and what the real data did to it.
Why laundering is a graph problem
Laundering is rare. In the data I used it runs to about 0.18 percent of transfers, and in the wild the uncommon variants are far rarer than that. At those rates, scoring transactions one at a time is close to pointless. A classifier that flags nothing is right 99.8 percent of the time, and whatever signal a single transfer carries is drowned by the base rate.
The signal is not in the transfer. It is in the shape of the money's path. Mules move funds through networks, and those networks take recognisable forms: many accounts feeding one (fan-in), one account feeding many (fan-out), money walked hop by hop along a chain, closed cycles that return to their start, and rapid pass-through where funds land in an account and leave again within hours. Lift the problem off the individual transaction and onto the structure of the account graph, and the signal concentrates where the typologies live.
So I built the account graph, computed features over it, scored transfers with those features, and ranked accounts for an investigator to review. The dataset was IBM's synthetic anti-money-laundering set, which ships with a per-transaction laundering flag and a separate file naming the typology of each laundering attempt. No bank releases real labelled mule data, so a synthetic set is the firm ceiling on what any of this can claim, and I will come back to exactly how that ceiling bit me.
The one discipline I carried over
The card-fraud post was, at bottom, about leakage: a feature that quietly peeks at the future inflates every number downstream, and none of it survives contact with reality. So this time I built the leakage guard in first.
The features split into two kinds, and keeping them apart is the whole discipline. Streaming features describe an account's history strictly before the current transfer: how many distinct counterparties it has had, how much it has moved, its pass-through ratio, and the burst counts, meaning how many counterparties arrived in a short trailing window. These use no future information, so they are what a real-time monitor could score on. A feature computed over an account's later transfers is the graph version of lookahead: knowledge you would not have at the moment you have to decide.
The second kind, retrospective features, describe an account across the whole window for an investigator working a case after the fact. They are useful, and they are never model inputs, because letting the future into training is how you build a model that works in a notebook and nowhere else.
The guard is a test, not a comment in the code. Compute the streaming features on the first half of the data, compute them again on the full data, and the early rows have to come out identical. If a later transfer changes an earlier row, information is leaking backward, and the test fails.
full, cols = prior_features(df)
for k in range(1, len(df)):
prefix, _ = prior_features(df.iloc[:k])
assert np.allclose(full[cols].iloc[:k].to_numpy(),
prefix[cols].to_numpy())
On top of those features sat a gradient-boosted model trained on a strict time split, its transaction scores aggregated to a per-account risk, and a queue that ranked accounts by expected laundered value with plain-language reasons attached, so an alert could explain itself.
All of it passed. The leakage test was green. And the model scored 0.99.
The number that was measuring the wrong thing
I had built a schema-faithful mock early on so the whole pipeline could run without the download, and I had tuned and tested against it. On that mock the model was close to perfect. Then the real file finished downloading, I pointed the loader at it, changed nothing else, and everything fell through the floor.
| synthetic mock | real IBM data | |
|---|---|---|
| transactions | 63,000 | 5,080,000 |
| laundering rate | 2.4% | 0.18% |
| transaction PR-AUC | 0.99 | 0.037 |
| per-typology recall | 93 to 100% | 27 to 51% |
| laundered value in top 2% | ~99% | 54.5% |
Not 0.99. 0.037. About twenty times better than random at that base rate, which is real signal, but a different universe from what the mock had promised. The same code, the same features, the same leakage guard, and the number fell by a factor of twenty-five the instant it met data I had not generated myself.
The mock had not been a test. It had been a mirror, the same one from the first post wearing different clothes. I had built the synthetic data to contain the patterns my model looked for, and I had left out the one thing that makes the real problem hard. The 0.99 was not a measurement of the model. It was a measurement of the mock, reflecting my own design decisions back at me and calling them accuracy.
Every signal that catches a mule also describes a bank
Here is what the mock left out, and what the real data is full of: legitimate high-throughput accounts.
Payment processors. Exchanges. Settlement accounts. Aggregators. Accounts that receive from hundreds of counterparties in a day and send to thousands, moving enormous volume through the network as their ordinary, entirely legal business. My mock had none of them. Every account in it was either a normal low-volume account or a mule, so the model never had to tell a mule apart from anything except a quiet legitimate account, which is easy.
Real data does not grant that. One of my investigation queries ranks accounts by fan-in, the number of distinct senders paying into them within a short window. Here is the top of that list on the real data:
account peak_senders_3d total_in laundering_share
70-100428660 545 638329 0.000
70-1004286A8 328 372750 0.000
21611-8051A3FA0 93 22231920 0.000
4726-808FAF7C0 88 1742220 0.000
Five hundred and forty-five distinct senders in three days, and a laundering share of zero. It is completely legitimate. It is also indistinguishable, by every structural signal I had built, from a collector at the center of a fan-in ring. High burst, high fan-in, balanced pass-through: a payment processor lights up all of them, because gathering money from many sources and moving it on is what a processor does and what a mule does. The structure is the same. The intent is not, and intent is not in the graph.
That is the confound, and it is the entire difficulty of the problem. On synthetic data my structural features looked like a laundering detector. On real data they are closer to a high-activity detector, and high activity is mostly legitimate.
The queue that filled up with banks
Nowhere was this clearer than the investigation queue. I had ranked accounts by expected laundered value, which is risk multiplied by the money the account moved, the same value-first logic that worked in the card-fraud project. On real data the top of the queue was dominated by legitimate accounts moving hundreds of billions of dollars, because raw value swamps everything. Two of the top ten were laundering. The other eight were banks.
The fix came from asking what a mule does that a processor does not, and the answer is timing. A processor is busy all year. A mule is busy for a few days. Its burst is not just large, it is concentrated: a large share of everything the account ever does falls inside one short window. So I added burst concentration, the peak burst over the account's total activity, near one for an account that fired in a single spike and near zero for one that is always active, and folded it into the ranking alongside a log of the value so a handful of mega-accounts could not dominate on size alone.
Here is the top of the queue after that change. The reasons column is generated from the structural signals, so each alert says why it is there:
concentration laundering reasons
0.985 1 fans in: 56 inflows; fans out: 3623 outflows; u-turn match 1.00
0.985 1 fans in: 539 inflows; fans out: 36075 outflows; u-turn match 1.00
1.000 0 elevated model score without a single clear structural pattern
1.000 1 fans in: 26 inflows in a short window
That lifted the laundering share at the top of the queue from two in ten to six in ten. The accounts it surfaces now include genuine laundering hubs with tens of thousands of transfers packed into a short burst, while the always-on processors drop away. It is a real improvement, and I want to be precise about what it is not. It is a first step, not a solution. The false positives that remain are concentrated accounts the model scored high, and concentration alone cannot separate a mule from a small legitimate account that simply had a busy week.
What the real number buys you
Stripped of the fantasy 0.99, the model still does something worth having. The precision-recall AUC of 0.037 is the wrong lens, because precision and recall on a 0.18 percent base rate punish you for the base rate. The lens that matters to a review team with a fixed number of alerts they can work is value. Rank every transfer by score, walk down the list, and by the time you have reviewed the top two percent you have recovered 54.5 percent of the laundered value and 38 percent of the laundering transfers.
That is a usable operating point, and it is the number I would put in front of an investigator, not the AUC.
Per typology the picture is even and modest, which is what real detection looks like. Recall across the labelled typologies runs from 27 percent on fan-out to 51 percent on fan-in, with cycles, stacks, scatter-gather and the rest in between.
Nothing is solved. Nothing is missed. The typologies whose activity is concentrated in time do a little better than the ones that blend into ordinary volume, which is exactly what you would predict from the confound.
Getting past this ceiling is not a matter of a better classifier on the same features. It needs peer-group baselines, so an account's throughput is judged against accounts of its own type rather than against everyone, and it needs account history, so a sudden burst on a young account counts for more than steady volume on an old one. It needs to score entities and networks, not lone transfers. That is a larger project than this one, and I would rather ship the version that is clear about its own ceiling than the one that hides behind a 0.99.
The same lesson, twice
Both of these projects ended in the same place, and I think it is the place worth ending in. In the first, the trap was temporal: a feature that peeked at the future made the model look like it had learned something it had not. This time I avoided that one. The leakage test was green from the start. And a subtler version of the same self-deception got me anyway, because a synthetic dataset is a kind of lookahead too. It lets you see a world you designed to be solvable and mistake it for the world.
The discipline that catches both is the same. When a number makes you look good, that is the moment to distrust it and go looking for the thing that breaks it. In the card-fraud project the thing that broke the chart was real transactions. Here it was a legitimate account moving a billion dollars that trips every mule signal there is. Both were sitting in the data the whole time. The only question was whether I would go find them before someone else did.
The code, the leakage test, and the full write-up of the real numbers are on GitHub.
Top comments (0)