A device-sharing signal looked perfect on my test data and collapsed on the real thing. So did my model's best score. Both failures pointed at the same lesson.
The alert sitting at the top of my fraud queue read: device shared by 4,661 cards. It was a legitimate purchase, and it was the best thing that happened in the whole project.
That alert was wrong because a chart I trusted was wrong, and the chart was wrong because I had, without meaning to, built the test data that made it look right. This is a story about two results that flattered me, how real data destroyed both, and why the results that survive contact with reality are the only ones worth putting your name on.
What I was building
A transaction fraud scoring pipeline, built around the constraint that actually shapes fraud operations: a review team can only look at a fixed number of alerts a day. You do not get to flag everything. So the goal is not "detect fraud" in the abstract. It is to prevent the most fraud value inside a fixed review budget, keep false declines low, and notice drift before precision quietly rots. Ranking under a budget, not raw accuracy, is the product. A model that catches most fraud but needs a human to review half of all transactions is worthless.
I built it on synthetic data, then took it to the real thing: the IEEE-CIS Fraud Detection dataset, 590,540 card-not-present transactions, a 3.5% fraud rate, and about $3.08M in fraud value. IEEE is a good stress test because it withholds what a toy pipeline leans on. There is no account id and no geolocation. You reconstruct identity from anonymised card attributes yourself, which is where the interesting work is, and where I got caught.
The signal that looked perfect
Organised fraud reuses infrastructure. One device drives many "different" cards. So a natural hypothesis: count the distinct cards seen on a single device, and a device touching a dozen cards is not a coincidence, it is a ring.
On my test data this worked beautifully. I bucketed transactions by how many cards had touched the device, and the fraud rate climbed cleanly with sharing, all the way to 100% at the top. It was the kind of chart you screenshot for a slide. Sharing on the x-axis, fraud rate marching up the y-axis, a tidy story about catching rings before the model even weighs in.
I believed it. I should not have, because I had built the data that guaranteed it. Every simulated device in my mock was unique, so the only devices touching many cards were the rings I had injected myself. The chart was real. It was also a mirror. It reflected my assumption back at me and I mistook the reflection for evidence.
The collapse
Then I ran it on the 590,000 real transactions, and the same chart came back flat. Fraud rate barely moved as sharing increased, and the biggest bucket, devices shared by seven or more cards, held nearly 18,000 transactions at only 8.6% fraud. That is a whisper above the 3.5% base. Whatever this was, it was not a ring detector.
And the queue produced that alert. Device shared by 4,661 cards, on a transaction that was not fraud. Here is the actual row, sitting near the top of the queue by expected loss:
TransactionID card1 amount risk_score expected_loss reason_codes is_fraud
3513661 6021 1600.00 0.417203 667.525150 device shared by 4661+ cards 0
The cause was obvious once I looked, and a little embarrassing. In the real data the device field is not a fingerprint. Its most common values are Windows, iOS Device, MacOS, Trident/7.0. Those are operating system and browser families, shared by enormous numbers of ordinary people. "Many cards on this device" mostly meant "many people use Windows." I was counting popularity and calling it fraud.
The part that stung: I had already seen this exact failure and guarded against it, just not here. Earlier I had ruled out "cards per email domain" and "cards per billing region" as ring signals, because everyone uses a handful of domains and lives in a handful of regions, so those counts are dominated by how common a value is, not by fraud. I knew the rule. I applied it to email and address. I missed it for the device, because my mock had quietly made every device unique, so the trap never sprang in testing.
That is the first lesson, and it is bigger than one field: a low-cardinality, high-frequency value will masquerade as a strong signal in any count of shared entities, and test data you built yourself will hide it if you built it too clean. The mock did not just fail to catch the bug. The mock made me confident in it.
Fixing it without lying
Two things needed fixing: the signal and the explanation.
For the signal, a device string alone is too coarse, so I built a more specific fingerprint by combining the device with the browser and the screen resolution, which splits one giant "Windows" bucket into many specific ones. Then, instead of only handing the model a raw shared-card count, I also gave it how common each fingerprint is overall, so the model can separate a rare fingerprint shared by five cards, which is interesting, from a popular one shared by five thousand, which is not. Both are structural counts. Neither touches the label, so neither can leak.
For the explanation, I gated the human-facing parts. The reason code and the deterministic ring rule now only fire when a fingerprint is specific enough to mean something. After that, the queue can no longer tell you a Windows machine is shared by four thousand cards. That sentence cannot be produced anymore.
The honest result: once the generic families are gated out, sharing does correlate with fraud, roughly 9% at two cards and 15% at four to six, against the 3.5% base. But it is modest and non-monotonic, and the largest specific bucket falls back toward the base rate. It is a useful investigator lens and a decent cold-start rule for the window before the model has evidence. It is not the clean 100% detector my test data promised. Demoting it from headline result to tested hypothesis was the most senior move in the project.
The same lesson, wearing a different disguise
While I was being honest about the device signal, the model handed me the same lesson in a different form.
IEEE has hundreds of engineered features, and strong solutions lean on them, so I fed the model all of them: the full Vesta set, the counting and timedelta columns, the identity fields, frequency encodings, and a client key anchored on a registration-date signal. Feature count went from 52 to about 460.
PR-AUC went from 0.444 to 0.442.
Adding the dataset's best features moved the top of the queue but not the curve. So I gave the model more capacity, deeper trees, more iterations, which took it to 0.468. More feature engineering took it to 0.468 again. Four configurations, and the ceiling sat at about 0.47.
| Configuration | Features | PR-AUC |
|---|---|---|
| Thin baseline | 52 | 0.444 |
| All Vesta + timedelta + identity | 451 | 0.442 |
| Higher-capacity model | 451 | 0.464 |
| D1-anchored client key + prior-amount | 459 | 0.468 |
When your best features do not move the curve and neither does more model, the ceiling is not in your features or your learner. It is in the method. The gap between 0.47 and the scores on the public leaderboard is bridged almost entirely by one move I refused to make, and it is the same trap as the mock, wearing a different disguise.
The move is aggregating features across the entire dataset. Compute the mean transaction amount for a client over all their rows, including future ones, and attach it to today's transaction. It reliably lifts the offline score. It is also feature lookahead: in production you do not have the client's future transactions when you score the current one, so a feature built from them is a promise you cannot keep. It is a flattering number you construct by peeking at data you will not have. Exactly like the mock, it produces a result that looks like evidence and is really a reflection of what you already assumed you would get. Every aggregate in my pipeline is computed strictly from prior rows, current transaction excluded. Weaker on paper, honest at decision time.
So 0.47 is not a disappointing number. It is what a single calibrated model can do on this data using only information you would actually have when the transaction arrives. I would rather ship that and explain it than post a bigger number I could not deploy or defend in a review.
What actually ships value
If the score is not the headline, the operating result is, because it maps to what the team does.
The queue ranks every transaction by expected loss, probability times amount, so it prioritises the cases that cost the most, not just the ones most likely to be fraud. At the cost-chosen operating point on held-out data, the model recovers $380,070 of $609,934 in fraud value, about 62%, at the review load the cost model picked. The value-versus-budget curve shows the rest of the trade-off: for whatever review capacity a team has, how much fraud value it recovers.
That curve is the real deliverable. It lets someone with a fixed headcount choose their point on it deliberately, instead of accepting whatever threshold a model happens to output. Probabilities are calibrated too, which is what makes multiplying a score by an amount mean anything in the first place.
What I would take from this
Evidence that flatters you is usually evidence you built. The clean chart came from a mock I had shaped, without noticing, to produce it. The bigger score comes from aggregation that peeks at the future. In both cases the result looked like proof and was really a reflection of my own assumptions. Real data, and production, are the only tests that do not tell you what you want to hear.
A count of shared entities lies when the entity is low-cardinality. OS families, email domains, regions, anything a large share of your population has in common, will dominate the count and look like signal. Guard for it, or your reason codes will confidently accuse a Windows laptop.
Leakage-safe is a discipline, and some famous tricks break it. Strictly-before, current row excluded, no aggregation over data you would not have at decision time. The move that lifts your offline score by using the future is the move that fails silently in production.
Report the honest number and the operating result, not the vanity metric. A PR-AUC out of context invites a snap judgment. "Recovers 62% of fraud value at this review budget, here is the curve" invites a conversation with whoever owns the budget. The second one is the one that matters.
The code, the leakage guards, the investigation queries, and the full device-fingerprint write-up are in the repo: github.com/gbadedata/transaction-fraud-scoring.
The best result in the whole project was a wrong alert about a Windows laptop, because it was the moment the data stopped agreeing with me. That is usually the moment something true starts.
Top comments (0)