How to detect fraud when it’s 0.17% of your data (and why 99.8% accuracy means nothing)

Pratyush Praveen — Thu, 15 Oct 2026 05:33:05 +0000

Flag the few transactions that are fraud among thousands that aren’t, without burying your review team in false alarms.

What you’ll build:

A model that scores every transaction by how likely it is to be fraud
An alert threshold chosen from what fraud and reviews cost
A plan for late labels and fraud that keeps changing

Tools: Python, pandas, scikit-learn, LightGBM

Step 1: Put a price on each mistake

A fraud model makes two kinds of mistakes, and they cost different amounts:

Missed fraud: you lose the transaction’s amount, plus fees and the customer’s trust.
False alarm: a person spends time reviewing a genuine transaction, or a real customer’s card is declined. A few dollars of staff time, and some annoyance.

Write both costs down, even roughly: for example, “a missed fraud costs its amount; a review costs $5.” You’ll use these numbers in Step 7 to set the alert threshold, which matters more than any modeling trick. Also note how many alerts your team can review per day. That limit is real, and the model has to live within it.

Step 2: Get labeled transactions

You need past transactions, each marked fraud or not. In your own data, the fraud labels usually come from chargebacks and confirmed cases. To learn, use the public credit card fraud dataset from Kaggle: 284,807 card transactions over two days, of which 492 are fraud, about 0.17%. Its columns are Time (seconds since the first transaction), Amount, Class (1 for fraud) and 28 anonymized features named V1 to V28.

Load the transactions

# pip install pandas scikit-learn lightgbm
import pandas as pd

tx = pd.read_csv("creditcard.csv").sort_values("Time")
print(f"{len(tx):,} transactions, {tx.Class.sum()} fraud ({tx.Class.mean():.2%})")

With your own data, you build the features yourself. Good ones compare a transaction with the customer’s history: the amount against their usual amount, how many transactions they made in the last hour, a new device or country. Compute each one only from what was known before the transaction, or you’ll leak the answer.

Step 3: Split by time, not at random

Fraud changes as fraudsters try new tricks, and the model will always be used on transactions that happen after its training data. Test it the same way: train on the earliest 60%, use the next 20% to make choices such as the threshold, and keep the latest 20% for the final test. A random split mixes future fraud patterns into training, which is a quiet form of data leakage.

Train, validation and test periods

features = [c for c in tx.columns if c not in ("Time", "Class")]
n = len(tx)
train, valid, test = tx.iloc[: int(n * 0.6)], tx.iloc[int(n * 0.6): int(n * 0.8)], tx.iloc[int(n * 0.8):]

X_train, y_train = train[features], train.Class
X_valid, y_valid = valid[features], valid.Class
X_test, y_test = test[features], test.Class
print("Fraud in each period:", train.Class.sum(), valid.Class.sum(), test.Class.sum())

Step 4: Don’t trust accuracy

With 0.17% fraud, a model that never flags anything is 99.8% accurate and completely useless. This is class imbalance at its most extreme. Measure two things instead: precision, the share of alerts that are really fraud, and recall, the share of all fraud you catch. Average precision sums up the trade-off between them in one number. A model that guesses at random scores about the fraud rate, roughly 0.002, so anything far above that is learning something.

The accuracy trap

from sklearn.metrics import accuracy_score

print("Accuracy when flagging nothing:", round(accuracy_score(y_test, [0] * len(y_test)), 4))

Step 5: Beat a simple baseline

Start with logistic regression, a fast, well-understood model. It’s the baseline every later model has to beat. Scaling the features first helps it train.

Logistic regression baseline

from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import average_precision_score

baseline = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))
baseline.fit(X_train, y_train)
print("Baseline average precision:", round(average_precision_score(y_test, baseline.predict_proba(X_test)[:, 1]), 3))

Step 6: Train a gradient-boosted model

Gradient-boosted trees usually do best on tables like this one: they find combinations such as “a large amount and an unusual pattern in V14” without being told. Early stopping on the validation period picks the number of trees, so you don’t have to guess.

LightGBM with early stopping

from lightgbm import LGBMClassifier, early_stopping

model = LGBMClassifier(n_estimators=1000, learning_rate=0.03, num_leaves=15, min_child_samples=50,
                       subsample=0.8, subsample_freq=1, colsample_bytree=0.8, verbose=-1)
model.fit(X_train, y_train, eval_set=[(X_valid, y_valid)], eval_metric="average_precision",
          callbacks=[early_stopping(50, verbose=False)])

p_valid = model.predict_proba(X_valid)[:, 1]
p_test = model.predict_proba(X_test)[:, 1]
print("LightGBM average precision:", round(average_precision_score(y_test, p_test), 3))

With only a few hundred fraud cases, scores jump around between runs and periods. Don’t celebrate a small gain over the baseline; look for a clear one, and check it holds on the test period too.

Step 7: Set the threshold from what fraud costs

The model gives each transaction a score from 0 to 1. The threshold decides which scores become alerts. Instead of the default 0.5, try every threshold on the validation period and keep the one with the lowest total cost, using the numbers from Step 1. Then check it once on the test period.

Choose the cheapest threshold

import numpy as np

REVIEW_COST = 5   # dollars of staff time to check one alert

def total_cost(threshold, y, p, amount):
    flagged = p >= threshold
    missed_fraud = amount[~flagged & (y == 1)].sum()
    return missed_fraud + flagged.sum() * REVIEW_COST

thresholds = np.linspace(0.01, 0.99, 99)
costs = [total_cost(t, y_valid.values, p_valid, valid.Amount.values) for t in thresholds]
best = thresholds[int(np.argmin(costs))]

flagged, fraud = p_test >= best, y_test.values == 1
print(f"Threshold {best:.2f}: {flagged.sum()} alerts, {(flagged & fraud).sum()} of {fraud.sum()} frauds caught")
print(f"Fraud value stopped: ${test.Amount.values[flagged & fraud].sum():,.0f} of ${test.Amount.values[fraud].sum():,.0f}")

Tip: If the cheapest threshold produces more alerts a day than your team can review, raise it until the alerts fit. A threshold nobody can keep up with is worse than a stricter one that gets every alert checked.

Step 8: Score transactions as they happen

Fraud checks usually run while the payment is happening, so the model has to answer in milliseconds; this is a latency budget. A LightGBM model scores one transaction in well under a millisecond, so the slow part is fetching the features, such as the customer’s recent history. Keep those precomputed and up to date. Save the model together with its feature list and threshold, so the service always uses all three together.

Save everything the service needs

import joblib

joblib.dump({"model": model, "features": features, "threshold": float(best)}, "fraud_model.joblib")

Keep your simple rules too, such as blocking a card used in two countries within an hour. Rules catch known patterns instantly and are easy to explain; the model catches the patterns nobody wrote a rule for. Serving it is the same as any small model API; the spam filter guide shows one with FastAPI.

Step 9: Plan for late labels and changing fraud

Fraud labels arrive late: a chargeback can come weeks after the transaction. So the latest few weeks of data look cleaner than they are. When you retrain, leave out the most recent month, whose labels aren’t final yet.

Fraudsters also adapt once a model starts stopping them. That’s data drift, and it’s faster here than in most problems. Track three numbers every week: the alert rate, the share of reviewed alerts that were really fraud, and the fraud found later that the model missed. Retrain on a regular schedule, monthly for most teams, and compare the new model with the current one on the latest settled period before switching.

Common pitfalls

Reporting accuracy. With rare fraud, it’s always close to 100% and says nothing.
Splitting at random instead of by time, which lets the model learn from future fraud.
Building features with information from after the transaction, such as “was this card later reported stolen.”
Keeping the default 0.5 threshold instead of one chosen from costs and review capacity.
Retraining on recent data whose fraud labels haven’t arrived yet.

This guide comes from Hello Model, a free planner that turns a machine learning idea into a step-by-step plan, with the right model, code and cloud costs. The original guide has a Colab notebook with all the code, and a short quiz to check what you learned.
``

DEV Community: Pratyush Praveen