DEV Community

Avijit Aditya.
Avijit Aditya.

Posted on

PaySafe India — Real-Time UPI Fraud Detection with Open-Source ML (No Cloud Required)

#hacktoberfest #opensource #python #machinelearning

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 4: Ghost in the Machine

What I Built

PaySafe India is an end-to-end UPI transaction fraud detection system powered entirely by open-source machine learning — trained locally, served locally, and explained locally. No cloud API. No data leaving your machine. No monthly bill.

The ghost in the machine here is invisible and everywhere: the silent, algorithmic haunting of over a billion digital payments that happen every single day in India through UPI. Most people never think about the unseen intelligence that decides, in milliseconds, whether their ₹500 chai payment is real — or a ghost transaction sliding money out of their account at 2 AM from a device they've never touched.

This project makes that invisible intelligence visible, auditable, and yours.

The spirit of Día de los Muertos — the thin veil between the living and the dead — maps perfectly onto fraud: the ghost of a transaction that looks legitimate but isn't. PaySafe India lifts that veil in real time.

Demo

The Flask web app runs locally at http://127.0.0.1:5000 after a one-time training run.

Live features:

  • 🔴 Real-time fraud probability score (0–100%)
  • ⚠️ Risk band: Low / Medium / High
  • 🔍 Counterfactual explainability — "which 3 signals drove this score, and by how much?"
  • 🎛️ Pre-loaded presets: Typical transaction vs Suspicious transaction
  • 📡 REST API endpoint /api/predict for integration

Headline model results on the held-out test set (5,279 transactions):

Model Precision Recall F1 ROC-AUC
Logistic Regression 0.962 0.884 0.921 0.965
Decision Tree 0.926 0.891 0.908 0.962
Random Forest (deployed ✅) 0.983 0.879 0.928 0.969
SVM (RBF) 0.996 0.866 0.926 0.966

A "predict everything legitimate" dummy baseline scores 82.8% accuracy with 0% recall — a model that catches zero fraud. This is why accuracy is not the metric to use for imbalanced fraud data.

Code

The full project is structured as a progressive pipeline of numbered scripts:

upi_fraud_project/
├── config.py / utils.py          # shared paths, feature list, preprocessing pipeline
├── 00_Dataset_Selection.py       # evidence for dataset choice (baseline on every dataset)
├── 01_NumPy_Foundations.py       # arrays, masks, broadcasting, vectorisation
├── 02_Pandas_DataCleaning.py     # missing values, duplicates, IQR outliers, merge
├── 03_Visualization.py           # 13 chart types
├── 04_Encoding_Preprocessing.py  # leakage audit, encoding, scaling, stratified split
├── 05_KMeans_Unsupervised.py     # elbow, silhouette, PCA, cluster profiles, anomaly score
├── 06_SupervisedModels.py        # 4 models, CV, threshold trade-off, saves deployed model
├── app/
│   ├── app.py                    # Flask web app with REST API
│   └── templates/index.html      # dark-mode UI with counterfactual explainability
└── models/
    ├── fraud_model.joblib        # deployed scikit-learn pipeline
    └── model_meta.json           # feature names, thresholds, medians for defaults
Enter fullscreen mode Exit fullscreen mode

Run it:

git clone <your-repo>
pip install -r requirements.txt
python run_all.py        # trains all models in ~2-3 minutes
python app/app.py        # opens http://127.0.0.1:5000
Enter fullscreen mode Exit fullscreen mode

How I Built It

The Open-Source Stack

Every single component is open-source:

  • scikit-learn — four classifiers (Logistic Regression, Decision Tree, Random Forest, SVM), cross-validation, preprocessing pipelines, and K-Means unsupervised clustering
  • pandas + NumPy — data cleaning, feature engineering, leakage auditing
  • matplotlib + seaborn — 13 chart types for EDA and model evaluation
  • joblib — model serialization for the Flask deployment
  • Flask — lightweight local web server with REST API

No LLM. No cloud ML service. No API keys. Just classical ML running at full speed on a laptop.

The ML Pipeline

1. Dataset selection with a leakage audit
The most important step that most tutorials skip. I evaluated three datasets:

  • fraud_dataset.csv (26,393 rows, 17.2% fraud) — chosen ✅ — AUC 0.97 with behavioral features
  • upi_transactions_2024.csv (250,000 rows, 0.19% fraud) — AUC 0.55, no signal
  • upi_fraud_india_dataset.csv (1,000 rows, 100% fraud) — impossible to classify against

Script 04 automates the leakage audit: any single feature with AUC ≥ 0.94 is a leak. Four columns (receiver_account_age, receiver_transaction_history, merchant_category_code, handle_verification_status) were purged. Without this, Random Forest scores AUC 1.000 — a model that "cheats" on training data and will fail in production.

2. The feature set that actually generalises

The deployed model uses 14 numeric + 2 categorical behavioral features:

NUMERIC_FEATURES = [
    "amount", "transaction_time_of_day",
    "transaction_amount_vs_sender_history",
    "authentication_attempts", "failed_transaction_count",
    "unusual_device_flag", "unusual_ip_flag",
    "time_pressure_indicators", "input_timing_consistency",
    "pin_entry_speed", "transaction_velocity",
    "otp_request_frequency", "session_duration",
    "geographic_disparity"
]
CATEGORICAL_FEATURES = ["session_source", "pin_entry_method"]
Enter fullscreen mode Exit fullscreen mode

These are behavioural telemetry signals — the digital fingerprints of how a human actually interacts with a payment app vs. how a script or social-engineering victim does.

3. Unsupervised pre-check (K-Means)
Before any labelled training, K-Means clustering on the full dataset finds one tight cluster that is 100% fraud. Ranking transactions by distance-to-centroid puts 99% of all fraud in the top 10% of anomalies. This is a free anomaly detection layer that requires no labels at all.

4. Four supervised classifiers, chosen by cross-validation

All models are trained inside a unified Pipeline(preprocessor → model) so that scaling and encoding are always fitted on training data only — no leakage from the test set.

The deployed model is chosen by 5-fold stratified CV F1 on the training set, not by test-set performance. Peeking at the test set to pick a model is a form of overfitting.

best = res.loc[deployable, "cv_f1_mean"].idxmax()
Enter fullscreen mode Exit fullscreen mode

5. Counterfactual explainability in the Flask app

The hardest part — and the most useful one. For every scored transaction, the app shows why it was flagged:

def drivers(values: dict, base_p: float, top: int = 3):
    """How much would the score drop if this ONE input were typical?"""
    out = []
    for f in FIELDS:
        alt = dict(values)
        alt[f["name"]] = f["default"]   # reset to legit median
        drop = base_p - score(alt)
        if drop >= 0.02:
            out.append((f["label"], drop))
    return sorted(out, key=lambda t: -t[1])[:top]
Enter fullscreen mode Exit fullscreen mode

This is counterfactual attribution: for each signal, ask "if this were normal, how much would the risk score fall?" The top three drivers are shown to the user in plain English — not just a probability, but a reason.

Why Does Open Innovation Matter?

1. Your payment data never leaves your machine

This is a fraud detector. The inputs are transaction amounts, authentication attempts, device anomalies, and geographic disparity. In India, sending this data to a third-party cloud API is not just a privacy concern — it is a regulatory one. PaySafe India runs entirely on-device. The model is a fraud_model.joblib file. No HTTP call is ever made to score a transaction.

2. You own the model — and can change what it flags

With a closed API, fraud thresholds are decided by whoever runs the service. With PaySafe India, you can retrain with python run_all.py, adjust the threshold trade-off table printed by Script 06, and redeploy in minutes. The threshold is a business decision, not a black-box setting:

threshold  precision  recall  f1   missed_fraud  false_alarms
  0.2       0.854     0.969  0.908      28          132
  0.5       0.983     0.879  0.928     110           14
  0.8       0.996     0.700  0.821     272            3
Enter fullscreen mode Exit fullscreen mode

A bank might accept 132 false alarms to miss only 28 frauds. A startup might flip the tradeoff. You decide.

3. It works with the internet off

No API keys. No model download at runtime. No cloud inference. Train once, run forever, offline. In a country where rural internet connectivity is still unreliable, this matters.

4. The leakage audit is part of the open process

Open-source ML means you see the audit trail. Script 04 explicitly prints every feature it tests, every one it rejects, and why. Script 00 runs every dataset through a baseline model and reports AUC, so you can defend your dataset choice. This kind of transparency is not possible with a closed API — you just get a score and a label.

5. The ghost cannot be locked in a box

Fraud detection built on a closed API disappears the day the service shuts down, pivots, or raises its price. This project — the data pipeline, the trained weights, the Flask app — lives on a filesystem. It is version-controlled. It can be forked, fine-tuned, and kept alive indefinitely.

That is the open-source answer to the ghost-in-the-machine problem: the ghost is ours now, and it doesn't vanish at someone else's discretion.

The Spirit of It

For the 600 million Indians who actively use UPI, a fraudulent transaction is not an abstract data-science problem. It is rent money gone at 2 AM. It is a retirement fund drained while someone sleeps.

The "ghost in the machine" is exactly right: there is already an invisible intelligence inside every payment system, deciding in milliseconds what passes and what gets blocked. Most people never see it, never understand it, and never get to question its decisions.

Building this with open-source tools — and making the explainability layer front-and-center in the UI — is a small act of making that ghost visible. Not spooky. Understood.

The moment it landed for me was running the leakage audit for the first time and watching a Random Forest score AUC 1.000 on a training set — a perfect score that was a lie. The ghost had been cheating, and the open code showed exactly how.

Prize Categories

Ghost in the Machine — Open-Source AI Challenge, Week 4

Top comments (0)