A few weeks ago, I decided to stop just watching ML tutorials and actually build something end-to-end — from messy real data to a working, served model. I picked credit card fraud detection, using the well-known ULB fraud dataset (284,807 real anonymized European transactions, with only ~0.17% actually fraudulent).
Here's what I learned along the way — including a mistake that taught me more than any tutorial could have.
The trap: 99.8% accuracy that means nothing
The first thing I checked was the class balance. Only 292 out of 170,799 transactions in my training data were fraud — about 0.17%.
That number matters more than it looks. If a model just predicted "not fraud" for every single transaction, it would score 99.83% accuracy — and be completely useless. It would catch zero fraud, ever.
This is the single biggest lesson from this project: never trust accuracy alone on imbalanced data. The real metrics that matter are precision and recall on the minority class — in my case, how many real fraud cases the model actually catches (recall), and how often it's right when it flags something as fraud (precision).
Cleaning data on purpose, badly, to learn how to fix it
Before touching the real dataset, I deliberately broke a copy of it — injecting missing values, duplicate rows, and inconsistent formatting (like "$618.00" mixed in with clean floats in the same column). Then I had to detect and fix each problem myself.
A few things stuck with me:
- Not all "bad" data should be treated the same way. A truly missing value and a badly-formatted-but-recoverable value need different fixes. I filled true missing values with the median (resistant to outliers), but parsed the messy currency strings back into numbers instead of throwing them away.
- Duplicate rows are sneaky. They don't look wrong — they're just the same real event logged twice — but they quietly bias a model into over-weighting whatever pattern got duplicated.
Comparing models honestly, not just picking one
I trained and compared several approaches on the same data:
| Model | Precision (fraud) | Recall (fraud) |
|---|---|---|
| Logistic Regression | 0.78 | 0.78 |
| Logistic Regression (scaled) | 0.87 | 0.71 |
| Logistic Regression + class_weight='balanced' | 0.05 | 0.98 |
| Random Forest | 0.96 | 0.83 |
That third row is a great cautionary example. Forcing the model to aggressively catch every fraud case sounds good in theory — but it tanked precision to 5%, meaning it flagged huge numbers of real, legitimate transactions as fraud just to avoid missing rare cases. Depending on the business, that tradeoff might be worth it — or it might be a disaster. There's no universal right answer; it depends on what a false alarm actually costs versus what a missed fraud case costs.
Random Forest ended up winning clearly, likely because it can carve out complex, non-linear patterns in the anonymized features — something a single linear decision boundary (Logistic Regression) struggles with.
Tracking experiments properly instead of a hand-written table
By the third or fourth model variant, I was manually copying accuracy numbers into a notes file to keep track of what I'd tried. That's exactly the problem experiment tracking tools like MLflow are built to solve — instead of a fragile personal table, every run's parameters and metrics get logged automatically, comparable side-by-side, permanently.
Once I actually felt that manual-tracking pain firsthand, adding MLflow made immediate sense — it wasn't just "a tool someone said to learn," it was solving a real annoyance I'd already run into.
Shipping it as an API — and a bug that traced back to an earlier mistake
I wrapped the final model in a FastAPI service so it could take a transaction's details and return a live prediction. Testing it threw a real error:
X has 30 features, but RandomForestClassifier is expecting 31 features as input.
Turned out there was a stray leftover index column (__index_level_0__) still present in the training data — an artifact from how the dataset had been exported — that I'd accidentally trained the model on. The fix wasn't in the API code at all; it was going back to the training script and dropping that column before retraining.
That was a good reminder that bugs discovered at serving time often point back to something upstream in data prep, not the code you're currently staring at.
What's next
This project is now cleaned, tracked in MLflow, and served through a working API that correctly classifies both fraud and legitimate transactions I tested it against. Next up: containerizing it with Docker, and building a Retrieval-Augmented Generation (RAG) project to round out the skill set.
If you're early in your ML journey too — my honest advice is to deliberately break something (like I did with the dirty data) and fix it yourself, rather than only ever working with clean, ready-made datasets. That's where the real learning happens.
Currently learning ML engineering and AI agent development hands-on, one real project at a time. Open to entry-level opportunities and would love to connect with anyone working in this space.
Top comments (0)