I recently did a customer churn prediction project that had a really high accuracy, but when I checked the classification report, I found that the model was missing almost every churner. That’s when I realised that it’s perfectly possible for a model to lazily predict and look very accurate when in reality it’s heavily misclassifying and missing the positive cases. This is especially the case when the classes are heavily imbalanced.
For this prompt, I chose an Employee Attrition dataset with 460 records. As soon as I loaded it, I saw how messy it was: the same departments spelled in several different ways, incomes stored as text with commas, impossible values like negative ages, and missing data. I cleaned it and ended up with 427 rows: 291 employees who stayed and 136 who left.
Since the prompt asks about a class imbalance where accuracy is 73%, I took random resamples of employees who stayed and concatenated them back to the dataset to recreate a 73:27 class imbalance (504 rows). In a real project you’d only resample the training data, never the data you evaluate on; here it’s just to recreate the 73:27 split.
You can read the full code, including the data cleaning, on GitHub: https://github.com/Rupali-Gauravaram/dev-prompt-generator/blob/main/posts/post_03.ipynb
Before and after cleaning: a few rows of the raw and cleaned data.
I then wanted to see the correlation between the various features and the target column (Attrition) using a seaborn heatmap. Most of the correlations are weak; the strongest was the ‘OverTime’ feature. So I decided to use that as the input and Attrition as the target, since the prompt asked me to build it with NumPy, without scikit-learn.
Next, I had to build the confusion matrix. I wrote down all the formulas for accuracy, precision, recall, specificity and F1 score, and then realised I needed to compute the True Positive, True Negative, False Positive and False Negative values first. It took a while of scratching my head to see how I could get these values manually. Turns out I just needed y_true and y_preds first, so I was reverse engineering as I wrote the code, because who writes this stuff manually when we have beautiful packages that do it in a few simple lines of code?
To get the y_preds, as you can guess, I needed a model that predicts, or a for loop that behaves like one. So I wrote a simple for loop that goes through every OverTime value and makes a prediction from that one column: if OverTime is 1 (Yes), the employee is likely to leave; if it’s 0 (No), they’re probably happy to keep their job.
I built a clean function to compute the metrics from two inputs (y_true and y_preds), the same way scikit-learn’s precision_score or recall_score would.
Here’s what the numbers showed. The lazy model that predicts nobody leaves scored 73% accuracy but caught 0% of the employees who left. The OverTime rule scored lower, 65.9% accuracy, but caught 46% of them (recall 46.3%, precision 38.9%). Lower accuracy, far more useful.
Then I had one last bit of the prompt left to complete: writing test cases.
Considering how much time I’d already spent on the code, I was starting to feel frustrated, but I pushed myself to complete it anyway: if I want to become a really good developer, I need to write good test cases and check my code’s sanity the traditional way.
Inside the function, an if check makes sure no denominator is 0 before dividing, so it doesn’t crash on divide-by-zero errors, which can happen when the denominator of any metric ends up being 0. The test cases then check that the function returns the right answers, including in that situation.
The three test cases:
- A normal model case
- A Perfect model case
- A lazy-model case
# testsy_true_test = np.array([1,0,1,0])# 1. Normal caseacc, prec, rec, spec, f1 = get_metrics(y_true_test, np.array([1, 0, 0, 0]))assert rec == 0.5assert prec == 1.0# 2. Perfect caseacc, prec, rec, spec, f1 = get_metrics(y_true_test, y_true_test)assert acc == 1.0assert rec == 1.0# 3. edge: model predicts no positivesacc, prec, rec, spec, f1 = get_metrics(y_true_test, np.array([0, 0, 0, 0]))assert prec == 0 assert rec == 0print("All 3 tests passed")
The takeaway: on imbalanced data, never trust accuracy alone. Check recall and precision too.
Have you ever built these metrics manually?
This was Day 3 of my 30 Days of Data Science writing challenge. This one was a real build: cleaning a messy dataset, writing the metrics from scratch in NumPy without scikit-learn, and adding test cases to check my own code.
If you’d like to see the Python code for this post, you can read it at: https://github.com/Rupali-Gauravaram/dev-prompt-generator/blob/main/posts/post_03.ipynb
Top comments (0)