Feature Engineer each class separately in Binary Classification

Bechir Jamoussi — Tue, 19 Jul 2022 10:03:40 +0000

I have an imbalanced tabular dataset, my problem is a binary classification. The dataset is strongly imbalanced so I have performed oversampling, but it did not solve the issue, you can find the Classification Report below:(The accuracy is 88% but I don't care, it does not represent well the performance since the dataset is imbalanced)

The only explanation that I found is that the features selected are well suited to detect "0" while they don't tell much information about the "1" class, is there any way to catch the best features that represent the "1" class, maybe split the dataset into 1_Class_Dataset and 0_Class_Dataset and catch the best features for each and then combine both? If it is not possible can you please suggest another solution?

Low F1-Score due to Imbalanced Dataset even after resampling

Bechir Jamoussi — Sat, 16 Jul 2022 23:50:18 +0000

I am performing a Binary Classification over an imbalanced dataset:

0: 16,263

1: 214

I have used multiple oversampling, undersampling, and combination techniques, below are the results that I have obtained:
I obtained this plots thanks to this piece of code:

def plot_resampling(X, y, sampler, ax, title=None):
    X_res, y_res = sampler.fit_resample(X, y)
    ax.scatter(X_res[:, 0], X_res[:, 1], c=y_res, alpha=0.8, edgecolor="k")
    if title is None:
        title = f"Resampling with {sampler.__class__.__name__}"
    ax.set_title(title)
    sns.despine(ax=ax, offset=10)

Clarification: The X and y are the X_train and y_train and I used it to show the distribution of my data points before and after the resampling.

For the RandomUnderSampler, the first one is without replacement and the second one is with replacement=True

You need to know also that I have multiple outliers in my dataset, and hence, multiple columns are skewed, so I chose to use models that are not sensitive to skewness like:

SVC
Naive Bayes Classifier
Ensemble XGboost
KNN

For now, the best result that I have obtained is with SVC(kernel = "rbf") and using the SMOTE technique(Of course the sampling is only performed on the training dataset since the test one should represent the real population):

Test Accuracy: 0.75
Training Accuracy: 0.88

But the classification report is not good, the f1-score is 0.51, there is a real issue with the 1 class even after the resampling!! as you can see below:

Here is also the Confusion Matrix:

Can you please help me improve the f1 score, what is your analysis of the situation, and what are your suggestions?

DEV Community: Bechir Jamoussi

Feature Engineer each class separately in Binary Classification

Low F1-Score due to Imbalanced Dataset even after resampling