Hi everyone! 👋 I'm currently working through the Titanic dataset as part of the CampusX YouTube course, and I ran into an interesting issue involving OneHotEncoder and SimpleImputer that I finally understood after digging into the problem.
This blog is all about that journey — what caused the shape mismatch between training and testing data, and how I fixed it. If you're also working on preprocessing categorical variables in machine learning, this might save you a few hours of debugging!
🧠 The Setup
We’re using the Titanic dataset for classification (predicting survival), and like most people, I’m preprocessing the Sex and Embarked columns using:
- SimpleImputer to handle missing values
- OneHotEncoder to convert categorical variables into numerical format
Here’s a snippet of what I had:
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import OneHotEncoder
si_embarked = SimpleImputer(strategy='most_frequent')
ohe_embarked = OneHotEncoder(sparse=False, handle_unknown='ignore')
# Imputation
x_train_embarked = si_embarked.fit_transform(x_train[['Embarked']])
x_test_embarked = si_embarked.transform(x_test[['Embarked']])
# Encoding
x_train_embarked = ohe_embarked.fit_transform(x_train[['Embarked']])
x_test_embarked = ohe_embarked.transform(x_test[['Embarked']])
⚠️ The Issue
After running this, I checked the shapes:
x_train_embarked.shape → (712, 4)
x_test_embarked.shape → (179, 3)
Wait — what?!
Why does the train set have 4 columns, and the test set has only 3, even though I used handle_unknown='ignore'?
Wasn’t that supposed to handle unknown categories safely?
🕵️♂️ Investigating the Root Cause
I ran a few more checks and realized something sneaky:
x_train['Embarked'].isnull().sum() # Output: 2
Hmm… that’s weird. I thought I had already imputed missing values. But then I remembered this part of my code:
x_train_embarked = si_embarked.fit_transform(x_train[['Embarked']])
Aha! 💡 I had imputed missing values into a new variable, x_train_embarked, but I never updated the original x_train DataFrame!
That means the original x_train['Embarked'] still had NaN values when I called .fit() on the encoder:
x_train['Embarked'] = ohe_embarked.fit_transform(x_train[['Embarked']])
This caused the encoder to treat NaN as a valid category, resulting in 4 categories being learned:
['C', 'Q', 'S', NaN]
But in the test data, there were no NaN values, only:
['C', 'Q', 'S']
So the encoder ignored the unseen NaN category, resulting in:
x_test_embarked.shape = (179, 3)
✅ The Fix
The correct way was to assign the imputed values back to the original DataFrame:
x_train['Embarked'] = si_embarked.fit_transform(x_train[['Embarked']])
x_test['Embarked'] = si_embarked.transform(x_test[['Embarked']])
Now, when I fit the encoder:
ohe_embarked = OneHotEncoder(sparse=False, handle_unknown='ignore')
x_train_embarked = ohe_embarked.fit_transform(x_train[['Embarked']])
x_test_embarked = ohe_embarked.transform(x_test[['Embarked']])
✅ The shapes finally matched:
x_train_embarked.shape → (712, 3)
x_test_embarked.shape → (179, 3)
📌 Bonus: What handle_unknown='ignore' Really Means
Here’s a quick visual explanation:
Imagine your training data had these categories:
['Red', 'Blue', 'Green']
And you encode them like this:
| Red | Blue | Green |
|---|---|---|
| 1 | 0 | 0 |
| 0 | 1 | 0 |
| 0 | 0 | 1 |
Now your test data contains a new category: 'Yellow'.
If you use:
OneHotEncoder(handle_unknown='ignore')
Then the encoder will just assign all 0s for 'Yellow':
| Red | Blue | Green |
|---|---|---|
| 0 | 0 | 0 |
✅ No crash. But also — you now have a row of all zeros!
🎓 Final Takeaways
- Always handle missing values before encoding
- If you’re using
SimpleImputer, assign the output back to your original DataFrame -
handle_unknown='ignore'prevents errors, but doesn’t fix shape mismatches caused by unseen categories during.fit()
This was a great learning moment for me while working through the Titanic dataset with CampusX. Hope this helps anyone else facing the same mystery! 🧩
Let me know if you've run into similar preprocessing surprises!
Top comments (0)