# ============================================================
# LEAKAGE-SAFE PRODUCT WEIGHT WINSORIZATION — IQR METHOD
# ============================================================
WEIGHT_COL = "Product weight (kg)"
weight_before = pd.to_numeric(
train_df[WEIGHT_COL],
errors="coerce"
)
def fit_winsorization(
X,
column
):
weight = pd.to_numeric(
X[column],
errors="coerce"
)
Q1 = weight.quantile(0.25)
Q3 = weight.quantile(0.75)
IQR = Q3 - Q1
lower_limit = Q1 - 1.5 * IQR
upper_limit = Q3 + 1.5 * IQR
return lower_limit, upper_limit
def apply_winsorization(
X,
column,
lower_limit,
upper_limit
):
X = X.copy()
X[column] = X[column].clip(
lower=lower_limit,
upper=upper_limit
)
return X
# Learn limits from training data only
weight_lower, weight_upper = fit_winsorization(
train_df,
WEIGHT_COL
)
print("Training weight maximum:",
weight_before.max(), "kg")
print("IQR-based lower limit:",
weight_lower)
print("IQR-based upper limit:",
weight_upper)
# Apply the same training-derived limits
train_df = apply_winsorization(
train_df,
WEIGHT_COL,
weight_lower,
weight_upper
)
test_df = apply_winsorization(
test_df,
WEIGHT_COL,
weight_lower,
weight_upper
)
# Store weight after Winsorisation
weight_after = train_df[WEIGHT_COL].copy()
# ============================================================
# BEFORE vs AFTER BOX PLOT
# ============================================================
plt.figure(figsize=(8, 5))
plt.boxplot(
[
weight_before.dropna(),
weight_after.dropna()
],
tick_labels=[
"Before Winsorisation",
"After Winsorisation"
],
showfliers=True
)
plt.ylabel("Product Weight (kg)")
plt.title("Product Weight Before and After Winsorisation")
plt.tight_layout()
plt.show()
LEAKAGE-SAFE PRODUCT WEIGHT WINSORIZATION — IQR METHOD
Purpose
This cell handles extreme values in the Product Weight feature using IQR-based Winsorization.
The main goals are:
- Reduce the influence of extreme Product Weight values
- Keep the observations instead of removing them
- Learn the limits from training data only
- Apply the same limits to both training and test data
- Visualise the change using a box plot
WEIGHT_COL = "Product weight (kg)"
Only the Product Weight feature is Winsorized.
Why Product Weight?
Product Weight is a continuous numerical feature.
Extreme values in this feature may have a strong influence on the ML model, so we use Winsorization to cap extreme values.
We do not apply the same method to every feature because different features have different types and meanings.
Flow
Product Weight
↓
Convert to numeric
↓
Calculate Q1 and Q3 from training data
↓
Calculate IQR
↓
Calculate lower and upper limits
↓
Apply limits to training data
↓
Apply SAME limits to test data
↓
Compare Before vs After
↓
Box Plot
Key Process
1. Select Product Weight
WEIGHT_COL = "Product weight (kg)"
This identifies the specific feature that will be processed.
Full dataset
↓
25 features
↓
Product Weight
↓
Winsorization
So, this is feature-specific preprocessing.
2. Keep the original values
weight_before = pd.to_numeric(
train_df[WEIGHT_COL],
errors="coerce"
)
The original training Product Weight values are stored in weight_before.
This is needed later to compare:
Before Winsorisation
vs
After Winsorisation
3. Calculate IQR-based limits
Q1 = weight.quantile(0.25)
Q3 = weight.quantile(0.75)
Here:
Q1 = 25th percentile
Q3 = 75th percentile
Then:
IQR = Q3 - Q1
The IQR represents the spread of the middle 50% of the data.
4. Calculate the lower and upper limits
lower_limit = Q1 - 1.5 * IQR
upper_limit = Q3 + 1.5 * IQR
These are the standard IQR-based boundaries.
Lower limit = Q1 - 1.5 × IQR
Upper limit = Q3 + 1.5 × IQR
For example:
Q1 = 10 kg
Q3 = 30 kg
IQR = 30 - 10
= 20 kg
Lower = 10 - 1.5×20
= -20 kg
Upper = 30 + 1.5×20
= 60 kg
So values above 60 kg would be capped at 60 kg.
5. Learn limits from training data only
weight_lower, weight_upper = fit_winsorization(
train_df,
WEIGHT_COL
)
This is important for data leakage prevention.
The limits are learned only from the training data.
Training data
↓
Q1 + Q3
↓
IQR
↓
Lower + Upper limits
The test data is not used to calculate these limits.
6. Apply Winsorization
X[column] = X[column].clip(
lower=lower_limit,
upper=upper_limit
)
clip() caps values outside the learned boundaries.
For example:
Original:
5
10
20
30
1000
Upper limit = 100
After:
5
10
20
30
100
The extreme observation is not deleted.
It is simply capped at the upper limit.
7. Apply the same limits to training and test
train_df = apply_winsorization(
train_df,
WEIGHT_COL,
weight_lower,
weight_upper
)
Then:
test_df = apply_winsorization(
test_df,
WEIGHT_COL,
weight_lower,
weight_upper
)
The important point is:
TRAIN
↓
Learn limits
↓
Lower + Upper
↓
TRAIN → Apply
TEST → Apply SAME limits
We do not calculate separate IQR limits for the test data.
This keeps the preprocessing leakage-safe.
8. Store the processed values
weight_after = train_df[WEIGHT_COL].copy()
Now we have:
weight_before → Original Product Weight
weight_after → Winsorized Product Weight
These are used for visual comparison.
9. Before vs After Box Plot
plt.boxplot(
[
weight_before.dropna(),
weight_after.dropna()
],
tick_labels=[
"Before Winsorisation",
"After Winsorisation"
],
showfliers=True
)
The box plot compares the distribution before and after Winsorization.
We can visually check whether the extreme values have been reduced or capped.
Before
↓
More extreme values
After
↓
Extreme values capped
What should I look for in the plot?
You should mainly look for:
- Very long whiskers before Winsorization
- Extreme points/fliers before Winsorization
- Reduced extreme range after Winsorization
- The main distribution being largely preserved
The goal is not to remove all outliers.
The goal is to reduce the influence of extreme values while retaining the observations.
Important distinction
There are two different concepts here:
IQR
↓
Defines the boundaries
and:
Winsorization
↓
Caps values at those boundaries
So we are using an IQR-based boundary to perform Winsorization.
Viva Cheat Sheet
Q: What did you do in this cell?
“I applied IQR-based Winsorization to the Product Weight feature to reduce the influence of extreme values.”
Q: Why Product Weight?
“It is a continuous numerical feature where extreme values may strongly influence the model.”
Q: How did you calculate the limits?
“I calculated Q1 and Q3 from the training data, calculated the IQR, and used Q1 minus 1.5 times IQR and Q3 plus 1.5 times IQR as the lower and upper limits.”
Q: Did you remove the outliers?
“No. I capped extreme values using Winsorization instead of removing observations.”
Q: Did you use the test data to calculate Q1 or Q3?
“No. The IQR limits were learned from the training data only, and the same limits were applied to the test data.”
Q: Why is this leakage-safe?
“Because the test distribution was not used to learn any preprocessing parameters.”
Q: What is the purpose of the box plot?
“It visually compares the Product Weight distribution before and after Winsorization and shows whether extreme values have been reduced.”
Remember
Product Weight only
↓
Q1 = 25th percentile
Q3 = 75th percentile
↓
IQR = Q3 - Q1
↓
Lower = Q1 - 1.5×IQR
Upper = Q3 + 1.5×IQR
↓
Cap extreme values
And the most important leakage rule:
TRAIN → Calculate Q1, Q3, IQR → Learn limits
TEST → Never calculate new limits
→ Use the same training-derived limits
One-line viva answer:
“I used training-derived IQR boundaries to Winsorize Product Weight, capped the extreme values instead of removing them, and applied the same learned limits to the test data to prevent leakage.”
Top comments (0)