# ============================================================
# FINAL STRUCTURED FEATURE BUILDER
#
# Training:
# Country mapping fitted on the complete development set
#
# Test:
# Same mapping applied to untouched test data
#
# Test target is NEVER used.
# ============================================================
def build_final_structured_features(
X_train,
y_train,
X_test
):
train_part = X_train.copy()
test_part = X_test.copy()
# --------------------------------------------------------
# Product Weight → Winsorization
# --------------------------------------------------------
lower_limit, upper_limit = fit_winsorization(
train_part,
WEIGHT_COL
)
train_part = apply_winsorization(
train_part,
WEIGHT_COL,
lower_limit,
upper_limit
)
test_part = apply_winsorization(
test_part,
WEIGHT_COL,
lower_limit,
upper_limit
)
# --------------------------------------------------------
# Numeric features
# --------------------------------------------------------
numeric_cols = [
YEAR_COL,
WEIGHT_COL
]
train_numeric = train_part[numeric_cols].apply(
pd.to_numeric,
errors="coerce"
)
test_numeric = test_part[numeric_cols].apply(
pd.to_numeric,
errors="coerce"
)
# --------------------------------------------------------
# Categorical features → One-Hot Encoding
# --------------------------------------------------------
other_ohe = OneHotEncoder(
handle_unknown="ignore",
sparse_output=False
)
train_other_ohe = other_ohe.fit_transform(
train_part[OTHER_CATEGORICAL_FEATURES]
)
test_other_ohe = other_ohe.transform(
test_part[OTHER_CATEGORICAL_FEATURES]
)
# --------------------------------------------------------
# Country → Target Encoding
#
# Mapping is learned from complete development data only.
# --------------------------------------------------------
country_means = (
pd.DataFrame({
"Country": train_part[COUNTRY_COL],
"Target": y_train
})
.groupby("Country")["Target"]
.mean()
)
global_mean = np.mean(y_train)
train_country = (
train_part[COUNTRY_COL]
.map(country_means)
.fillna(global_mean)
.to_numpy()
.reshape(-1, 1)
)
test_country = (
test_part[COUNTRY_COL]
.map(country_means)
.fillna(global_mean)
.to_numpy()
.reshape(-1, 1)
)
# --------------------------------------------------------
# Combine structured features
# --------------------------------------------------------
X_train_structured = np.hstack([
train_numeric.to_numpy(),
train_other_ohe,
train_country
])
X_test_structured = np.hstack([
test_numeric.to_numpy(),
test_other_ohe,
test_country
])
return (
X_train_structured,
X_test_structured
)
Final Structured Feature Builder — Purpose
This function prepares the final structured features for the last stage of the project, where you train the final model on the complete development set and then evaluate it on the untouched independent test set.
The key difference from the previous build_structured_features() function is:
During CV, preprocessing is fitted separately inside each fold. Here, after model/feature selection is complete, the preprocessing is fitted on the complete development set and then applied to the test set.
Most importantly:
The test target is never used.
1. Define the function
```python id="c2r7xq"
def build_final_structured_features(
X_train,
y_train,
X_test
):
The function receives three inputs:
* `X_train` → complete development features
* `y_train` → development-set target
* `X_test` → untouched test features
Here, `X_train` is actually your **full development set**, not just one CV fold.
The target `y_train` is required because Country uses **target encoding**.
`X_test` contains the test features but its target is deliberately not passed into the function.
### Viva question
**Why does the function need `y_train` but not `y_test`?**
Answer:
> “Because country target encoding requires the training target to calculate country-level target means. I do not need and deliberately do not use `y_test`, because the independent test target must remain unseen until final evaluation.”
---
# 2. Copy the data
```python id="w7w5kq"
train_part = X_train.copy()
test_part = X_test.copy()
Creates copies of both datasets.
This means the original DataFrames are not modified directly.
Product Weight → Winsorization
3. Learn winsorization limits from development data
```python id="j3f5r9"
lower_limit, upper_limit = fit_winsorization(
train_part,
WEIGHT_COL
)
This calculates the lower and upper limits for Product Weight using **only the development data**.
Winsorization limits extreme values by capping observations beyond the selected boundaries.
Conceptually:
```text
Very small weight → lower limit
Normal weight → unchanged
Very large weight → upper limit
Important leakage point
The limits are learned from training/development data only.
You do not calculate the limits using the test data.
4. Apply limits to development data
```python id="6t7y1e"
train_part = apply_winsorization(
train_part,
WEIGHT_COL,
lower_limit,
upper_limit
)
The previously calculated limits are applied to the development data.
---
### 5. Apply the SAME limits to test data
```python id="4d1q3r"
test_part = apply_winsorization(
test_part,
WEIGHT_COL,
lower_limit,
upper_limit
)
This is a very important line.
You do not fit new limits on the test set.
Instead:
Development data
↓
learn limits
↓
┌───────────────┐
│ same limits │
│ │
├───────────────┤
│ Development │
│ Test │
└───────────────┘
This keeps the transformation consistent and prevents test-data information from influencing preprocessing.
Viva answer
“I fit the winsorization thresholds on the complete development set and then applied those same thresholds to the test set. I did not estimate any preprocessing parameter from the test distribution.”
Numeric Features
6. Define numeric columns
```python id="8h9f3s"
numeric_cols = [
YEAR_COL,
WEIGHT_COL
]
The numeric features are:
* Year of reporting
* Product weight
---
### 7. Convert development numeric columns
```python id="5m2d8a"
train_numeric = train_part[numeric_cols].apply(
pd.to_numeric,
errors="coerce"
)
Attempts to convert the numeric columns into numeric values.
If something cannot be converted, it becomes NaN.
For example:
"2024" → 2024
"10.5" → 10.5
"unknown" → NaN
8. Convert test numeric columns
```python id="1j8v6c"
test_numeric = test_part[numeric_cols].apply(
pd.to_numeric,
errors="coerce"
)
The exact same type of conversion is applied to test data.
No test target is involved.
---
# Categorical Features → One-Hot Encoding
### 9. Create OneHotEncoder
```python id="q5z8n2"
other_ohe = OneHotEncoder(
handle_unknown="ignore",
sparse_output=False
)
This encoder converts categorical variables into numerical binary columns.
Your categorical variables include:
- Company's GICS Industry
- Protocol used for PCF
- Stage-level CO₂e available
For example:
Industry:
Food
Technology
Automotive
could become columns such as:
Industry_Food
Industry_Technology
Industry_Automotive
handle_unknown="ignore"
This is particularly important for the test set.
Suppose the development data contains:
Food
Technology
Automotive
but the test set contains a previously unseen:
Mining
The encoder does not crash.
It ignores the unseen category.
Viva answer
“I used
handle_unknown='ignore'because the test set may contain categorical values that were not present in the development set. This allows the transformation to proceed without creating an error or fitting the encoder using test information.”
10. Fit encoder on development data
```python id="q0e7v3"
train_other_ohe = other_ohe.fit_transform(
train_part[OTHER_CATEGORICAL_FEATURES]
)
This is where the encoder **learns the category structure**.
For example, it learns which industry/protocol/stage categories exist in the development data.
---
### 11. Transform test data
```python id="m8v4q6"
test_other_ohe = other_ohe.transform(
test_part[OTHER_CATEGORICAL_FEATURES]
)
Notice:
transform()
not:
fit_transform()
This is crucial.
The encoder has already been fitted using development data.
The test data is only transformed using the existing mapping.
Viva question
Why not use fit_transform() on the test set?
Answer:
“Because fitting on the test set would allow information from the test distribution to influence the preprocessing. I fit the encoder only on the development data and then use
transform()on the test data.”
Country → Target Encoding
This is probably the most important section of this function.
12. Calculate country means
```python id="v4k8s2"
country_means = (
pd.DataFrame({
"Country": train_part[COUNTRY_COL],
"Target": y_train
})
.groupby("Country")["Target"]
.mean()
)
This creates a mapping:
```text
Country → Mean target
For example, conceptually:
Germany → 25.4
France → 18.7
Japan → 31.2
The actual values depend on your dataset.
Critical point
These means are calculated using:
Complete development data
and:
y_train
only.
There is no y_test here.
13. Calculate global mean
```python id="1k7d3x"
global_mean = np.mean(y_train)
This calculates the overall mean target of the development set.
It is used as a fallback.
---
### 14. Encode development countries
```python id="f0j5c7"
train_country = (
train_part[COUNTRY_COL]
.map(country_means)
.fillna(global_mean)
.to_numpy()
.reshape(-1, 1)
)
Let's break this into functions.
.map(country_means)
Maps each country to its calculated mean target.
Example:
Germany → 25.4
France → 18.7
Japan → 31.2
.fillna(global_mean)
If a country doesn't have a mapping, use the overall development-set mean instead.
.to_numpy()
Converts the pandas Series into a NumPy array.
.reshape(-1, 1)
Converts it into a two-dimensional column:
(n,)
becomes:
(n, 1)
This makes it suitable for np.hstack() later.
15. Encode test countries
```python id="n9w3p5"
test_country = (
test_part[COUNTRY_COL]
.map(country_means)
.fillna(global_mean)
.to_numpy()
.reshape(-1, 1)
)
This uses the **same `country_means` learned from the development data**.
This is extremely important.
The test target is not used.
If the test contains a country that wasn't present in development:
```text
Unknown country
↓
global development mean
Viva answer
“For the final test transformation, country target encoding is fitted using the complete development set. Test countries are mapped using that fixed mapping. If a test country was unseen during development, I use the development global mean as a fallback. Therefore, the test target is never involved.”
16. Combine structured features
Development features
```python id="g3w8r1"
X_train_structured = np.hstack([
train_numeric.to_numpy(),
train_other_ohe,
train_country
])
`np.hstack()` means **horizontal stacking**.
It combines:
```text
Numeric features
+
One-hot categorical features
+
Country target encoding
↓
Final structured feature matrix
Test features
```python id="k2m6v8"
X_test_structured = np.hstack([
test_numeric.to_numpy(),
test_other_ohe,
test_country
])
The same feature structure is created for the test data.
Therefore, the training and test matrices have compatible columns.
---
# 17. Return the final feature matrices
```python id="z4r1t6"
return (
X_train_structured,
X_test_structured
)
The function returns:
Final development structured features
+
Final test structured features
These can then be combined with:
SBERT → PCA 50
to create the final hybrid features.
Why this function is different from the previous one
This is very important for your viva.
During CV
You used:
build_structured_features()
because each fold needs its own preprocessing.
For example:
Fold 1:
80% training → fit preprocessing
20% validation → transform
Fold 2:
different 80% → fit preprocessing
different 20% → transform
Final model
Now model selection is finished.
So you use:
build_final_structured_features()
with:
Complete development set
↓
fit preprocessing
↓
Independent test set
↓
transform only
The workflow is:
DEVELOPMENT DATA
│
▼
Fit preprocessing
│
┌────────────────┴───────────────┐
▼ ▼
Complete development Test data
transform transform
│ │
▼ ▼
Train final model Final evaluation
Most important leakage explanation
Your examiner may ask:
“How did you prevent test-set leakage?”
Strong answer:
“I kept the test set completely untouched during model selection. In the final structured feature builder, winsorization thresholds, one-hot encoding categories, and country target-encoding mappings are all learned from the complete development set. The test set is only transformed using those learned parameters. The test target is never used during preprocessing or training.”
One subtle point you should know
There is an important distinction between test leakage and training self-encoding.
Here:
country_means
is calculated using the complete development set, so each development observation contributes to its own country's mean.
That is acceptable for fitting the final model after all CV/model-selection decisions are complete, because you are now using all available development information to train the final model.
It would not be appropriate for generating unbiased CV predictions.
That's exactly why your earlier CV function used OOF country target encoding, while this final function uses the complete development mapping.
Viva question
“Why did you use OOF encoding during CV but normal target encoding here?”
Answer:
“During cross-validation, OOF encoding was necessary to prevent an observation's own target from contributing to its encoded training feature. After model selection, I retrain the final model using the complete development data, so I use the full development target information to construct the final training representation. The independent test set is still encoded only from that development-derived mapping.”
Full final pipeline
This function fits into the final stage like this:
Original dataset
↓
80% Development / 20% Test
↓
CV + PCA selection + Model comparison + XGB tuning
↓
All modelling decisions fixed
↓
Complete Development Set
│
├── Winsorization limits
├── OHE mapping
└── Country target mapping
↓
Final structured features
│
└──────────────┐
▼
Test data ──transform using same mappings
│
▼
Structured test features
Then you combine these with the final:
SBERT
↓
PCA = 50
↓
Hybrid features
↓
Final tuned XGBoost
↓
Independent test prediction
↓
MAE / RMSE / R²
Best 30-second viva answer
“This function prepares the structured features for the final independent test evaluation. Unlike the CV preprocessing, I now fit the preprocessing on the complete development set because model selection has already finished. Product weight winsorization thresholds, one-hot encoding categories, and country target-encoding mappings are learned only from the development data. The same learned transformations are then applied to the untouched test data. In particular, the test target is never used. This gives me a leakage-controlled final feature representation for training the final model and evaluating it on genuinely unseen data.”
For your final evaluation
- Check target-encoding bias in final training
- Verify feature-column consistency
Top comments (0)