# ============================================================
# MODEL COMPARISON
# PCA = 50 + OOF Country Target Encoding
# Log Target + Stratified 5-Fold CV
# ============================================================
model_results = []
for model_name, model_template in models.items():
print(f"\nRunning: {model_name}")
model_fold_results = []
for fold, (train_idx, valid_idx) in enumerate(
skf.split(train_df, target_bins),
start=1
):
# ----------------------------------------------------
# 1. Fold data
# ----------------------------------------------------
X_fold_train = train_df.iloc[train_idx].copy()
X_fold_valid = train_df.iloc[valid_idx].copy()
y_fold_train_log = (
y_log.iloc[train_idx].to_numpy()
)
y_fold_valid_raw = (
train_df.iloc[valid_idx][TARGET].to_numpy()
)
# ----------------------------------------------------
# 2. Structured features
# Country = OOF Target Encoding
# ----------------------------------------------------
X_struct_train, X_struct_valid = (
build_structured_features(
X_fold_train,
y_fold_train_log,
X_fold_valid
)
)
# ----------------------------------------------------
# 3. SBERT features
# ----------------------------------------------------
X_sbert_train = X_train_sbert[train_idx]
X_sbert_valid = X_train_sbert[valid_idx]
# ----------------------------------------------------
# 4. PCA = 50
# ----------------------------------------------------
(
X_sbert_train_pca,
X_sbert_valid_pca,
_
) = apply_pca_to_sbert(
X_sbert_train,
X_sbert_valid,
n_components=50
)
# ----------------------------------------------------
# 5. Hybrid feature fusion
# ----------------------------------------------------
X_hybrid_train = fuse_features(
X_sbert_train_pca,
X_struct_train
)
X_hybrid_valid = fuse_features(
X_sbert_valid_pca,
X_struct_valid
)
# ----------------------------------------------------
# 6. Fresh model instance
# ----------------------------------------------------
model = clone(model_template)
# ----------------------------------------------------
# 7. Scaling
# ----------------------------------------------------
scale_features = model_name in [
"Linear Regression",
"Ridge Regression",
"SVR"
]
# ----------------------------------------------------
# 8. Evaluate
# ----------------------------------------------------
mae, rmse, r2 = evaluate_model(
model=model,
X_train=X_hybrid_train,
y_train_log=y_fold_train_log,
X_valid=X_hybrid_valid,
y_valid_raw=y_fold_valid_raw,
scale_features=scale_features
)
model_fold_results.append({
"Model": model_name,
"Fold": fold,
"MAE": mae,
"RMSE": rmse,
"R2": r2
})
print(
f" Fold {fold}: "
f"MAE={mae:.4f}, "
f"RMSE={rmse:.4f}, "
f"R²={r2:.4f}"
)
# Store all folds
model_results.extend(
model_fold_results
)
# ============================================================
# MODEL COMPARISON SUMMARY
# ============================================================
model_results_df = pd.DataFrame(
model_results
)
model_summary = (
model_results_df
.groupby("Model")
.agg(
Mean_MAE=("MAE", "mean"),
Mean_RMSE=("RMSE", "mean"),
Mean_R2=("R2", "mean"),
Std_R2=("R2", "std")
)
.reset_index()
.sort_values(
"Mean_R2",
ascending=False
)
)
display(model_summary)
Model Comparison
This is the main model-comparison cell in your notebook.
At this stage, you have already decided:
- PCA = 50 components
- Country = OOF target encoding
- Target = log-transformed PCF
- Evaluation = stratified 5-fold cross-validation
- Features = SBERT + structured features
Now the question becomes:
Which regression algorithm performs best with this final hybrid feature representation?
Your pipeline is:
Development data
↓
Stratified 5-Fold CV
↓
For each model
↓
Build structured features
+
SBERT features
↓
PCA = 50
↓
Hybrid features
↓
Train model
↓
Predict validation fold
↓
MAE / RMSE / R²
↓
Repeat for all 5 folds
↓
Average results
↓
Compare models
1. Create result storage
```python id="mc01"
model_results = []
Creates an empty list where all model/fold evaluation results will be stored.
For example, one result might eventually look like:
```text id="x8g5dk"
{
"Model": "Random Forest",
"Fold": 1,
"MAE": ...,
"RMSE": ...,
"R2": ...
}
Because you have multiple models and five folds, the list will contain one record for every model × fold combination.
2. Loop through every model
```python id="mc02"
for model_name, model_template in models.items():
This loops through the model dictionary you created earlier.
For example:
```text id="2nqk3c"
Linear Regression
Ridge Regression
SVR
Random Forest
XGBoost
CatBoost
For each model, the entire five-fold evaluation is performed.
model_name
The readable model name:
```text id="mc03"
"Random Forest"
#### `model_template`
The corresponding model object/configuration.
```text id="mc04"
RandomForestRegressor(...)
3. Print the current model
```python id="mc05"
print(f"\nRunning: {model_name}")
Displays which model is currently being evaluated.
For example:
```text id="v7d2fa"
Running: Random Forest
The \n adds a blank line before the message to make the output easier to read.
4. Create fold-result storage
```python id="mc06"
model_fold_results = []
Creates an empty list for the current model.
For example, while evaluating Random Forest:
```text id="f8n1x4"
Random Forest
↓
Fold 1
Fold 2
Fold 3
Fold 4
Fold 5
The five results are stored in this list.
Five-Fold Cross-Validation
```python id="mc07"
for fold, (train_idx, valid_idx) in enumerate(
skf.split(train_df, target_bins),
start=1
):
This generates the five cross-validation folds.
You previously created:
```python id="mc08"
skf = StratifiedKFold(
n_splits=5,
shuffle=True,
random_state=RANDOM_STATE
)
and:
```python id="mc09"
target_bins = pd.qcut(...)
So here you are actually using them.
The output of each iteration is:
```text id="mc10"
train_idx
valid_idx
train_idx
Rows used to train the model in this fold.
valid_idx
Rows held out for validation in this fold.
The process looks like:
```text id="q4c7w8"
Fold 1 → 80% train / 20% validation
Fold 2 → 80% train / 20% validation
Fold 3 → 80% train / 20% validation
Fold 4 → 80% train / 20% validation
Fold 5 → 80% train / 20% validation
Each observation is used for validation once.
---
# 1. Fold Data
```python id="mc11"
X_fold_train = train_df.iloc[train_idx].copy()
X_fold_valid = train_df.iloc[valid_idx].copy()
These lines select the actual rows for the current fold.
Training fold
```python id="mc12"
X_fold_train = train_df.iloc[train_idx].copy()
Selects the training observations.
### Validation fold
```python id="mc13"
X_fold_valid = train_df.iloc[valid_idx].copy()
Selects the validation observations.
The .copy() creates independent DataFrames so later operations do not unintentionally modify the original train_df.
5. Training target
```python id="mc14"
y_fold_train_log = (
y_log.iloc[train_idx].to_numpy()
)
Selects the log-transformed PCF values corresponding to the training fold.
So the model receives:
```text id="mc15"
X_fold_train
+
y_fold_train_log
6. Validation target
```python id="mc16"
y_fold_valid_raw = (
train_df.iloc[valid_idx][TARGET].to_numpy()
)
Selects the **original/raw PCF values** for the validation observations.
This is important because you eventually evaluate predictions on the original PCF scale.
So:
```text id="mc17"
Training target
→ log scale
Validation target
→ raw PCF scale
2. Build Structured Features
```python id="mc18"
X_struct_train, X_struct_valid = (
build_structured_features(
X_fold_train,
y_fold_train_log,
X_fold_valid
)
)
This calls your structured-feature builder.
It processes the current fold's structured data.
The function performs:
```text id="mc19"
Year + Weight
↓
Numeric features
Industry + Protocol + Stage-level availability
↓
One-Hot Encoding
Country + training target
↓
OOF Target Encoding
and returns:
```text id="mc20"
X_struct_train
X_struct_valid
### Why is this inside the fold?
This is a **critical leakage-control mechanism**.
The preprocessing is rebuilt for every fold.
For example:
```text id="mc21"
Fold 1
↓
Fold-1 training data
↓
Fit preprocessing
↓
Transform Fold-1 validation
Fold 2
↓
Fold-2 training data
↓
Fit preprocessing
↓
Transform Fold-2 validation
You are not fitting the structured preprocessing once on the entire development dataset.
Viva answer
“I perform structured feature engineering separately inside each fold because some transformations learn information from the data. Fitting them on the entire development dataset before cross-validation could allow information from the validation fold to influence the transformation.”
3. Select SBERT Features
```python id="mc22"
X_sbert_train = X_train_sbert[train_idx]
X_sbert_valid = X_train_sbert[valid_idx]
You already generated SBERT embeddings for the development dataset.
Here, you simply select the embeddings corresponding to the current fold.
### Training
```python id="mc23"
X_sbert_train = X_train_sbert[train_idx]
Gets the training observations' SBERT embeddings.
Validation
```python id="mc24"
X_sbert_valid = X_train_sbert[valid_idx]
Gets the validation observations' SBERT embeddings.
You are not fitting SBERT here.
It is a **pretrained fixed encoder**.
---
# 4. PCA = 50
```python id="mc25"
(
X_sbert_train_pca,
X_sbert_valid_pca,
_
) = apply_pca_to_sbert(
X_sbert_train,
X_sbert_valid,
n_components=50
)
Now you apply the PCA configuration selected in your previous experiment.
You already compared:
```text id="mc26"
50
100
150
200
and found that:
```text id="mc27"
50 components
performed best among those tested.
Therefore, this final model-comparison experiment fixes:
```python id="mc28"
n_components=50
### What happens inside?
The PCA function does:
```text id="mc29"
Fold-training SBERT
↓
fit PCA
↓
50 components
Fold-validation SBERT
↓
transform using same PCA
↓
50 components
Why is _ used?
```python id="mc30"
_
The PCA function returns:
```text id="mc31"
X_train_pca
X_valid_pca
pca
But here you don't need the fitted PCA object after this operation.
So:
```python id="mc32"
_
means:
> “I intentionally don't need this returned value.”
This is a common Python convention.
---
# 5. Hybrid Feature Fusion
```python id="mc33"
X_hybrid_train = fuse_features(
X_sbert_train_pca,
X_struct_train
)
Combines:
```text id="mc34"
Reduced SBERT features
+
Structured features
into one feature matrix.
Similarly:
```python id="mc35"
X_hybrid_valid = fuse_features(
X_sbert_valid_pca,
X_struct_valid
)
creates the validation hybrid features.
Conceptually:
```text id="mc36"
SBERT
+
Structured
↓
Hybrid
↓
ML algorithm
This is the representation on which all models are compared.
---
# 6. Fresh Model Instance
```python id="mc37"
model = clone(model_template)
This is an important line.
clone() creates a fresh, unfitted copy of the model configuration.
Why is this important?
Because you are repeatedly training the model across five folds.
You do not want the model to carry learned information from a previous fold.
Conceptually:
```text id="mc38"
Fold 1
Fresh model
↓
fit
↓
discard
Fold 2
Fresh model
↓
fit
↓
discard
### Viva question: Why use `clone()`?
> “I use `clone()` to create a fresh unfitted estimator for each fold. This prevents parameters learned in one fold from carrying over into another fold.”
This is a **very good technical point**.
---
# 7. Decide Whether Scaling Is Needed
```python id="mc39"
scale_features = model_name in [
"Linear Regression",
"Ridge Regression",
"SVR"
]
This automatically determines whether the current model should receive scaled features.
If the model is:
```text id="mc40"
Linear Regression
Ridge Regression
SVR
then:
```text id="mc41"
scale_features = True
For:
```text id="mc42"
Random Forest
XGBoost
CatBoost
it becomes:
```text id="mc43"
scale_features = False
Why?
Because feature scaling is more important for models that depend on feature magnitudes, distances, or coefficient regularisation.
Tree-based models generally do not require standardisation.
Strong viva answer
“I enable scaling for the linear and SVR models because these algorithms can be sensitive to feature scale, particularly Ridge and RBF-SVR. Tree-based ensemble models generally do not require standardisation, so I leave scaling disabled for them.”
8. Evaluate the Model
```python id="mc44"
mae, rmse, r2 = evaluate_model(
model=model,
X_train=X_hybrid_train,
y_train_log=y_fold_train_log,
X_valid=X_hybrid_valid,
y_valid_raw=y_fold_valid_raw,
scale_features=scale_features
)
This calls your previously defined `evaluate_model()` function.
The function:
```text id="mc45"
1. Optionally scales features
2. Fits the model
3. Predicts log PCF
4. Converts predictions back to raw PCF
5. Prevents negative predictions
6. Calculates MAE
7. Calculates RMSE
8. Calculates R²
The returned values are:
```text id="mc46"
mae
rmse
r2
---
# Store Fold Results
```python id="mc47"
model_fold_results.append({
"Model": model_name,
"Fold": fold,
"MAE": mae,
"RMSE": rmse,
"R2": r2
})
A dictionary containing the current fold's results is added to the list.
For example:
```text id="mc48"
Model = Random Forest
Fold = 1
MAE = ...
RMSE = ...
R2 = ...
After five folds, you have five records for that model.
---
# Print Fold Performance
```python id="mc49"
print(
f" Fold {fold}: "
f"MAE={mae:.4f}, "
f"RMSE={rmse:.4f}, "
f"R²={r2:.4f}"
)
Displays the performance for the current fold.
:.4f
Means the number is displayed with four decimal places.
For example:
```text id="mc50"
R²=0.6547
This is mainly for readable output.
---
# Store All Folds for the Model
```python id="mc51"
model_results.extend(
model_fold_results
)
Once all five folds for a model have finished, their results are added to the main model_results list.
The process is:
```text id="mc52"
Linear Regression
→ 5 results
↓
model_results
Ridge
→ 5 results
↓
model_results
SVR
→ 5 results
↓
model_results
...
---
# MODEL COMPARISON SUMMARY
Now you convert all those fold-level results into a summary table.
### 1. Convert results into DataFrame
```python id="mc53"
model_results_df = pd.DataFrame(
model_results
)
This converts the list of dictionaries into a pandas DataFrame.
Conceptually:
| Model | Fold | MAE | RMSE | R² |
|---|---|---|---|---|
| Linear Regression | 1 | ... | ... | ... |
| Linear Regression | 2 | ... | ... | ... |
| ... | ... | ... | ... | ... |
| Random Forest | 5 | ... | ... | ... |
2. Group by Model
```python id="mc54"
model_summary = (
model_results_df
.groupby("Model")
Groups all five folds belonging to the same model.
For example:
```text id="mc55"
Random Forest
→ Fold 1
→ Fold 2
→ Fold 3
→ Fold 4
→ Fold 5
3. Calculate Mean MAE
```python id="mc56"
Mean_MAE=("MAE", "mean"),
Calculates the average MAE across the five folds.
Lower is better.
---
# 4. Calculate Mean RMSE
```python id="mc57"
Mean_RMSE=("RMSE", "mean"),
Calculates average RMSE across the five folds.
Lower is better.
5. Calculate Mean R²
```python id="mc58"
Mean_R2=("R2", "mean"),
Calculates average R² across the five folds.
Higher is better.
---
# 6. Calculate Standard Deviation of R²
```python id="mc59"
Std_R2=("R2", "std")
This is an especially useful addition.
It measures how much the R² varies between folds.
For example:
```text id="mc60"
Mean R² = 0.65
Std R² = 0.02
suggests relatively stable fold performance.
But:
```text id="mc61"
Mean R² = 0.65
Std R² = 0.20
would indicate much greater variability.
So Std_R2 helps you understand stability, not just average performance.
7. Reset Index
```python id="mc62"
.reset_index()
Converts the grouped model name back into a normal DataFrame column.
---
# 8. Sort by R²
```python id="mc63"
.sort_values(
"Mean_R2",
ascending=False
)
Sorts models according to mean R².
```text id="mc64"
Highest R²
↓
Best position
because:
```text id="mc65"
ascending=False
means descending order.
Important
This sorting does not mean R² is the only metric that matters.
You should still examine:
- Mean MAE
- Mean RMSE
- Mean R²
- Std R²
A model with the highest R² but dramatically worse MAE/RMSE should be investigated rather than automatically declared best.
9. Display the Summary
```python id="mc66"
display(model_summary)
Displays the final model-comparison table.
Conceptually:
| Model | Mean MAE ↓ | Mean RMSE ↓ | Mean R² ↑ | Std R² ↓ |
| ------- | ---------: | ----------: | --------: | -------: |
| Model A | ... | ... | ... | ... |
| Model B | ... | ... | ... | ... |
| Model C | ... | ... | ... | ... |
This becomes the basis for your model selection.
---
# The Most Important Concept in This Cell
Your model comparison is **fairer than simply training all models once** because every model goes through essentially the same evaluation structure:
```text id="mc67"
Same development data
↓
Same 5 folds
↓
Same PCA = 50
↓
Same hybrid representation
↓
Same target transformation
↓
Same evaluation metrics
↓
Different ML algorithm
Therefore, the major experimental difference is the model algorithm, rather than changing the dataset or evaluation strategy for each model.
That is exactly what you want in a model comparison.
Very Important Viva Question: Why Compare Models on the Same Features?
“To make the comparison fair. I fixed the feature representation at PCA-50 hybrid features and used the same cross-validation folds, target transformation and evaluation metrics for each algorithm. Therefore, differences in performance are primarily attributable to the modelling algorithm rather than different preprocessing choices.”
Excellent answer.
Very Important Viva Question: Why Is PCA Fixed at 50 Here?
“I previously evaluated 50, 100, 150 and 200 components using five-fold cross-validation. Since 50 produced the strongest average performance among those candidates, I fixed PCA at 50 for the subsequent model comparison. This avoids simultaneously changing the PCA dimension and the regression algorithm.”
This is a very important experimental-design answer.
Very Important Viva Question: Is This Data Leakage?
Your answer should distinguish the different stages.
“The independent test set is not used in this model comparison. Within each cross-validation fold, the structured preprocessing and PCA are fitted only on the fold-training data and then applied to the validation fold. The regression model is also freshly fitted for each fold. This prevents validation information from influencing model training or learned preprocessing.”
Your leakage-control pipeline
```text id="mc68"
Fold training
↓
Fit winsorization
Fit OHE
Fit target encoding
Fit PCA
Fit scaler if required
Fit ML model
↓
Validation
↓
Transform only
↓
Predict
That is one of the strongest methodological points in your notebook.
---
# Very Important Viva Question: Why `clone()`?
> “Because each fold must start with an unfitted model. `clone()` creates a fresh estimator with the same hyperparameter configuration, preventing learned parameters from one fold carrying into another fold.”
---
# Very Important Viva Question: Why Do You Use Raw Target for Metrics?
> “Although the models are trained on the log-transformed target, I inverse-transform the predictions back to the original PCF scale before calculating MAE, RMSE and R². This makes the evaluation directly interpretable in the original target scale.”
---
# Very Important Viva Question: Why Not Use the Independent Test Set Here?
> “Because the test set should remain untouched during model and feature-selection decisions. I use cross-validation within the development data for model comparison and reserve the independent test set for the final unbiased evaluation.”
This is perhaps one of the **most important answers in your whole viva**.
---
# One Subtle Point You Should Understand
Your code uses:
```python id="mc69"
y_fold_train_log
for model training, while:
```python id="mc70"
target_bins
is used for **stratification**.
These are not the same thing.
```text id="mc71"
y_fold_train_log
→ actual training target
target_bins
→ temporary labels used only to create balanced folds
The model never learns to predict target_bins.
The prediction task remains:
```text id="mc72"
Hybrid features
↓
Continuous PCF prediction
---
# What This Experiment Ultimately Answers
This cell answers:
> **“Given the same PCA-50 hybrid representation and the same evaluation procedure, which regression algorithm performs best for PCF prediction?”**
Your previous PCA experiment answered:
> **“How many SBERT components should I retain?”**
So your experimental logic is:
```text id="mc73"
Experiment 1
PCA = 50 / 100 / 150 / 200
↓
Select best PCA dimension
↓
50
Experiment 2
Fix PCA = 50
↓
Compare ML algorithms
↓
Select best model
That is a much stronger explanation than simply saying:
“I tried different models.”
Presentation Script
“After selecting 50 PCA components from the previous cross-validation experiment, I use that configuration for the final model comparison. I evaluate multiple regression algorithms using stratified five-fold cross-validation on the development data. For each fold, I first separate the training and validation observations and build the structured features using only the fold-training information. I then select the corresponding SBERT embeddings, fit PCA with 50 components only on the fold-training embeddings, and transform the validation embeddings using the same PCA transformation. The reduced SBERT features are then fused with the structured features to form the hybrid representation.
For each model, I create a fresh estimator using
clone()so that models do not carry learned information between folds. Feature scaling is enabled for Linear Regression, Ridge and SVR, while it is disabled for the tree-based ensemble models. Each model is trained on the log-transformed PCF target, and predictions are converted back to the original PCF scale before calculating MAE, RMSE and R². Finally, I average the metrics across the five folds and also calculate the standard deviation of R² to assess performance stability. The models are then ranked by mean R² while considering MAE and RMSE as complementary performance measures.”
The 10 lines you should know perfectly for viva
```text id="mc74"
skf.split(...)
→ creates the five train/validation foldsbuild_structured_features(...)
→ creates leakage-controlled structured featuresX_train_sbert[train_idx]
→ selects SBERT features for the training foldapply_pca_to_sbert(..., n_components=50)
→ reduces SBERT dimensions using training-fold-fitted PCAfuse_features(...)
→ combines SBERT + structured featuresclone(model_template)
→ creates a fresh model for each foldscale_features = ...
→ scaling depends on the algorithmevaluate_model(...)
→ trains, predicts and calculates metricsgroupby("Model").agg(...)
→ averages performance across five folds-
sort_values("Mean_R2", ascending=False)
→ ranks models by average R²
The one-sentence summary
“I fixed the selected PCA dimension at 50 and compared different regression algorithms using the same leakage-controlled hybrid features and stratified five-fold development evaluation, selecting the strongest model based on cross-validated predictive performance.”
Top comments (0)