# ============================================================
# TUNED XGBOOST — OOF DEVELOPMENT EVALUATION
#
# Target = log1p(PCF)
# Country = OOF Target Encoding
# SBERT = PCA 50
# CV = Stratified 5-Fold
# ============================================================
BEST_XGB_PARAMS = {
"n_estimators": 200,
"learning_rate": 0.08,
"max_depth": 6,
"min_child_weight": 3,
"subsample": 0.7,
"colsample_bytree": 0.8
}
xgb_oof_results = []
for fold, (train_idx, valid_idx) in enumerate(
skf.split(train_df, target_bins),
start=1
):
print(f"Running fold {fold}/5...")
# --------------------------------------------------------
# 1. Fold data
# --------------------------------------------------------
X_fold_train = (
train_df.iloc[train_idx].copy()
)
X_fold_valid = (
train_df.iloc[valid_idx].copy()
)
# --------------------------------------------------------
# 2. Log target
# --------------------------------------------------------
y_fold_train_log = (
y_log.iloc[train_idx].to_numpy()
)
y_fold_valid_raw = (
train_df.iloc[valid_idx][TARGET].to_numpy()
)
# --------------------------------------------------------
# 3. Structured features
# Country = OOF Target Encoding
# --------------------------------------------------------
X_struct_train, X_struct_valid = (
build_structured_features(
X_fold_train,
y_fold_train_log,
X_fold_valid
)
)
# --------------------------------------------------------
# 4. SBERT features
# --------------------------------------------------------
X_sbert_train = (
X_train_sbert[train_idx]
)
X_sbert_valid = (
X_train_sbert[valid_idx]
)
# --------------------------------------------------------
# 5. PCA = 50
# Fit only on outer training fold
# --------------------------------------------------------
(
X_sbert_train_pca,
X_sbert_valid_pca,
_
) = apply_pca_to_sbert(
X_sbert_train,
X_sbert_valid,
n_components=50
)
# --------------------------------------------------------
# 6. Hybrid features
# --------------------------------------------------------
X_hybrid_train = fuse_features(
X_sbert_train_pca,
X_struct_train
)
X_hybrid_valid = fuse_features(
X_sbert_valid_pca,
X_struct_valid
)
# --------------------------------------------------------
# 7. Tuned XGBoost
# --------------------------------------------------------
model = XGBRegressor(
objective="reg:squarederror",
random_state=42,
n_jobs=-1,
**BEST_XGB_PARAMS
)
# --------------------------------------------------------
# 8. Train
# --------------------------------------------------------
model.fit(
X_hybrid_train,
y_fold_train_log
)
# --------------------------------------------------------
# 9. Predict
# --------------------------------------------------------
y_pred_log = model.predict(
X_hybrid_valid
)
# Convert back to original PCF scale
y_pred_raw = np.expm1(
y_pred_log
)
y_pred_raw = np.maximum(
y_pred_raw,
0
)
# --------------------------------------------------------
# 10. Metrics
# --------------------------------------------------------
mae = mean_absolute_error(
y_fold_valid_raw,
y_pred_raw
)
rmse = np.sqrt(
mean_squared_error(
y_fold_valid_raw,
y_pred_raw
)
)
r2 = r2_score(
y_fold_valid_raw,
y_pred_raw
)
xgb_oof_results.append({
"Fold": fold,
"MAE": mae,
"RMSE": rmse,
"R2": r2
})
print(
f" MAE={mae:.4f}, "
f"RMSE={rmse:.4f}, "
f"R²={r2:.4f}"
)
# ============================================================
# OOF SUMMARY
# ============================================================
xgb_oof_results_df = pd.DataFrame(
xgb_oof_results
)
xgb_oof_summary = pd.DataFrame({
"Mean_MAE": [
xgb_oof_results_df["MAE"].mean()
],
"Mean_RMSE": [
xgb_oof_results_df["RMSE"].mean()
],
"Mean_R2": [
xgb_oof_results_df["R2"].mean()
],
"Std_R2": [
xgb_oof_results_df["R2"].std()
]
})
display(xgb_oof_results_df)
display(xgb_oof_summary)
Tuned XGBoost — OOF Development Evaluation
This cell is the next step after hyperparameter tuning.
The previous cell asked:
“Among the 20 randomly sampled XGBoost configurations, which configuration performs best?”
This cell takes the selected configuration:
BEST_XGB_PARAMS = {
"n_estimators": 200,
"learning_rate": 0.08,
"max_depth": 6,
"min_child_weight": 3,
"subsample": 0.7,
"colsample_bytree": 0.8
}
and evaluates it again using 5-fold cross-validation on the development dataset.
The important point is:
This cell is not searching anymore. It is evaluating the selected/tuned XGBoost configuration across all five development folds.
1. Selected XGBoost parameters
```python id="n8n1qd"
BEST_XGB_PARAMS = {
"n_estimators": 200,
"learning_rate": 0.08,
"max_depth": 6,
"min_child_weight": 3,
"subsample": 0.7,
"colsample_bytree": 0.8
}
These are the hyperparameters selected from the previous tuning experiment.
The values mean:
| Parameter | Value | Meaning |
| ------------------ | ----: | ---------------------------------------- |
| `n_estimators` | 200 | Number of boosting trees |
| `learning_rate` | 0.08 | Contribution/shrinkage of each tree |
| `max_depth` | 6 | Maximum depth of each tree |
| `min_child_weight` | 3 | Makes splitting more conservative |
| `subsample` | 0.7 | Uses 70% of rows for each boosting stage |
| `colsample_bytree` | 0.8 | Uses 80% of features for each tree |
### Viva point
Do **not** say:
> “These are the globally optimal parameters.”
Say:
> **“These were the best-performing parameters among the 20 randomly sampled configurations evaluated during tuning.”**
That is scientifically safer.
---
### 2. Create result storage
```python id="psjq8j"
xgb_oof_results = []
Creates an empty list to store the performance of the tuned XGBoost model for each fold.
After the loop, there will be five results:
```text id="d5cy1c"
Fold 1
Fold 2
Fold 3
Fold 4
Fold 5
---
### 3. Start 5-fold evaluation
```python id="bqj4kq"
for fold, (train_idx, valid_idx) in enumerate(
skf.split(train_df, target_bins),
start=1
):
This uses the same Stratified 5-Fold CV strategy defined earlier.
For each fold:
```text id="qlz07h"
Development data
↓
┌───────────────┐
│ Training fold │
└───────────────┘
+
┌────────────────┐
│ Validation fold│
└────────────────┘
This happens five times.
### Why `target_bins`?
Because the original PCF target is continuous.
You created approximate quantile bins earlier so that each fold has a reasonably similar distribution of the target.
---
### 4. Print fold progress
```python id="4sfy9f"
print(f"Running fold {fold}/5...")
Just shows which fold is currently being processed.
Fold-level preprocessing
5. Separate training and validation data
```python id="f3k5wo"
X_fold_train = (
train_df.iloc[train_idx].copy()
)
X_fold_valid = (
train_df.iloc[valid_idx].copy()
)
The current fold is divided into:
* training portion
* validation portion
The validation portion is not used to fit the model.
This is important for estimating generalisation performance.
---
### 6. Training target
```python id="s0u2tx"
y_fold_train_log = (
y_log.iloc[train_idx].to_numpy()
)
The model is trained using:
```text id="q1x0kd"
log1p(PCF)
rather than raw PCF.
---
### 7. Validation target
```python id="s2r8c5"
y_fold_valid_raw = (
train_df.iloc[valid_idx][TARGET].to_numpy()
)
The validation target remains in the original PCF scale.
So the process is:
```text id="u8p5q6"
Training:
log1p(PCF)
Validation:
original PCF
This is intentional.
---
# Structured features
### 8. Build structured features
```python id="n7n8gq"
X_struct_train, X_struct_valid = (
build_structured_features(
X_fold_train,
y_fold_train_log,
X_fold_valid
)
)
This generates the structured feature representation.
It includes:
- Year
- Product weight
- Industry
- Protocol
- Stage-level CO₂e availability
- Country target encoding
Most importantly, the country encoding is OOF target encoded for the training portion.
This is important because country encoding uses the target variable.
SBERT representation
9. Select SBERT features
```python id="e9o6r2"
X_sbert_train = (
X_train_sbert[train_idx]
)
X_sbert_valid = (
X_train_sbert[valid_idx]
)
The SBERT embeddings corresponding to the current training and validation observations are selected.
SBERT itself was already generated earlier.
You are not retraining SBERT here.
---
# PCA
### 10. Reduce SBERT to 50 dimensions
```python id="6q6u9m"
(
X_sbert_train_pca,
X_sbert_valid_pca,
_
) = apply_pca_to_sbert(
X_sbert_train,
X_sbert_valid,
n_components=50
)
This applies the selected PCA configuration.
```text id="l0xk8b"
SBERT embedding
↓
PCA
↓
50 dimensions
### Very important line
```python id="q8h5tq"
X_sbert_train,
X_sbert_valid
are passed separately.
Inside apply_pca_to_sbert():
```python id="w7w3xj"
pca.fit_transform(X_train_sbert)
pca.transform(X_valid_sbert)
Therefore:
> **PCA is fitted only on the training fold and then applied to the validation fold.**
This prevents validation information from influencing the PCA transformation.
---
# Hybrid features
### 11. Combine SBERT and structured features
```python id="a3x0j4"
X_hybrid_train = fuse_features(
X_sbert_train_pca,
X_struct_train
)
and:
```python id="6e4jpv"
X_hybrid_valid = fuse_features(
X_sbert_valid_pca,
X_struct_valid
)
The model gets both representations:
```text id="qk2h9w"
Semantic text information
+
Structured information
↓
Hybrid feature representation
This is the core idea of the hybrid model.
Tuned XGBoost
12. Create the model
```python id="y8zq2a"
model = XGBRegressor(
objective="reg:squarederror",
random_state=42,
n_jobs=-1,
**BEST_XGB_PARAMS
)
Creates an XGBoost regression model using the selected parameters.
The:
```python id="t7jv1z"
**BEST_XGB_PARAMS
inserts all six tuned hyperparameters.
So the model is effectively configured with:
```text id="x3v0ms"
200 trees
learning rate = 0.08
depth = 6
min child weight = 3
row sampling = 70%
feature sampling = 80%
---
### 13. Train
```python id="c5f4vq"
model.fit(
X_hybrid_train,
y_fold_train_log
)
XGBoost learns:
```text id="z6o9wa"
Hybrid features
↓
log1p(PCF)
using only the training fold.
---
# Prediction
### 14. Predict validation fold
```python id="w0p3z8"
y_pred_log = model.predict(
X_hybrid_valid
)
The predictions are initially on the log-transformed scale.
15. Convert predictions back
```python id="b6v3h9"
y_pred_raw = np.expm1(
y_pred_log
)
Because training used:
```python
np.log1p(PCF)
the inverse transformation is:
np.expm1()
Therefore:
```text id="q9q4t1"
log1p(PCF)
↓
model
↓
predicted log1p(PCF)
↓
expm1
↓
predicted PCF
---
### 16. Enforce non-negative predictions
```python id="f7q2k3"
y_pred_raw = np.maximum(
y_pred_raw,
0
)
Any negative prediction is replaced by zero.
This makes the prediction consistent with the physical interpretation of PCF.
Metrics
17. MAE
```python id="x0k8s2"
mae = mean_absolute_error(
y_fold_valid_raw,
y_pred_raw
)
Measures average absolute prediction error.
**Lower is better.**
---
### 18. RMSE
```python id="j3m6v9"
rmse = np.sqrt(
mean_squared_error(
y_fold_valid_raw,
y_pred_raw
)
)
RMSE penalises large errors more heavily than MAE.
Lower is better.
19. R²
```python id="p4r7d1"
r2 = r2_score(
y_fold_valid_raw,
y_pred_raw
)
Measures predictive explanatory performance relative to a baseline.
**Higher is better.**
---
### 20. Store each fold
```python id="w5c2n8"
xgb_oof_results.append({
"Fold": fold,
"MAE": mae,
"RMSE": rmse,
"R2": r2
})
Stores the three metrics for each fold.
Your actual results
You obtained:
| Fold | MAE | RMSE | R² |
|---|---|---|---|
| 1 | 5,195.20 | 41,449.90 | 0.8475 |
| 2 | 31,455.28 | 260,513.71 | 0.4092 |
| 3 | 1,029.55 | 2,780.88 | 0.8705 |
| 4 | 11,496.24 | 116,050.04 | 0.8254 |
| 5 | 2,811.77 | 14,114.92 | 0.5251 |
| Mean | 10,397.61 | 86,981.89 | 0.6955 |
And:
```text id="v4u7n2"
Std R² = 0.2131
---
# How to interpret these results
The average performance is:
### Mean MAE
**10,397.61 kg CO₂e**
On average, the absolute prediction error was approximately **10,398 kg CO₂e** across the five validation folds.
### Mean RMSE
**86,981.89 kg CO₂e**
The much larger RMSE compared with MAE tells you that some predictions have **very large errors/outliers**.
This is an important observation.
### Mean R²
**0.6955**
The model achieved an average R² of approximately **0.696** across the five folds.
So the model demonstrates reasonably strong predictive performance overall, but performance is not equally stable across all folds.
---
# The most important result: Fold variation
Look carefully at your R²:
```text
Fold 1 → 0.8475
Fold 2 → 0.4092
Fold 3 → 0.8705
Fold 4 → 0.8254
Fold 5 → 0.5251
There is substantial variation.
That is reflected by:
Std R² = 0.2131
This is not something you should hide in the viva.
Instead, use it as a limitation/interpretation.
A strong answer is:
“The mean R² was approximately 0.696, but the standard deviation was about 0.213, indicating noticeable variation across folds. This suggests that model performance depends on the composition of the validation subset, which may reflect heterogeneity or extreme observations in the PCF dataset.”
That is much more defensible than saying:
“My model achieved 69.5% accuracy.”
Do not call R² accuracy.
Why is Fold 2 so bad?
Fold 2 has:
MAE = 31,455
RMSE = 260,514
R² = 0.409
while Fold 3 has:
MAE = 1,030
RMSE = 2,781
R² = 0.871
That is a huge difference.
A reasonable interpretation is:
“The PCF dataset is heterogeneous and likely contains extreme or difficult observations. A validation fold containing influential high-value observations or patterns poorly represented in its training portion can produce much larger errors.”
But do not claim that Fold 2 definitely contains outliers unless you actually inspect the fold data.
If the examiner asks:
“Why exactly was Fold 2 poor?”
The safest answer is:
“The results indicate that Fold 2 contains observations that were harder for the model to predict, but I would need to inspect the actual observations and residuals in that fold to identify the exact cause. I would not attribute it definitively to outliers without that analysis.”
That's a very strong viva answer because you acknowledge what the experiment shows without inventing a cause.
Why is RMSE so much higher than MAE?
Your:
Mean MAE ≈ 10,398
Mean RMSE ≈ 86,982
RMSE is dramatically higher.
That suggests the model has some very large individual errors.
Why?
Because RMSE squares errors before averaging:
error
↓
error²
↓
average
↓
square root
Therefore, a few very large errors can dominate RMSE.
Viva answer
“The substantially higher RMSE compared with MAE indicates that although the typical absolute error is lower, there are some very large prediction errors. RMSE is sensitive to those large errors because it squares the residuals.”
What does “OOF Development Evaluation” mean?
This can easily confuse you.
Here, OOF does not mean you are producing one prediction for every observation and using those predictions as the final model output.
In this context, you are using out-of-fold validation within the development dataset.
Each observation acts as validation data in one fold and training data in the other four folds.
For example:
Fold 1:
80% → train
20% → validation
Fold 2:
80% → train
20% → validation
...
Fold 5:
80% → train
20% → validation
Together, every development observation gets evaluated in a fold where it was not used for training.
Very important: this is NOT your final test performance
You should be very clear about this.
Your workflow is:
Original dataset
↓
80% Development 20% Independent Test
↓ ↓
5-fold CV KEEP UNTOUCHED
↓
PCA selection
↓
Model comparison
↓
XGBoost tuning
↓
Selected XGBoost
↓
Final model
↓
Evaluate ONCE
↓
Independent test performance
Therefore, the:
Mean R² = 0.6955
is development cross-validation performance, not your final independent test-set R².
If you have a later cell that trains the final model on the full development set and evaluates it on test_df, that test result is the one you should report as your final generalisation result.
Why repeat the evaluation after tuning?
The previous tuning cell already calculated CV performance for the 20 configurations.
So an examiner might ask:
“Why are you doing another 5-fold evaluation?”
Answer:
“The tuning stage used cross-validation to compare candidate configurations. After selecting the best configuration, I reran the fixed configuration across the five development folds to obtain a clean fold-level summary of its performance and stability. The independent test set remains untouched for the final evaluation.”
This is a reasonable explanation.
However, be precise: the CV estimate after selecting the best configuration is not fully unbiased for model-selection performance, because the configuration was selected using the same development CV process. The truly independent estimate comes from the untouched test set.
If challenged:
“The post-selection CV result is useful for describing development performance and fold stability, but I treat the untouched test set as the final unbiased evaluation because the development CV was involved in model selection.”
That is an excellent research-methodology answer.
One-minute viva explanation
If your supervisor asks you to explain the whole cell:
“After tuning XGBoost, I selected the best-performing configuration from the 20 randomly sampled parameter sets. I then evaluated that fixed configuration using stratified 5-fold cross-validation on the development dataset. In each fold, I rebuilt the structured features with leakage-aware OOF country target encoding, generated the hybrid representation by combining structured features with SBERT embeddings reduced to 50 PCA components, and trained XGBoost on the log-transformed PCF target. Predictions were converted back to the original PCF scale using
expm1, and I calculated MAE, RMSE and R². The average R² was 0.6955, with an R² standard deviation of 0.2131, showing reasonable overall predictive performance but noticeable variation across folds. The independent test set was not used in this evaluation.”
The 5 numbers you should remember
Best XGBoost:
n_estimators = 200
learning_rate = 0.08
max_depth = 6
min_child_weight = 3
subsample = 0.7
colsample_bytree = 0.8
Development CV:
Mean MAE = 10,397.61
Mean RMSE = 86,981.89
Mean R² = 0.6955
Std R² = 0.2131
And the single most important conceptual sentence:
“The 0.6955 R² is a 5-fold development cross-validation result after selecting the XGBoost configuration; it is not the final independent test-set performance.”
For your XGBoost evaluation
- Check whether Fold 2 contains influential outliers
- Compare pooled OOF metrics with mean fold metrics
Top comments (0)