# ============================================================
# XGBOOST HYPERPARAMETER TUNING
#
# Fixed:
# Target = log1p(PCF)
# Country = OOF Target Encoding
# SBERT = PCA 50
# CV = Stratified 5-Fold
#
# Only XGBoost hyperparameters change
# ============================================================
from sklearn.model_selection import ParameterSampler
# ============================================================
# 1. Generate 20 parameter combinations
# ============================================================
xgb_param_candidates = list(
ParameterSampler(
xgb_param_grid,
n_iter=20,
random_state=42
)
)
print(
f"Testing {len(xgb_param_candidates)} XGBoost parameter sets."
)
# ============================================================
# 2. Storage
# ============================================================
xgb_tuning_results = []
# ============================================================
# 3. Parameter search
# ============================================================
for param_number, params in enumerate(
xgb_param_candidates,
start=1
):
print(
f"\nParameter set "
f"{param_number}/{len(xgb_param_candidates)}"
)
fold_results = []
# ========================================================
# 4. Stratified 5-Fold CV
# ========================================================
for fold, (train_idx, valid_idx) in enumerate(
skf.split(train_df, target_bins),
start=1
):
# ----------------------------------------------------
# Fold data
# ----------------------------------------------------
X_fold_train = (
train_df.iloc[train_idx].copy()
)
X_fold_valid = (
train_df.iloc[valid_idx].copy()
)
# ----------------------------------------------------
# Log target
# ----------------------------------------------------
y_fold_train_log = (
y_log.iloc[train_idx].to_numpy()
)
y_fold_valid_raw = (
train_df.iloc[valid_idx][TARGET].to_numpy()
)
# ----------------------------------------------------
# Structured features
# Country = OOF Target Encoding
# ----------------------------------------------------
X_struct_train, X_struct_valid = (
build_structured_features(
X_fold_train,
y_fold_train_log,
X_fold_valid
)
)
# ----------------------------------------------------
# SBERT features
# ----------------------------------------------------
X_sbert_train = (
X_train_sbert[train_idx]
)
X_sbert_valid = (
X_train_sbert[valid_idx]
)
# ----------------------------------------------------
# PCA = 50
# PCA fitted only on training fold
# ----------------------------------------------------
(
X_sbert_train_pca,
X_sbert_valid_pca,
_
) = apply_pca_to_sbert(
X_sbert_train,
X_sbert_valid,
n_components=50
)
# ----------------------------------------------------
# Hybrid feature fusion
# ----------------------------------------------------
X_hybrid_train = fuse_features(
X_sbert_train_pca,
X_struct_train
)
X_hybrid_valid = fuse_features(
X_sbert_valid_pca,
X_struct_valid
)
# ----------------------------------------------------
# Fresh XGBoost model
# ----------------------------------------------------
model = XGBRegressor(
objective="reg:squarederror",
random_state=42,
n_jobs=-1,
**params
)
# ----------------------------------------------------
# Train
# ----------------------------------------------------
model.fit(
X_hybrid_train,
y_fold_train_log
)
# ----------------------------------------------------
# Predict
# ----------------------------------------------------
y_pred_log = model.predict(
X_hybrid_valid
)
y_pred_raw = np.expm1(
y_pred_log
)
y_pred_raw = np.maximum(
y_pred_raw,
0
)
# ----------------------------------------------------
# Metrics
# ----------------------------------------------------
mae = mean_absolute_error(
y_fold_valid_raw,
y_pred_raw
)
rmse = np.sqrt(
mean_squared_error(
y_fold_valid_raw,
y_pred_raw
)
)
r2 = r2_score(
y_fold_valid_raw,
y_pred_raw
)
fold_results.append({
"Fold": fold,
"MAE": mae,
"RMSE": rmse,
"R2": r2
})
# ========================================================
# 5. Average CV performance
# ========================================================
fold_results_df = pd.DataFrame(
fold_results
)
mean_mae = fold_results_df["MAE"].mean()
mean_rmse = fold_results_df["RMSE"].mean()
mean_r2 = fold_results_df["R2"].mean()
std_r2 = fold_results_df["R2"].std()
# ========================================================
# 6. Store results
# ========================================================
result = {
"Parameter_Set": param_number,
"Mean_MAE": mean_mae,
"Mean_RMSE": mean_rmse,
"Mean_R2": mean_r2,
"Std_R2": std_r2
}
result.update(params)
xgb_tuning_results.append(
result
)
print(
f" Mean R²: {mean_r2:.4f} | "
f"Mean RMSE: {mean_rmse:.2f}"
)
# ============================================================
# XGBOOST TUNING SUMMARY
# ============================================================
xgb_tuning_results_df = pd.DataFrame(
xgb_tuning_results
)
xgb_tuning_results_df = (
xgb_tuning_results_df
.sort_values(
"Mean_R2",
ascending=False
)
.reset_index(drop=True)
)
display(
xgb_tuning_results_df
)
XGBoost Hyperparameter Tuning — Overall Purpose
This cell performs randomized hyperparameter tuning of XGBoost.
The important idea is:
Everything in the modelling pipeline is kept fixed, and only the XGBoost hyperparameters are changed.
Fixed components:
- Target →
log1p(PCF) - Country → OOF target encoding
- SBERT → PCA with 50 components
- Validation → Stratified 5-fold CV
- Features → SBERT + structured features
- Metrics → MAE, RMSE, R²
So this experiment specifically answers:
“Given the selected hybrid feature representation, which XGBoost hyperparameter configuration performs best?”
1. Import ParameterSampler
from sklearn.model_selection import ParameterSampler
ParameterSampler is used to randomly select combinations of hyperparameters from the search space.
You previously defined:
xgb_param_grid = {
"n_estimators": [200, 300, 500],
"learning_rate": [0.03, 0.05, 0.08],
"max_depth": [3, 4, 6],
"min_child_weight": [1, 3, 5],
"subsample": [0.7, 0.8, 1.0],
"colsample_bytree": [0.7, 0.8, 1.0]
}
There are:
3 × 3 × 3 × 3 × 3 × 3 = 729 possible combinations.
Instead of testing all 729, you randomly test only 20.
2. Generate 20 parameter combinations
xgb_param_candidates = list(
ParameterSampler(
xgb_param_grid,
n_iter=20,
random_state=42
)
)
ParameterSampler(...)
This randomly samples combinations from xgb_param_grid.
n_iter=20
This means:
Test only 20 randomly selected hyperparameter combinations.
If you used all combinations, there would be 729.
So this reduces computation considerably.
random_state=42
Makes the random selection reproducible.
If you run the notebook again with the same search space and seed, you should get the same 20 sampled combinations.
list(...)
Converts the sampler output into a list so that you can iterate over it multiple times.
3. Print number of combinations
print(
f"Testing {len(xgb_param_candidates)} XGBoost parameter sets."
)
This simply tells you how many combinations will actually be tested.
Expected:
Testing 20 XGBoost parameter sets.
4. Create result storage
xgb_tuning_results = []
An empty list is created to store the performance of each parameter configuration.
Eventually it will contain approximately:
20 parameter sets
with their:
- Mean MAE
- Mean RMSE
- Mean R²
- Std R²
- hyperparameter values
5. Start parameter search
for param_number, params in enumerate(
xgb_param_candidates,
start=1
):
This loops through the 20 parameter combinations.
For example, one iteration might contain:
{
"n_estimators": 300,
"learning_rate": 0.05,
"max_depth": 4,
...
}
param_number gives the configuration number:
1
2
3
...
20
params contains the actual hyperparameter values.
6. Display current parameter set
print(
f"\nParameter set "
f"{param_number}/{len(xgb_param_candidates)}"
)
This gives progress information.
For example:
Parameter set 7/20
It does not affect the model.
7. Storage for fold results
fold_results = []
For the current hyperparameter configuration, this list stores the performance from each of the 5 folds.
So one parameter set produces:
Fold 1
Fold 2
Fold 3
Fold 4
Fold 5
Then those five results are averaged.
8. Stratified 5-Fold Cross-Validation
for fold, (train_idx, valid_idx) in enumerate(
skf.split(train_df, target_bins),
start=1
):
This is very important.
For each XGBoost parameter set, you run the model through the same 5 CV folds.
Therefore:
20 parameter sets × 5 folds = 100 XGBoost training runs.
This is the main computational cost of the tuning.
Why use the same folds?
For fair comparison.
Every hyperparameter configuration sees the same training/validation partitions.
Therefore, differences in performance are more likely to come from the hyperparameters rather than different data splits.
9. Create fold datasets
X_fold_train = (
train_df.iloc[train_idx].copy()
)
X_fold_valid = (
train_df.iloc[valid_idx].copy()
)
The indices generated by skf are used to separate the development dataset into:
- fold training data
- fold validation data
.copy() creates independent DataFrames.
The independent test set is still untouched.
10. Prepare training target
y_fold_train_log = (
y_log.iloc[train_idx].to_numpy()
)
The training target is the log-transformed PCF.
Earlier:
y_log = np.log1p(data[TARGET])
So XGBoost learns:
log1p(PCF)
rather than raw PCF.
This helps reduce the influence of extreme values and skew during model training.
11. Prepare validation target
y_fold_valid_raw = (
train_df.iloc[valid_idx][TARGET].to_numpy()
)
Notice something important:
The validation target is kept in the original PCF scale.
So:
Training → log scale
Evaluation → original PCF scale
This allows the final metrics to have meaningful PCF units.
12. Build structured features
X_struct_train, X_struct_valid = (
build_structured_features(
X_fold_train,
y_fold_train_log,
X_fold_valid
)
)
This creates the structured feature representation.
It includes:
- numeric features
- one-hot encoded categorical features
- country target encoding
The important point is that preprocessing is rebuilt inside every fold.
That helps prevent data leakage.
In particular, Country target encoding uses:
y_fold_train_log
and generates OOF encodings for the training portion.
13. Select SBERT features for the fold
X_sbert_train = (
X_train_sbert[train_idx]
)
X_sbert_valid = (
X_train_sbert[valid_idx]
)
The SBERT embeddings corresponding to the current fold are selected.
You already generated SBERT embeddings earlier.
You are not retraining SBERT here.
You are simply selecting the relevant rows.
14. Apply PCA = 50
(
X_sbert_train_pca,
X_sbert_valid_pca,
_
) = apply_pca_to_sbert(
X_sbert_train,
X_sbert_valid,
n_components=50
)
This reduces the SBERT representation to 50 dimensions.
The important part is:
n_components=50
Why 50?
Because your earlier PCA experiment compared:
50
100
150
200
and 50 performed best among those tested options.
Therefore, 50 is now fixed during XGBoost tuning.
Very important viva point
You are not tuning PCA and XGBoost simultaneously here.
The experimental sequence is:
PCA comparison
↓
Select PCA = 50
↓
Model comparison
↓
Choose/tune XGBoost
This makes the experiment easier to interpret.
15. Hybrid feature fusion
X_hybrid_train = fuse_features(
X_sbert_train_pca,
X_struct_train
)
and:
X_hybrid_valid = fuse_features(
X_sbert_valid_pca,
X_struct_valid
)
The two feature types are horizontally concatenated:
SBERT/PCA features
+
Structured features
↓
Hybrid features
So XGBoost receives both:
- semantic information from text
- explicit structured information such as year, weight, industry, protocol, country, etc.
16. Create a fresh XGBoost model
model = XGBRegressor(
objective="reg:squarederror",
random_state=42,
n_jobs=-1,
**params
)
This creates a new XGBoost model for every parameter configuration and every fold.
objective="reg:squarederror"
Specifies that this is a regression problem using squared-error loss.
random_state=42
Makes the model's stochastic behaviour reproducible where applicable.
n_jobs=-1
Uses all available CPU cores for parallel computation.
**params
This is particularly important.
It inserts the sampled hyperparameters into the model.
For example:
**params
might effectively become:
n_estimators=500,
learning_rate=0.03,
max_depth=4,
...
So each parameter set creates a different XGBoost configuration.
17. Train XGBoost
model.fit(
X_hybrid_train,
y_fold_train_log
)
XGBoost learns the relationship between:
Hybrid features
↓
log1p(PCF)
Only the training fold is used here.
The validation fold is not used for fitting.
18. Predict validation data
y_pred_log = model.predict(
X_hybrid_valid
)
The model predicts PCF on the log scale.
So these predictions represent:
predicted log1p(PCF)
19. Convert predictions back to original scale
y_pred_raw = np.expm1(
y_pred_log
)
expm1 is the inverse of:
np.log1p()
So:
log1p(PCF)
↓
expm1()
↓
PCF
Now predictions are back in the original PCF scale.
20. Prevent negative predictions
y_pred_raw = np.maximum(
y_pred_raw,
0
)
PCF cannot logically be negative.
This line replaces any negative prediction with zero.
For example:
-5 → 0
20 → 20
100 → 100
This enforces a physically meaningful non-negative prediction.
21. Calculate MAE
mae = mean_absolute_error(
y_fold_valid_raw,
y_pred_raw
)
MAE = Mean Absolute Error.
It measures the average absolute difference between:
Actual PCF
vs
Predicted PCF
Lower is better.
Because this is calculated on the raw scale, its unit is approximately:
kg CO₂e
22. Calculate RMSE
rmse = np.sqrt(
mean_squared_error(
y_fold_valid_raw,
y_pred_raw
)
)
RMSE = Root Mean Squared Error.
It penalises large errors more strongly than MAE.
Lower is better.
This is especially useful when you want to identify whether the model makes some very large prediction errors.
23. Calculate R²
r2 = r2_score(
y_fold_valid_raw,
y_pred_raw
)
R² measures how much of the variation in the target is explained by the model relative to a baseline.
Higher is generally better.
For example:
R² = 0.70
means the model explains substantial variation in the validation data, relative to the baseline, but does not mean 70% of every individual prediction is correct.
24. Store fold performance
fold_results.append({
"Fold": fold,
"MAE": mae,
"RMSE": rmse,
"R2": r2
})
The results for the current fold are stored.
After five folds, you have five sets of:
MAE
RMSE
R²
for that particular hyperparameter configuration.
25. Average the 5 folds
fold_results_df = pd.DataFrame(
fold_results
)
Converts the fold results into a DataFrame.
Then:
mean_mae = fold_results_df["MAE"].mean()
calculates average MAE across the five folds.
mean_rmse = fold_results_df["RMSE"].mean()
calculates average RMSE.
mean_r2 = fold_results_df["R2"].mean()
calculates average R².
And:
std_r2 = fold_results_df["R2"].std()
calculates the standard deviation of R² across folds.
Why Std_R2?
It gives an indication of performance stability across folds.
For example:
Mean R² = 0.70
Std R² = 0.02
suggests relatively consistent fold performance.
Whereas:
Mean R² = 0.70
Std R² = 0.20
suggests much greater variation between folds.
26. Store the parameter-set result
result = {
"Parameter_Set": param_number,
"Mean_MAE": mean_mae,
"Mean_RMSE": mean_rmse,
"Mean_R2": mean_r2,
"Std_R2": std_r2
}
This creates one summary record for one hyperparameter configuration.
Then:
result.update(params)
adds the actual hyperparameter values.
So one row eventually looks conceptually like:
| Parameter Set | Mean MAE | Mean RMSE | Mean R² | Std R² | n_estimators | learning_rate | max_depth |
|---|---|---|---|---|---|---|---|
| 1 | ... | ... | ... | ... | 300 | 0.05 | 4 |
27. Save result
xgb_tuning_results.append(
result
)
Adds the current parameter configuration to the overall results list.
After all 20 configurations:
xgb_tuning_results
↓
20 parameter-set results
28. Print progress
print(
f" Mean R²: {mean_r2:.4f} | "
f"Mean RMSE: {mean_rmse:.2f}"
)
Displays the average performance for the current configuration.
For example:
Mean R²: 0.7123 | Mean RMSE: 85000.25
This is just progress output.
29. Create final tuning DataFrame
xgb_tuning_results_df = pd.DataFrame(
xgb_tuning_results
)
Converts all 20 parameter-set results into a DataFrame.
30. Sort by Mean R²
xgb_tuning_results_df = (
xgb_tuning_results_df
.sort_values(
"Mean_R2",
ascending=False
)
.reset_index(drop=True)
)
The configurations are sorted from highest to lowest:
Mean R²
↓
highest first
Therefore, the first row represents the configuration with the highest mean R² among the 20 tested configurations.
reset_index(drop=True) simply gives the sorted DataFrame a clean index:
0
1
2
...
31. Display final results
display(
xgb_tuning_results_df
)
Displays the complete tuning table.
You can now compare all 20 configurations.
The complete workflow
The entire cell can be understood as:
729 possible combinations
↓
Randomly sample 20
↓
For each parameter set
↓
5-fold CV
↓
Build structured features
↓
OOF country encoding
↓
SBERT features
↓
PCA = 50
↓
Hybrid fusion
↓
Train XGBoost
↓
Predict validation fold
↓
Convert log prediction → raw PCF
↓
MAE / RMSE / R²
↓
Average 5 folds
↓
Repeat for all 20 sets
↓
Rank configurations
The most important viva answer
If your supervisor asks:
“What exactly did you do in this cell?”
Say:
“I performed randomized hyperparameter tuning for XGBoost. I defined a search space containing six hyperparameters, which gives 729 possible combinations, but instead of exhaustively testing all of them, I randomly sampled 20 configurations using
ParameterSampler. For each configuration, I used the same stratified 5-fold cross-validation and the same hybrid feature pipeline: OOF country target encoding, SBERT reduced to 50 PCA components, and structured features. Only the XGBoost hyperparameters changed. I then calculated MAE, RMSE, and R² on the original PCF scale and ranked the 20 configurations based on mean R².”
Challenging viva questions
“Why did you test only 20 combinations when there are 729?”
“Testing all 729 combinations with 5-fold CV would require 3,645 model fits. I therefore used randomized search to explore 20 combinations within the predefined search space while keeping the computational cost manageable. I acknowledge that this does not guarantee finding the global optimum.”
“Why is this called tuning?”
“Because I am selecting model hyperparameters based on cross-validation performance. The model's learned parameters are still estimated during training; these are settings controlling how that learning process behaves.”
“Did you use the test set during tuning?”
“No. The independent test set was kept separate. Hyperparameter selection was performed using cross-validation within the development data.”
This is one of the most important answers.
“Why did you keep PCA fixed at 50?”
“I previously compared several PCA dimensions using cross-validation. Among the tested values, 50 produced the best validation performance, so I fixed PCA at 50 before tuning XGBoost. This isolates the effect of XGBoost hyperparameters.”
“Are you guaranteed to find the best XGBoost parameters?”
“No. I only identify the best-performing configuration among the 20 sampled configurations. A larger or exhaustive search could potentially find a better configuration.”
“Why do you calculate the metrics after expm1()?”
“Because the model is trained on
log1p(PCF)to manage the skewed target distribution, but I want MAE, RMSE and R² to represent performance in the original PCF scale, which is more interpretable.”
“Why do you use OOF target encoding here?”
“Country target encoding uses the target variable, so directly calculating the country mean from the same observations could leak target information into the feature. OOF encoding ensures each training observation is encoded using information from other training folds rather than its own target.”
One important distinction to remember
There are three different things happening:
1. Hyperparameter search
20 XGBoost configurations
2. Cross-validation
5 folds per configuration
3. Independent test
Not touched during tuning
Therefore:
20 × 5 = 100 CV model fits, followed later by one final evaluation on the untouched test set.
That distinction is very likely to matter in your viva.
Top comments (0)