# ============================================================
# XGBOOST HYPERPARAMETER SEARCH SPACE
# ============================================================
xgb_param_grid = {
"n_estimators": [200, 300, 500],
"learning_rate": [0.03, 0.05, 0.08],
"max_depth": [3, 4, 6],
"min_child_weight": [1, 3, 5],
"subsample": [0.7, 0.8, 1.0],
"colsample_bytree": [0.7, 0.8, 1.0]
}
XGBoost Hyperparameter Search Space
This cell defines the hyperparameter search space for XGBoost.
The important distinction is:
```text id="xgbs1"
Model parameters
→ learned automatically during training
Hyperparameters
→ chosen before training
→ control how the model learns
Here, you are defining several possible values for each hyperparameter so that a later search procedure can determine which combination performs best.
### 1. Create the parameter grid
```python id="xgb01"
xgb_param_grid = {
This creates a Python dictionary called xgb_param_grid.
The dictionary has the structure:
```text id="xgb02"
Hyperparameter → possible values
For example:
```text id="xgb03"
n_estimators → [200, 300, 500]
A later search algorithm can try combinations from this space.
2. n_estimators
```python id="xgb04"
"n_estimators": [200, 300, 500],
Controls the **number of boosting trees/iterations**.
You are considering:
```text id="xgb05"
200
300
500
Conceptually:
```text id="xgb06"
More trees
↓
potentially more learning capacity
↓
but also more computation
Too many trees can contribute to overfitting depending on the other settings.
#### Viva answer
> “I search over 200, 300 and 500 estimators to investigate whether a smaller or larger boosting ensemble gives better validation performance.”
---
### 3. `learning_rate`
```python id="xgb07"
"learning_rate": [0.03, 0.05, 0.08],
Controls how much each new tree contributes to the overall prediction.
You test:
```text id="xgb08"
0.03
0.05
0.08
A smaller learning rate means the model learns more gradually.
There is generally a trade-off between:
```text id="xgb09"
lower learning rate
+
more trees
vs.
higher learning rate
+
fewer trees
That is why learning_rate should not be considered independently of n_estimators.
4. max_depth
```python id="xgb10"
"max_depth": [3, 4, 6],
Controls the maximum depth of each decision tree.
You test:
```text id="xgb11"
3
4
6
Smaller depth
Produces simpler trees and can reduce model complexity.
Larger depth
Allows the model to capture more complex interactions, but can increase overfitting risk.
Viva answer
“
max_depthcontrols tree complexity. I include shallow and deeper trees to evaluate the trade-off between capturing complex relationships and controlling overfitting.”
5. min_child_weight
```python id="xgb12"
"min_child_weight": [1, 3, 5],
Controls the minimum amount of weight required in a child node before XGBoost makes a further split.
You test:
```text id="xgb13"
1
3
5
Increasing this value generally makes the model more conservative about creating new branches.
Conceptually:
```text id="xgb14"
Lower min_child_weight
→ easier to split
→ more complex trees
Higher min_child_weight
→ harder to split
→ more conservative trees
This can therefore help control overfitting.
---
### 6. `subsample`
```python id="xgb15"
"subsample": [0.7, 0.8, 1.0],
Controls the proportion of training observations used when constructing each boosting tree.
You test:
```text id="xgb16"
0.7 → 70%
0.8 → 80%
1.0 → 100%
For example:
```text id="xgb17"
subsample = 0.8
means approximately 80% of the available training observations are sampled for a boosting iteration.
Using less than 1.0 introduces randomness and can help reduce overfitting.
7. colsample_bytree
```python id="xgb18"
"colsample_bytree": [0.7, 0.8, 1.0]
Controls the proportion of features considered for each tree.
You test:
```text id="xgb19"
0.7 → 70% of features
0.8 → 80% of features
1.0 → 100% of features
This is particularly relevant in your project because your hybrid representation can contain a relatively large number of features:
```text id="xgb20"
PCA-reduced SBERT
+
Structured features
Using feature subsampling can introduce additional randomness and potentially improve generalisation.
---
# How Large Is Your Search Space?
You have:
```text id="xgb21"
n_estimators → 3
learning_rate → 3
max_depth → 3
min_child_weight → 3
subsample → 3
colsample_bytree → 3
Therefore, a full Cartesian grid contains:
```text id="xgb22"
3 × 3 × 3 × 3 × 3 × 3
= 729
possible hyperparameter combinations.
That's an important number for your viva.
### Viva question: How many combinations are there?
> “There are 729 possible combinations because each of the six hyperparameters has three candidate values, giving \(3^6 = 729\) combinations.”
If you use ordinary `GridSearchCV` with 5-fold CV, that could mean:
```text id="xgb23"
729 combinations
× 5 folds
= 3,645 model fits
So this is a fairly large search computationally.
Why These Hyperparameters?
These parameters cover several important aspects of XGBoost behaviour:
| Hyperparameter | Controls |
|---|---|
n_estimators |
Number of boosting trees |
learning_rate |
Contribution of each tree |
max_depth |
Tree complexity |
min_child_weight |
Minimum requirement for further splitting |
subsample |
Fraction of observations used |
colsample_bytree |
Fraction of features used |
So you are not tuning only one aspect of the model.
You are exploring:
```text id="xgb24"
Learning speed
+
Model complexity
+
Number of trees
+
Row sampling
+
Feature sampling
---
# Very Important: This Cell Does NOT Perform the Search
This is a likely examiner trap.
This cell only defines:
```text id="xgb25"
SEARCH SPACE
It does not actually find the best parameters.
You still need a search method such as:
GridSearchCV(...)
or:
RandomizedSearchCV(...)
or another optimisation procedure.
So if the examiner asks:
“Have you performed hyperparameter tuning here?”
Say:
“This cell defines the candidate search space. The actual tuning is performed in the subsequent hyperparameter-search step.”
Do not say the parameters are already optimised just because you created the grid.
Grid Search vs Random Search
Because you have 729 combinations, an examiner may ask why you chose grid search or random search.
Grid Search
Tests every combination.
```text id="xgb26"
729 combinations
→ all are evaluated
Advantage:
> systematic and exhaustive within the specified grid.
Disadvantage:
> computationally expensive.
### Randomized Search
Samples a specified number of combinations.
For example:
```text id="xgb27"
729 possible combinations
↓
randomly sample 50
↓
evaluate only 50
Advantage:
much cheaper computationally.
Disadvantage:
does not evaluate every possible combination.
Strong viva answer
“The search space contains 729 combinations, so an exhaustive grid search can be computationally expensive, especially when combined with cross-validation. A randomized search can be more computationally efficient, while grid search is appropriate when exhaustive evaluation of the defined candidate values is required.”
Very Important for Your Project: Avoiding Leakage During Hyperparameter Search
Because your project uses:
- target encoding
- winsorization
- PCA
- scaling
- cross-validation
you must be careful about where each transformation is fitted.
The ideal principle is:
```text id="xgb28"
Training fold
↓
Fit preprocessing
↓
Fit XGBoost
↓
Validation fold
↓
Evaluate
Not:
```text id="xgb29"
Entire development dataset
↓
Fit preprocessing
↓
Cross-validation
The second approach can produce optimistic results if the preprocessing learns from validation observations.
Particularly important for your Country Target Encoding
Because country target encoding uses the PCF target, it has a direct leakage risk.
Your OOF approach is therefore especially important during CV.
Why Tune XGBoost After Selecting PCA = 50?
Your experimental sequence is becoming:
```text id="xgb30"
Step 1
Feature engineering
↓
SBERT + structured
Step 2
PCA selection
50 / 100 / 150 / 200
↓
Choose 50
Step 3
Model comparison
Linear / Ridge / SVR / RF / XGBoost / CatBoost
↓
Identify strong candidate
Step 4
XGBoost hyperparameter search
↓
Find better XGBoost configuration
This is a logical progression.
You first establish the feature representation, then compare algorithms, and then tune the selected candidate model.
---
# Presentation Wording
> “This cell defines the hyperparameter search space for XGBoost. I vary six parameters: the number of estimators, learning rate, maximum tree depth, minimum child weight, row subsampling, and feature subsampling. Each parameter has three candidate values, resulting in 729 possible combinations. These parameters control different aspects of model complexity, learning rate, and regularisation. This cell only defines the search space; the actual hyperparameter optimisation is performed in the subsequent search procedure using cross-validation.”
### The 5 viva points to memorise
1. **This is a search space, not the search itself.**
2. **There are 729 combinations.**
3. `n_estimators` + `learning_rate` control boosting capacity and learning speed.
4. `max_depth` + `min_child_weight` control tree complexity.
5. `subsample` + `colsample_bytree` introduce row/feature sampling and can help generalisation.
For your XGBoost tuning
* Explain why 729 trials may be inefficient
* Show leakage-safe tuning structure
Top comments (0)