DEV Community: TS

Cell-15-FINAL-PREDICTION-ANALYSIS

TS — Mon, 14 Sep 2026 11:37:25 +0000

# ============================================================
# FINAL PREDICTION ANALYSIS
# ============================================================

final_predictions_df = pd.DataFrame({
    "Actual_PCF": y_test_final_raw,
    "Predicted_PCF": y_test_pred_raw
})

# Residual = Actual - Predicted
final_predictions_df["Residual"] = (
    final_predictions_df["Actual_PCF"]
    - final_predictions_df["Predicted_PCF"]
)

# Absolute error
final_predictions_df["Absolute_Error"] = (
    final_predictions_df["Residual"].abs()
)

# Absolute percentage error
final_predictions_df["Absolute_Percentage_Error"] = (
    final_predictions_df["Absolute_Error"]
    / np.maximum(
        final_predictions_df["Actual_PCF"],
        1e-8
    )
) * 100

display(
    final_predictions_df.head(10)
)

Final Prediction Analysis — Purpose

This cell creates a row-by-row comparison between the actual PCF and the final model's predicted PCF on the independent test set.

This is different from the previous evaluation cell where you calculated overall:

MAE
RMSE
R²

Here, you are going deeper and asking:

“For each individual test observation, how close was the prediction to the actual PCF?”

This is useful for error analysis and interpretation.

1. Create the prediction DataFrame

```python id="v6p2k1"
final_predictions_df = pd.DataFrame({
"Actual_PCF": y_test_final_raw,
"Predicted_PCF": y_test_pred_raw
})




This creates a DataFrame with two columns:



```text id="9n2h7x"
Actual_PCF
Predicted_PCF

`Actual_PCF`

Contains the real PCF values from the independent test set.

`Predicted_PCF`

Contains the PCF values predicted by your final tuned XGBoost model.

So each row represents:

```text id="j6s8q3"
One test observation
↓
Actual PCF vs Predicted PCF




For example, your first row is:



```text id="8c4v1z"
Actual     = 2300.00
Predicted  = 1740.69

2. Calculate residual

```python id="m3r7k9"
final_predictions_df["Residual"] = (
final_predictions_df["Actual_PCF"]
- final_predictions_df["Predicted_PCF"]
)




The residual is:

**Residual = Actual − Predicted**

This tells you the **direction of the prediction error**.

### If residual is positive



```text id="8w1h5d"
Actual > Predicted

The model underestimated the PCF.

If residual is negative

```text id="2p7k4m"
Actual < Predicted




The model **overestimated** the PCF.

---

### Example: Row 0



```text id="y7k2q1"
Actual     = 2300.00
Predicted  = 1740.69

Residual = 2300 - 1740.69
         = +559.31

Positive residual → the model underpredicted by approximately 559 kg CO₂e.

Example: Row 1

```text id="q8d3m6"
Actual = 580.00
Predicted = 688.73

Residual = 580 - 688.73
= -108.73




Negative residual → the model **overpredicted** by approximately 109 kg CO₂e.

---

# 3. Calculate absolute error



```python id="r5n2v8"
final_predictions_df["Absolute_Error"] = (
    final_predictions_df["Residual"].abs()
)

.abs() takes the absolute value.

So:

```text id="f1s9k4"
Residual = +559 → Absolute Error = 559
Residual = -109 → Absolute Error = 109




The sign is removed.

Why?

Because absolute error is concerned only with:

> **How large was the error?**

not whether the model over- or underpredicted.

This is closely related to MAE.

---

# 4. Calculate Absolute Percentage Error



```python id="c7m4x1"
final_predictions_df["Absolute_Percentage_Error"] = (
    final_predictions_df["Absolute_Error"]
    / np.maximum(
        final_predictions_df["Actual_PCF"],
        1e-8
    )
) * 100

This calculates the Absolute Percentage Error (APE) for each observation.

The formula is:

$$
APE =
\frac{|Actual-Predicted|}
{Actual}
\times100
$$

So it tells you:

“The prediction error represents what percentage of the actual PCF?”

Example: Row 0

Actual:

```text id="x2q7m4"
2300




Absolute error:



```text id="r3k8p1"
559.31

Therefore:

$$
\frac{559.31}{2300}\times100
\approx24.32\%
$$

Your table shows:

```text id="s5j1q8"
24.317823%




So the prediction is approximately **24.3% away from the actual value**.

---

# 5. Why `np.maximum(..., 1e-8)`?

This part:



```python id="h2v6c9"
np.maximum(
    final_predictions_df["Actual_PCF"],
    1e-8
)

protects against division by zero.

Suppose:

```text id="p4n7x2"
Actual PCF = 0




Then:



```text id="k9m3s6"
Absolute error / 0

would create a division-by-zero problem.

So the denominator is forced to be at least:

```text id="e1r8w5"
0.00000001




### Important viva point

This does **not** make percentage error well-defined for zero actual values.

It only prevents a numerical division-by-zero error.

If the dataset contains actual PCF values of zero, percentage-based metrics should be interpreted carefully.

---

# 6. Display first 10 observations



```python id="u4q8z2"
display(
    final_predictions_df.head(10)
)

.head(10) displays the first 10 rows.

It lets you inspect individual predictions without printing the entire test dataset.

Understanding your actual results

Your first 10 observations show:

Row	Actual	Predicted	Residual	Absolute Error	APE
0	2300.00	1740.69	+559.31	559.31	24.32%
1	580.00	688.73	−108.73	108.73	18.75%
2	58.00	144.31	−86.31	86.31	148.81%
3	2219.00	1447.95	+771.05	771.05	34.75%
4	15.00	16.25	−1.25	1.25	8.33%
5	66.16	96.84	−30.68	30.68	46.37%
6	17.10	18.84	−1.74	1.74	10.15%
7	584.00	655.04	−71.04	71.04	12.16%
8	6240.00	3961.29	+2278.71	2278.71	36.52%
9	2000.00	1740.86	+259.14	259.14	12.96%

The most interesting example: Row 2

Look at:

```text id="k3m7v2"
Actual = 58
Predicted = 144.31




The absolute error is only:



```text id="n6x1q8"
86.31 kg CO₂e

But the percentage error is:

```text id="w9c4s7"
148.81%




Why?

Because the actual value is very small.

$$
\frac{86.31}{58}\times100
\approx148.8\%
$$

This demonstrates an important limitation of percentage-based errors:

> **APE can become very large when the actual target is small, even when the absolute error is relatively modest.**

### Viva question

**“Why is the percentage error 148% when the absolute error is only 86?”**

Strong answer:

> “Because percentage error is relative to the actual value. The actual PCF is only 58 kg CO₂e, so an 86 kg CO₂e error is larger than the actual value itself. This is why I would not use APE alone to judge the model, particularly for small PCF values.”

Excellent answer.

---

# Row 8 — Large absolute error

Row 8:



```text id="3y7p2n"
Actual    = 6240
Predicted = 3961.29

Absolute error:

```text id="7c4m9x"
2278.71 kg CO₂e




APE:



```text id="q5n8r1"
36.52%

This shows the opposite situation.

The percentage error is not as extreme as row 2, but the absolute error is very large.

This is why your project reports multiple metrics rather than relying on one metric.

Why do you need both residual and absolute error?

Because they answer different questions.

Residual

Shows direction:

```text id="r8x2k6"
Positive → underprediction
Negative → overprediction




### Absolute Error

Shows **magnitude**:



```text id="p4m7q1"
How far away was the prediction?

For example:

```text id="c9v3z5"
Residual = -108.73




means the model overpredicted.

But:



```text id="a6k1w8"
Absolute Error = 108.73

tells us the size of that error without caring about direction.

How this connects to MAE

Your:

```text id="x7n2m5"
Absolute_Error




column contains the individual absolute errors.

MAE is essentially:

$$
MAE = mean(|Actual-Predicted|)
$$

So:



```text id="h4q9s2"
Absolute_Error
      ↓
average all rows
      ↓
MAE

Therefore, this table lets you understand what is behind the overall MAE.

How this connects to RMSE

RMSE is particularly affected by large residuals.

For example, row 8 has an error of approximately:

```text id="z2k6v9"
2278.71




Large errors like this receive much more influence under RMSE because the residual is squared.

That's why your earlier:



```text id="v8q3m1"
Mean MAE  ≈ 10,398
Mean RMSE ≈ 86,982

can differ dramatically.

Important limitation of this analysis

You are displaying only:

```python id="s4m8q2"
.head(10)




So this cell **does not yet analyse the entire test set**.

It only creates the full table and displays the first 10 observations.

If your supervisor asks:

**“Can you identify which observations have the largest errors?”**

This cell has the information available, but `head(10)` does not show the largest errors.

You would need to sort the DataFrame by `Absolute_Error`, for example:



```python
final_predictions_df.sort_values(
    "Absolute_Error",
    ascending=False
).head(10)

That would show the 10 largest absolute errors.

Similarly, you could sort by:

"Absolute_Percentage_Error"

to investigate the observations with the largest relative errors.

Very important viva distinction

If asked:

“Is this calculating your final model performance?”

Answer:

“Not directly. The overall final model performance is calculated using aggregate metrics such as MAE, RMSE and R². This cell performs row-level error analysis by comparing actual and predicted PCF values and calculating residuals, absolute errors and absolute percentage errors.”

That's the correct distinction.

30-second viva answer

“This cell performs row-level error analysis on the independent test predictions. I create a DataFrame containing the actual and predicted PCF values, then calculate residuals as actual minus predicted to identify under- and overprediction. I calculate absolute error to measure the magnitude of each error and absolute percentage error to express the error relative to the actual PCF. I use a small epsilon in the denominator to avoid division by zero. The analysis helps explain individual prediction behaviour behind the aggregate MAE, RMSE and R² results. However, percentage error can become very large for small actual PCF values, so I would not use it as the sole evaluation metric.”

Examiner trap

If they point to:

58 actual → 144 predicted → 148.8% error

and ask “Is your model bad?”

Don't immediately say yes.

Say:

“This individual prediction has a large relative error, but its absolute error is about 86 kg CO₂e. Because the actual value is small, the percentage error is amplified. I would assess it together with MAE, RMSE, R² and the distribution of errors across the complete test set.”

That answer shows you understand both the model and the limitations of the metric.

For your final error analysis

Check whether errors vary by PCF size
Inspect bias across the test set

Cell-14-FINAL STRUCTURED FEATURE BUILDER

TS — Mon, 14 Sep 2026 11:33:41 +0000

# ============================================================
# FINAL STRUCTURED FEATURE BUILDER
#
# Training:
#   Country mapping fitted on the complete development set
#
# Test:
#   Same mapping applied to untouched test data
#
# Test target is NEVER used.
# ============================================================

def build_final_structured_features(
    X_train,
    y_train,
    X_test
):

    train_part = X_train.copy()
    test_part = X_test.copy()


    # --------------------------------------------------------
    # Product Weight → Winsorization
    # --------------------------------------------------------

    lower_limit, upper_limit = fit_winsorization(
        train_part,
        WEIGHT_COL
    )

    train_part = apply_winsorization(
        train_part,
        WEIGHT_COL,
        lower_limit,
        upper_limit
    )

    test_part = apply_winsorization(
        test_part,
        WEIGHT_COL,
        lower_limit,
        upper_limit
    )


    # --------------------------------------------------------
    # Numeric features
    # --------------------------------------------------------

    numeric_cols = [
        YEAR_COL,
        WEIGHT_COL
    ]

    train_numeric = train_part[numeric_cols].apply(
        pd.to_numeric,
        errors="coerce"
    )

    test_numeric = test_part[numeric_cols].apply(
        pd.to_numeric,
        errors="coerce"
    )


    # --------------------------------------------------------
    # Categorical features → One-Hot Encoding
    # --------------------------------------------------------

    other_ohe = OneHotEncoder(
        handle_unknown="ignore",
        sparse_output=False
    )

    train_other_ohe = other_ohe.fit_transform(
        train_part[OTHER_CATEGORICAL_FEATURES]
    )

    test_other_ohe = other_ohe.transform(
        test_part[OTHER_CATEGORICAL_FEATURES]
    )


    # --------------------------------------------------------
    # Country → Target Encoding
    #
    # Mapping is learned from complete development data only.
    # --------------------------------------------------------

    country_means = (
        pd.DataFrame({
            "Country": train_part[COUNTRY_COL],
            "Target": y_train
        })
        .groupby("Country")["Target"]
        .mean()
    )

    global_mean = np.mean(y_train)


    train_country = (
        train_part[COUNTRY_COL]
        .map(country_means)
        .fillna(global_mean)
        .to_numpy()
        .reshape(-1, 1)
    )

    test_country = (
        test_part[COUNTRY_COL]
        .map(country_means)
        .fillna(global_mean)
        .to_numpy()
        .reshape(-1, 1)
    )


    # --------------------------------------------------------
    # Combine structured features
    # --------------------------------------------------------

    X_train_structured = np.hstack([
        train_numeric.to_numpy(),
        train_other_ohe,
        train_country
    ])

    X_test_structured = np.hstack([
        test_numeric.to_numpy(),
        test_other_ohe,
        test_country
    ])


    return (
        X_train_structured,
        X_test_structured
    )

Final Structured Feature Builder — Purpose

This function prepares the final structured features for the last stage of the project, where you train the final model on the complete development set and then evaluate it on the untouched independent test set.

The key difference from the previous build_structured_features() function is:

During CV, preprocessing is fitted separately inside each fold. Here, after model/feature selection is complete, the preprocessing is fitted on the complete development set and then applied to the test set.

Most importantly:

The test target is never used.

1. Define the function

```python id="c2r7xq"
def build_final_structured_features(
X_train,
y_train,
X_test
):




The function receives three inputs:

* `X_train` → complete development features
* `y_train` → development-set target
* `X_test` → untouched test features

Here, `X_train` is actually your **full development set**, not just one CV fold.

The target `y_train` is required because Country uses **target encoding**.

`X_test` contains the test features but its target is deliberately not passed into the function.

### Viva question

**Why does the function need `y_train` but not `y_test`?**

Answer:

> “Because country target encoding requires the training target to calculate country-level target means. I do not need and deliberately do not use `y_test`, because the independent test target must remain unseen until final evaluation.”

---

# 2. Copy the data



```python id="w7w5kq"
train_part = X_train.copy()
test_part = X_test.copy()

Creates copies of both datasets.

This means the original DataFrames are not modified directly.

Product Weight → Winsorization

3. Learn winsorization limits from development data

```python id="j3f5r9"
lower_limit, upper_limit = fit_winsorization(
train_part,
WEIGHT_COL
)




This calculates the lower and upper limits for Product Weight using **only the development data**.

Winsorization limits extreme values by capping observations beyond the selected boundaries.

Conceptually:



```text
Very small weight → lower limit
Normal weight     → unchanged
Very large weight → upper limit

Important leakage point

The limits are learned from training/development data only.

You do not calculate the limits using the test data.

4. Apply limits to development data

```python id="6t7y1e"
train_part = apply_winsorization(
train_part,
WEIGHT_COL,
lower_limit,
upper_limit
)




The previously calculated limits are applied to the development data.

---

### 5. Apply the SAME limits to test data



```python id="4d1q3r"
test_part = apply_winsorization(
    test_part,
    WEIGHT_COL,
    lower_limit,
    upper_limit
)

This is a very important line.

You do not fit new limits on the test set.

Instead:

Development data
       ↓
learn limits
       ↓
┌───────────────┐
│ same limits   │
│               │
├───────────────┤
│ Development   │
│ Test          │
└───────────────┘

This keeps the transformation consistent and prevents test-data information from influencing preprocessing.

Viva answer

“I fit the winsorization thresholds on the complete development set and then applied those same thresholds to the test set. I did not estimate any preprocessing parameter from the test distribution.”

Numeric Features

6. Define numeric columns

```python id="8h9f3s"
numeric_cols = [
YEAR_COL,
WEIGHT_COL
]




The numeric features are:

* Year of reporting
* Product weight

---

### 7. Convert development numeric columns



```python id="5m2d8a"
train_numeric = train_part[numeric_cols].apply(
    pd.to_numeric,
    errors="coerce"
)

Attempts to convert the numeric columns into numeric values.

If something cannot be converted, it becomes NaN.

For example:

"2024" → 2024
"10.5" → 10.5
"unknown" → NaN

8. Convert test numeric columns

```python id="1j8v6c"
test_numeric = test_part[numeric_cols].apply(
pd.to_numeric,
errors="coerce"
)




The exact same type of conversion is applied to test data.

No test target is involved.

---

# Categorical Features → One-Hot Encoding

### 9. Create OneHotEncoder



```python id="q5z8n2"
other_ohe = OneHotEncoder(
    handle_unknown="ignore",
    sparse_output=False
)

This encoder converts categorical variables into numerical binary columns.

Your categorical variables include:

Company's GICS Industry
Protocol used for PCF
Stage-level CO₂e available

For example:

Industry:
Food
Technology
Automotive

could become columns such as:

Industry_Food
Industry_Technology
Industry_Automotive

`handle_unknown="ignore"`

This is particularly important for the test set.

Suppose the development data contains:

Food
Technology
Automotive

but the test set contains a previously unseen:

Mining

The encoder does not crash.

It ignores the unseen category.

Viva answer

“I used handle_unknown='ignore' because the test set may contain categorical values that were not present in the development set. This allows the transformation to proceed without creating an error or fitting the encoder using test information.”

10. Fit encoder on development data

```python id="q0e7v3"
train_other_ohe = other_ohe.fit_transform(
train_part[OTHER_CATEGORICAL_FEATURES]
)




This is where the encoder **learns the category structure**.

For example, it learns which industry/protocol/stage categories exist in the development data.

---

### 11. Transform test data



```python id="m8v4q6"
test_other_ohe = other_ohe.transform(
    test_part[OTHER_CATEGORICAL_FEATURES]
)

Notice:

transform()

not:

fit_transform()

This is crucial.

The encoder has already been fitted using development data.

The test data is only transformed using the existing mapping.

Viva question

Why not use fit_transform() on the test set?

Answer:

“Because fitting on the test set would allow information from the test distribution to influence the preprocessing. I fit the encoder only on the development data and then use transform() on the test data.”

Country → Target Encoding

This is probably the most important section of this function.

12. Calculate country means

```python id="v4k8s2"
country_means = (
pd.DataFrame({
"Country": train_part[COUNTRY_COL],
"Target": y_train
})
.groupby("Country")["Target"]
.mean()
)




This creates a mapping:



```text
Country → Mean target

For example, conceptually:

Germany → 25.4
France  → 18.7
Japan   → 31.2

The actual values depend on your dataset.

Critical point

These means are calculated using:

Complete development data

and:

y_train

only.

There is no y_test here.

13. Calculate global mean

```python id="1k7d3x"
global_mean = np.mean(y_train)




This calculates the overall mean target of the development set.

It is used as a fallback.

---

### 14. Encode development countries



```python id="f0j5c7"
train_country = (
    train_part[COUNTRY_COL]
    .map(country_means)
    .fillna(global_mean)
    .to_numpy()
    .reshape(-1, 1)
)

Let's break this into functions.

`.map(country_means)`

Maps each country to its calculated mean target.

Example:

Germany → 25.4
France  → 18.7
Japan   → 31.2

`.fillna(global_mean)`

If a country doesn't have a mapping, use the overall development-set mean instead.

`.to_numpy()`

Converts the pandas Series into a NumPy array.

`.reshape(-1, 1)`

Converts it into a two-dimensional column:

(n,)

becomes:

(n, 1)

This makes it suitable for np.hstack() later.

15. Encode test countries

```python id="n9w3p5"
test_country = (
test_part[COUNTRY_COL]
.map(country_means)
.fillna(global_mean)
.to_numpy()
.reshape(-1, 1)
)




This uses the **same `country_means` learned from the development data**.

This is extremely important.

The test target is not used.

If the test contains a country that wasn't present in development:



```text
Unknown country
       ↓
global development mean

Viva answer

“For the final test transformation, country target encoding is fitted using the complete development set. Test countries are mapped using that fixed mapping. If a test country was unseen during development, I use the development global mean as a fallback. Therefore, the test target is never involved.”

16. Combine structured features

Development features

```python id="g3w8r1"
X_train_structured = np.hstack([
train_numeric.to_numpy(),
train_other_ohe,
train_country
])




`np.hstack()` means **horizontal stacking**.

It combines:



```text
Numeric features
      +
One-hot categorical features
      +
Country target encoding
      ↓
Final structured feature matrix

Test features

```python id="k2m6v8"
X_test_structured = np.hstack([
test_numeric.to_numpy(),
test_other_ohe,
test_country
])




The same feature structure is created for the test data.

Therefore, the training and test matrices have compatible columns.

---

# 17. Return the final feature matrices



```python id="z4r1t6"
return (
    X_train_structured,
    X_test_structured
)

The function returns:

Final development structured features
+
Final test structured features

These can then be combined with:

SBERT → PCA 50

to create the final hybrid features.

Why this function is different from the previous one

This is very important for your viva.

During CV

You used:

build_structured_features()

because each fold needs its own preprocessing.

For example:

Fold 1:
80% training → fit preprocessing
20% validation → transform

Fold 2:
different 80% → fit preprocessing
different 20% → transform

Final model

Now model selection is finished.

So you use:

build_final_structured_features()

with:

Complete development set
        ↓
fit preprocessing
        ↓
Independent test set
        ↓
transform only

The workflow is:

                    DEVELOPMENT DATA
                           │
                           ▼
                  Fit preprocessing
                           │
          ┌────────────────┴───────────────┐
          ▼                                ▼
 Complete development                 Test data
      transform                       transform
          │                                │
          ▼                                ▼
   Train final model              Final evaluation

Most important leakage explanation

Your examiner may ask:

“How did you prevent test-set leakage?”

Strong answer:

“I kept the test set completely untouched during model selection. In the final structured feature builder, winsorization thresholds, one-hot encoding categories, and country target-encoding mappings are all learned from the complete development set. The test set is only transformed using those learned parameters. The test target is never used during preprocessing or training.”

One subtle point you should know

There is an important distinction between test leakage and training self-encoding.

Here:

country_means

is calculated using the complete development set, so each development observation contributes to its own country's mean.

That is acceptable for fitting the final model after all CV/model-selection decisions are complete, because you are now using all available development information to train the final model.

It would not be appropriate for generating unbiased CV predictions.

That's exactly why your earlier CV function used OOF country target encoding, while this final function uses the complete development mapping.

Viva question

“Why did you use OOF encoding during CV but normal target encoding here?”

Answer:

“During cross-validation, OOF encoding was necessary to prevent an observation's own target from contributing to its encoded training feature. After model selection, I retrain the final model using the complete development data, so I use the full development target information to construct the final training representation. The independent test set is still encoded only from that development-derived mapping.”

Full final pipeline

This function fits into the final stage like this:

Original dataset
       ↓
80% Development / 20% Test
       ↓
CV + PCA selection + Model comparison + XGB tuning
       ↓
All modelling decisions fixed
       ↓
Complete Development Set
       │
       ├── Winsorization limits
       ├── OHE mapping
       └── Country target mapping
       ↓
Final structured features
       │
       └──────────────┐
                      ▼
Test data ──transform using same mappings
                      │
                      ▼
             Structured test features

Then you combine these with the final:

SBERT
  ↓
PCA = 50
  ↓
Hybrid features
  ↓
Final tuned XGBoost
  ↓
Independent test prediction
  ↓
MAE / RMSE / R²

Best 30-second viva answer

“This function prepares the structured features for the final independent test evaluation. Unlike the CV preprocessing, I now fit the preprocessing on the complete development set because model selection has already finished. Product weight winsorization thresholds, one-hot encoding categories, and country target-encoding mappings are learned only from the development data. The same learned transformations are then applied to the untouched test data. In particular, the test target is never used. This gives me a leakage-controlled final feature representation for training the final model and evaluating it on genuinely unseen data.”

For your final evaluation

Check target-encoding bias in final training
Verify feature-column consistency

Cell-13-TUNED XGBOOST — OOF DEVELOPMENT EVALUATION

TS — Mon, 14 Sep 2026 11:29:22 +0000

# ============================================================
# TUNED XGBOOST — OOF DEVELOPMENT EVALUATION
#
# Target   = log1p(PCF)
# Country  = OOF Target Encoding
# SBERT    = PCA 50
# CV       = Stratified 5-Fold
# ============================================================

BEST_XGB_PARAMS = {
    "n_estimators": 200,
    "learning_rate": 0.08,
    "max_depth": 6,
    "min_child_weight": 3,
    "subsample": 0.7,
    "colsample_bytree": 0.8
}


xgb_oof_results = []


for fold, (train_idx, valid_idx) in enumerate(
    skf.split(train_df, target_bins),
    start=1
):

    print(f"Running fold {fold}/5...")


    # --------------------------------------------------------
    # 1. Fold data
    # --------------------------------------------------------

    X_fold_train = (
        train_df.iloc[train_idx].copy()
    )

    X_fold_valid = (
        train_df.iloc[valid_idx].copy()
    )


    # --------------------------------------------------------
    # 2. Log target
    # --------------------------------------------------------

    y_fold_train_log = (
        y_log.iloc[train_idx].to_numpy()
    )

    y_fold_valid_raw = (
        train_df.iloc[valid_idx][TARGET].to_numpy()
    )


    # --------------------------------------------------------
    # 3. Structured features
    #    Country = OOF Target Encoding
    # --------------------------------------------------------

    X_struct_train, X_struct_valid = (
        build_structured_features(
            X_fold_train,
            y_fold_train_log,
            X_fold_valid
        )
    )


    # --------------------------------------------------------
    # 4. SBERT features
    # --------------------------------------------------------

    X_sbert_train = (
        X_train_sbert[train_idx]
    )

    X_sbert_valid = (
        X_train_sbert[valid_idx]
    )


    # --------------------------------------------------------
    # 5. PCA = 50
    #    Fit only on outer training fold
    # --------------------------------------------------------

    (
        X_sbert_train_pca,
        X_sbert_valid_pca,
        _
    ) = apply_pca_to_sbert(
        X_sbert_train,
        X_sbert_valid,
        n_components=50
    )


    # --------------------------------------------------------
    # 6. Hybrid features
    # --------------------------------------------------------

    X_hybrid_train = fuse_features(
        X_sbert_train_pca,
        X_struct_train
    )

    X_hybrid_valid = fuse_features(
        X_sbert_valid_pca,
        X_struct_valid
    )


    # --------------------------------------------------------
    # 7. Tuned XGBoost
    # --------------------------------------------------------

    model = XGBRegressor(
        objective="reg:squarederror",
        random_state=42,
        n_jobs=-1,
        **BEST_XGB_PARAMS
    )


    # --------------------------------------------------------
    # 8. Train
    # --------------------------------------------------------

    model.fit(
        X_hybrid_train,
        y_fold_train_log
    )


    # --------------------------------------------------------
    # 9. Predict
    # --------------------------------------------------------

    y_pred_log = model.predict(
        X_hybrid_valid
    )


    # Convert back to original PCF scale
    y_pred_raw = np.expm1(
        y_pred_log
    )

    y_pred_raw = np.maximum(
        y_pred_raw,
        0
    )


    # --------------------------------------------------------
    # 10. Metrics
    # --------------------------------------------------------

    mae = mean_absolute_error(
        y_fold_valid_raw,
        y_pred_raw
    )

    rmse = np.sqrt(
        mean_squared_error(
            y_fold_valid_raw,
            y_pred_raw
        )
    )

    r2 = r2_score(
        y_fold_valid_raw,
        y_pred_raw
    )


    xgb_oof_results.append({
        "Fold": fold,
        "MAE": mae,
        "RMSE": rmse,
        "R2": r2
    })


    print(
        f"  MAE={mae:.4f}, "
        f"RMSE={rmse:.4f}, "
        f"R²={r2:.4f}"
    )


# ============================================================
# OOF SUMMARY
# ============================================================

xgb_oof_results_df = pd.DataFrame(
    xgb_oof_results
)


xgb_oof_summary = pd.DataFrame({
    "Mean_MAE": [
        xgb_oof_results_df["MAE"].mean()
    ],
    "Mean_RMSE": [
        xgb_oof_results_df["RMSE"].mean()
    ],
    "Mean_R2": [
        xgb_oof_results_df["R2"].mean()
    ],
    "Std_R2": [
        xgb_oof_results_df["R2"].std()
    ]
})


display(xgb_oof_results_df)
display(xgb_oof_summary)

Tuned XGBoost — OOF Development Evaluation

This cell is the next step after hyperparameter tuning.

The previous cell asked:

“Among the 20 randomly sampled XGBoost configurations, which configuration performs best?”

This cell takes the selected configuration:

BEST_XGB_PARAMS = {
    "n_estimators": 200,
    "learning_rate": 0.08,
    "max_depth": 6,
    "min_child_weight": 3,
    "subsample": 0.7,
    "colsample_bytree": 0.8
}

and evaluates it again using 5-fold cross-validation on the development dataset.

The important point is:

This cell is not searching anymore. It is evaluating the selected/tuned XGBoost configuration across all five development folds.

1. Selected XGBoost parameters

```python id="n8n1qd"
BEST_XGB_PARAMS = {
"n_estimators": 200,
"learning_rate": 0.08,
"max_depth": 6,
"min_child_weight": 3,
"subsample": 0.7,
"colsample_bytree": 0.8
}




These are the hyperparameters selected from the previous tuning experiment.

The values mean:

| Parameter          | Value | Meaning                                  |
| ------------------ | ----: | ---------------------------------------- |
| `n_estimators`     |   200 | Number of boosting trees                 |
| `learning_rate`    |  0.08 | Contribution/shrinkage of each tree      |
| `max_depth`        |     6 | Maximum depth of each tree               |
| `min_child_weight` |     3 | Makes splitting more conservative        |
| `subsample`        |   0.7 | Uses 70% of rows for each boosting stage |
| `colsample_bytree` |   0.8 | Uses 80% of features for each tree       |

### Viva point

Do **not** say:

> “These are the globally optimal parameters.”

Say:

> **“These were the best-performing parameters among the 20 randomly sampled configurations evaluated during tuning.”**

That is scientifically safer.

---

### 2. Create result storage



```python id="psjq8j"
xgb_oof_results = []

Creates an empty list to store the performance of the tuned XGBoost model for each fold.

After the loop, there will be five results:

```text id="d5cy1c"
Fold 1
Fold 2
Fold 3
Fold 4
Fold 5




---

### 3. Start 5-fold evaluation



```python id="bqj4kq"
for fold, (train_idx, valid_idx) in enumerate(
    skf.split(train_df, target_bins),
    start=1
):

This uses the same Stratified 5-Fold CV strategy defined earlier.

For each fold:

```text id="qlz07h"
Development data
↓
┌───────────────┐
│ Training fold │
└───────────────┘
+
┌────────────────┐
│ Validation fold│
└────────────────┘




This happens five times.

### Why `target_bins`?

Because the original PCF target is continuous.

You created approximate quantile bins earlier so that each fold has a reasonably similar distribution of the target.

---

### 4. Print fold progress



```python id="4sfy9f"
print(f"Running fold {fold}/5...")

Just shows which fold is currently being processed.

Fold-level preprocessing

5. Separate training and validation data

```python id="f3k5wo"
X_fold_train = (
train_df.iloc[train_idx].copy()
)

X_fold_valid = (
train_df.iloc[valid_idx].copy()
)




The current fold is divided into:

* training portion
* validation portion

The validation portion is not used to fit the model.

This is important for estimating generalisation performance.

---

### 6. Training target



```python id="s0u2tx"
y_fold_train_log = (
    y_log.iloc[train_idx].to_numpy()
)

The model is trained using:

```text id="q1x0kd"
log1p(PCF)




rather than raw PCF.

---

### 7. Validation target



```python id="s2r8c5"
y_fold_valid_raw = (
    train_df.iloc[valid_idx][TARGET].to_numpy()
)

The validation target remains in the original PCF scale.

So the process is:

```text id="u8p5q6"
Training:
log1p(PCF)

Validation:
original PCF




This is intentional.

---

# Structured features

### 8. Build structured features



```python id="n7n8gq"
X_struct_train, X_struct_valid = (
    build_structured_features(
        X_fold_train,
        y_fold_train_log,
        X_fold_valid
    )
)

This generates the structured feature representation.

It includes:

Year
Product weight
Industry
Protocol
Stage-level CO₂e availability
Country target encoding

Most importantly, the country encoding is OOF target encoded for the training portion.

This is important because country encoding uses the target variable.

SBERT representation

9. Select SBERT features

```python id="e9o6r2"
X_sbert_train = (
X_train_sbert[train_idx]
)

X_sbert_valid = (
X_train_sbert[valid_idx]
)




The SBERT embeddings corresponding to the current training and validation observations are selected.

SBERT itself was already generated earlier.

You are not retraining SBERT here.

---

# PCA

### 10. Reduce SBERT to 50 dimensions



```python id="6q6u9m"
(
    X_sbert_train_pca,
    X_sbert_valid_pca,
    _
) = apply_pca_to_sbert(
    X_sbert_train,
    X_sbert_valid,
    n_components=50
)

This applies the selected PCA configuration.

```text id="l0xk8b"
SBERT embedding
↓
PCA
↓
50 dimensions




### Very important line



```python id="q8h5tq"
X_sbert_train,
X_sbert_valid

are passed separately.

Inside apply_pca_to_sbert():

```python id="w7w3xj"
pca.fit_transform(X_train_sbert)
pca.transform(X_valid_sbert)




Therefore:

> **PCA is fitted only on the training fold and then applied to the validation fold.**

This prevents validation information from influencing the PCA transformation.

---

# Hybrid features

### 11. Combine SBERT and structured features



```python id="a3x0j4"
X_hybrid_train = fuse_features(
    X_sbert_train_pca,
    X_struct_train
)

and:

```python id="6e4jpv"
X_hybrid_valid = fuse_features(
X_sbert_valid_pca,
X_struct_valid
)




The model gets both representations:



```text id="qk2h9w"
Semantic text information
        +
Structured information
        ↓
Hybrid feature representation

This is the core idea of the hybrid model.

Tuned XGBoost

12. Create the model

```python id="y8zq2a"
model = XGBRegressor(
objective="reg:squarederror",
random_state=42,
n_jobs=-1,
**BEST_XGB_PARAMS
)




Creates an XGBoost regression model using the selected parameters.

The:



```python id="t7jv1z"
**BEST_XGB_PARAMS

inserts all six tuned hyperparameters.

So the model is effectively configured with:

```text id="x3v0ms"
200 trees
learning rate = 0.08
depth = 6
min child weight = 3
row sampling = 70%
feature sampling = 80%




---

### 13. Train



```python id="c5f4vq"
model.fit(
    X_hybrid_train,
    y_fold_train_log
)

XGBoost learns:

```text id="z6o9wa"
Hybrid features
↓
log1p(PCF)




using only the training fold.

---

# Prediction

### 14. Predict validation fold



```python id="w0p3z8"
y_pred_log = model.predict(
    X_hybrid_valid
)

The predictions are initially on the log-transformed scale.

15. Convert predictions back

```python id="b6v3h9"
y_pred_raw = np.expm1(
y_pred_log
)




Because training used:



```python
np.log1p(PCF)

the inverse transformation is:

np.expm1()

Therefore:

```text id="q9q4t1"
log1p(PCF)
↓
model
↓
predicted log1p(PCF)
↓
expm1
↓
predicted PCF




---

### 16. Enforce non-negative predictions



```python id="f7q2k3"
y_pred_raw = np.maximum(
    y_pred_raw,
    0
)

Any negative prediction is replaced by zero.

This makes the prediction consistent with the physical interpretation of PCF.

Metrics

17. MAE

```python id="x0k8s2"
mae = mean_absolute_error(
y_fold_valid_raw,
y_pred_raw
)




Measures average absolute prediction error.

**Lower is better.**

---

### 18. RMSE



```python id="j3m6v9"
rmse = np.sqrt(
    mean_squared_error(
        y_fold_valid_raw,
        y_pred_raw
    )
)

RMSE penalises large errors more heavily than MAE.

Lower is better.

19. R²

```python id="p4r7d1"
r2 = r2_score(
y_fold_valid_raw,
y_pred_raw
)




Measures predictive explanatory performance relative to a baseline.

**Higher is better.**

---

### 20. Store each fold



```python id="w5c2n8"
xgb_oof_results.append({
    "Fold": fold,
    "MAE": mae,
    "RMSE": rmse,
    "R2": r2
})

Stores the three metrics for each fold.

Your actual results

You obtained:

Fold	MAE	RMSE	R²
1	5,195.20	41,449.90	0.8475
2	31,455.28	260,513.71	0.4092
3	1,029.55	2,780.88	0.8705
4	11,496.24	116,050.04	0.8254
5	2,811.77	14,114.92	0.5251
Mean	10,397.61	86,981.89	0.6955

And:

```text id="v4u7n2"
Std R² = 0.2131




---

# How to interpret these results

The average performance is:

### Mean MAE

**10,397.61 kg CO₂e**

On average, the absolute prediction error was approximately **10,398 kg CO₂e** across the five validation folds.

### Mean RMSE

**86,981.89 kg CO₂e**

The much larger RMSE compared with MAE tells you that some predictions have **very large errors/outliers**.

This is an important observation.

### Mean R²

**0.6955**

The model achieved an average R² of approximately **0.696** across the five folds.

So the model demonstrates reasonably strong predictive performance overall, but performance is not equally stable across all folds.

---

# The most important result: Fold variation

Look carefully at your R²:



```text
Fold 1 → 0.8475
Fold 2 → 0.4092
Fold 3 → 0.8705
Fold 4 → 0.8254
Fold 5 → 0.5251

There is substantial variation.

That is reflected by:

Std R² = 0.2131

This is not something you should hide in the viva.

Instead, use it as a limitation/interpretation.

A strong answer is:

“The mean R² was approximately 0.696, but the standard deviation was about 0.213, indicating noticeable variation across folds. This suggests that model performance depends on the composition of the validation subset, which may reflect heterogeneity or extreme observations in the PCF dataset.”

That is much more defensible than saying:

“My model achieved 69.5% accuracy.”

Do not call R² accuracy.

Why is Fold 2 so bad?

Fold 2 has:

MAE  = 31,455
RMSE = 260,514
R²   = 0.409

while Fold 3 has:

MAE  = 1,030
RMSE = 2,781
R²   = 0.871

That is a huge difference.

A reasonable interpretation is:

“The PCF dataset is heterogeneous and likely contains extreme or difficult observations. A validation fold containing influential high-value observations or patterns poorly represented in its training portion can produce much larger errors.”

But do not claim that Fold 2 definitely contains outliers unless you actually inspect the fold data.

If the examiner asks:

“Why exactly was Fold 2 poor?”

The safest answer is:

“The results indicate that Fold 2 contains observations that were harder for the model to predict, but I would need to inspect the actual observations and residuals in that fold to identify the exact cause. I would not attribute it definitively to outliers without that analysis.”

That's a very strong viva answer because you acknowledge what the experiment shows without inventing a cause.

Why is RMSE so much higher than MAE?

Your:

Mean MAE  ≈ 10,398
Mean RMSE ≈ 86,982

RMSE is dramatically higher.

That suggests the model has some very large individual errors.

Why?

Because RMSE squares errors before averaging:

error
  ↓
error²
  ↓
average
  ↓
square root

Therefore, a few very large errors can dominate RMSE.

Viva answer

“The substantially higher RMSE compared with MAE indicates that although the typical absolute error is lower, there are some very large prediction errors. RMSE is sensitive to those large errors because it squares the residuals.”

What does “OOF Development Evaluation” mean?

This can easily confuse you.

Here, OOF does not mean you are producing one prediction for every observation and using those predictions as the final model output.

In this context, you are using out-of-fold validation within the development dataset.

Each observation acts as validation data in one fold and training data in the other four folds.

For example:

Fold 1:
80% → train
20% → validation

Fold 2:
80% → train
20% → validation

...

Fold 5:
80% → train
20% → validation

Together, every development observation gets evaluated in a fold where it was not used for training.

Very important: this is NOT your final test performance

You should be very clear about this.

Your workflow is:

Original dataset
       ↓
80% Development       20% Independent Test
       ↓                       ↓
5-fold CV             KEEP UNTOUCHED
       ↓
PCA selection
       ↓
Model comparison
       ↓
XGBoost tuning
       ↓
Selected XGBoost
       ↓
Final model
       ↓
Evaluate ONCE
       ↓
Independent test performance

Therefore, the:

Mean R² = 0.6955

is development cross-validation performance, not your final independent test-set R².

If you have a later cell that trains the final model on the full development set and evaluates it on test_df, that test result is the one you should report as your final generalisation result.

Why repeat the evaluation after tuning?

The previous tuning cell already calculated CV performance for the 20 configurations.

So an examiner might ask:

“Why are you doing another 5-fold evaluation?”

Answer:

“The tuning stage used cross-validation to compare candidate configurations. After selecting the best configuration, I reran the fixed configuration across the five development folds to obtain a clean fold-level summary of its performance and stability. The independent test set remains untouched for the final evaluation.”

This is a reasonable explanation.

However, be precise: the CV estimate after selecting the best configuration is not fully unbiased for model-selection performance, because the configuration was selected using the same development CV process. The truly independent estimate comes from the untouched test set.

If challenged:

“The post-selection CV result is useful for describing development performance and fold stability, but I treat the untouched test set as the final unbiased evaluation because the development CV was involved in model selection.”

That is an excellent research-methodology answer.

One-minute viva explanation

If your supervisor asks you to explain the whole cell:

“After tuning XGBoost, I selected the best-performing configuration from the 20 randomly sampled parameter sets. I then evaluated that fixed configuration using stratified 5-fold cross-validation on the development dataset. In each fold, I rebuilt the structured features with leakage-aware OOF country target encoding, generated the hybrid representation by combining structured features with SBERT embeddings reduced to 50 PCA components, and trained XGBoost on the log-transformed PCF target. Predictions were converted back to the original PCF scale using expm1, and I calculated MAE, RMSE and R². The average R² was 0.6955, with an R² standard deviation of 0.2131, showing reasonable overall predictive performance but noticeable variation across folds. The independent test set was not used in this evaluation.”

The 5 numbers you should remember

Best XGBoost:
n_estimators       = 200
learning_rate      = 0.08
max_depth          = 6
min_child_weight   = 3
subsample          = 0.7
colsample_bytree   = 0.8

Development CV:
Mean MAE  = 10,397.61
Mean RMSE = 86,981.89
Mean R²   = 0.6955
Std R²    = 0.2131

And the single most important conceptual sentence:

“The 0.6955 R² is a 5-fold development cross-validation result after selecting the XGBoost configuration; it is not the final independent test-set performance.”

For your XGBoost evaluation

Check whether Fold 2 contains influential outliers
Compare pooled OOF metrics with mean fold metrics

Cell-12-XGBOOST HYPERPARAMETER TUNING

TS — Mon, 14 Sep 2026 11:26:37 +0000

# ============================================================
# XGBOOST HYPERPARAMETER TUNING
#
# Fixed:
#   Target    = log1p(PCF)
#   Country   = OOF Target Encoding
#   SBERT     = PCA 50
#   CV        = Stratified 5-Fold
#
# Only XGBoost hyperparameters change
# ============================================================

from sklearn.model_selection import ParameterSampler


# ============================================================
# 1. Generate 20 parameter combinations
# ============================================================

xgb_param_candidates = list(
    ParameterSampler(
        xgb_param_grid,
        n_iter=20,
        random_state=42
    )
)


print(
    f"Testing {len(xgb_param_candidates)} XGBoost parameter sets."
)


# ============================================================
# 2. Storage
# ============================================================

xgb_tuning_results = []


# ============================================================
# 3. Parameter search
# ============================================================

for param_number, params in enumerate(
    xgb_param_candidates,
    start=1
):

    print(
        f"\nParameter set "
        f"{param_number}/{len(xgb_param_candidates)}"
    )

    fold_results = []


    # ========================================================
    # 4. Stratified 5-Fold CV
    # ========================================================

    for fold, (train_idx, valid_idx) in enumerate(
        skf.split(train_df, target_bins),
        start=1
    ):

        # ----------------------------------------------------
        # Fold data
        # ----------------------------------------------------

        X_fold_train = (
            train_df.iloc[train_idx].copy()
        )

        X_fold_valid = (
            train_df.iloc[valid_idx].copy()
        )


        # ----------------------------------------------------
        # Log target
        # ----------------------------------------------------

        y_fold_train_log = (
            y_log.iloc[train_idx].to_numpy()
        )

        y_fold_valid_raw = (
            train_df.iloc[valid_idx][TARGET].to_numpy()
        )


        # ----------------------------------------------------
        # Structured features
        # Country = OOF Target Encoding
        # ----------------------------------------------------

        X_struct_train, X_struct_valid = (
            build_structured_features(
                X_fold_train,
                y_fold_train_log,
                X_fold_valid
            )
        )


        # ----------------------------------------------------
        # SBERT features
        # ----------------------------------------------------

        X_sbert_train = (
            X_train_sbert[train_idx]
        )

        X_sbert_valid = (
            X_train_sbert[valid_idx]
        )


        # ----------------------------------------------------
        # PCA = 50
        # PCA fitted only on training fold
        # ----------------------------------------------------

        (
            X_sbert_train_pca,
            X_sbert_valid_pca,
            _
        ) = apply_pca_to_sbert(
            X_sbert_train,
            X_sbert_valid,
            n_components=50
        )


        # ----------------------------------------------------
        # Hybrid feature fusion
        # ----------------------------------------------------

        X_hybrid_train = fuse_features(
            X_sbert_train_pca,
            X_struct_train
        )

        X_hybrid_valid = fuse_features(
            X_sbert_valid_pca,
            X_struct_valid
        )


        # ----------------------------------------------------
        # Fresh XGBoost model
        # ----------------------------------------------------

        model = XGBRegressor(
            objective="reg:squarederror",
            random_state=42,
            n_jobs=-1,
            **params
        )


        # ----------------------------------------------------
        # Train
        # ----------------------------------------------------

        model.fit(
            X_hybrid_train,
            y_fold_train_log
        )


        # ----------------------------------------------------
        # Predict
        # ----------------------------------------------------

        y_pred_log = model.predict(
            X_hybrid_valid
        )

        y_pred_raw = np.expm1(
            y_pred_log
        )

        y_pred_raw = np.maximum(
            y_pred_raw,
            0
        )


        # ----------------------------------------------------
        # Metrics
        # ----------------------------------------------------

        mae = mean_absolute_error(
            y_fold_valid_raw,
            y_pred_raw
        )

        rmse = np.sqrt(
            mean_squared_error(
                y_fold_valid_raw,
                y_pred_raw
            )
        )

        r2 = r2_score(
            y_fold_valid_raw,
            y_pred_raw
        )


        fold_results.append({
            "Fold": fold,
            "MAE": mae,
            "RMSE": rmse,
            "R2": r2
        })


    # ========================================================
    # 5. Average CV performance
    # ========================================================

    fold_results_df = pd.DataFrame(
        fold_results
    )

    mean_mae = fold_results_df["MAE"].mean()
    mean_rmse = fold_results_df["RMSE"].mean()
    mean_r2 = fold_results_df["R2"].mean()
    std_r2 = fold_results_df["R2"].std()


    # ========================================================
    # 6. Store results
    # ========================================================

    result = {
        "Parameter_Set": param_number,
        "Mean_MAE": mean_mae,
        "Mean_RMSE": mean_rmse,
        "Mean_R2": mean_r2,
        "Std_R2": std_r2
    }

    result.update(params)

    xgb_tuning_results.append(
        result
    )


    print(
        f"  Mean R²: {mean_r2:.4f} | "
        f"Mean RMSE: {mean_rmse:.2f}"
    )


# ============================================================
# XGBOOST TUNING SUMMARY
# ============================================================

xgb_tuning_results_df = pd.DataFrame(
    xgb_tuning_results
)

xgb_tuning_results_df = (
    xgb_tuning_results_df
    .sort_values(
        "Mean_R2",
        ascending=False
    )
    .reset_index(drop=True)
)


display(
    xgb_tuning_results_df
)

XGBoost Hyperparameter Tuning — Overall Purpose

This cell performs randomized hyperparameter tuning of XGBoost.

The important idea is:

Everything in the modelling pipeline is kept fixed, and only the XGBoost hyperparameters are changed.

Fixed components:

Target → log1p(PCF)
Country → OOF target encoding
SBERT → PCA with 50 components
Validation → Stratified 5-fold CV
Features → SBERT + structured features
Metrics → MAE, RMSE, R²

So this experiment specifically answers:

“Given the selected hybrid feature representation, which XGBoost hyperparameter configuration performs best?”

1. Import `ParameterSampler`

from sklearn.model_selection import ParameterSampler

ParameterSampler is used to randomly select combinations of hyperparameters from the search space.

You previously defined:

xgb_param_grid = {
    "n_estimators": [200, 300, 500],
    "learning_rate": [0.03, 0.05, 0.08],
    "max_depth": [3, 4, 6],
    "min_child_weight": [1, 3, 5],
    "subsample": [0.7, 0.8, 1.0],
    "colsample_bytree": [0.7, 0.8, 1.0]
}

There are:

3 × 3 × 3 × 3 × 3 × 3 = 729 possible combinations.

Instead of testing all 729, you randomly test only 20.

2. Generate 20 parameter combinations

xgb_param_candidates = list(
    ParameterSampler(
        xgb_param_grid,
        n_iter=20,
        random_state=42
    )
)

`ParameterSampler(...)`

This randomly samples combinations from xgb_param_grid.

`n_iter=20`

This means:

Test only 20 randomly selected hyperparameter combinations.

If you used all combinations, there would be 729.

So this reduces computation considerably.

`random_state=42`

Makes the random selection reproducible.

If you run the notebook again with the same search space and seed, you should get the same 20 sampled combinations.

`list(...)`

Converts the sampler output into a list so that you can iterate over it multiple times.

3. Print number of combinations

print(
    f"Testing {len(xgb_param_candidates)} XGBoost parameter sets."
)

This simply tells you how many combinations will actually be tested.

Expected:

Testing 20 XGBoost parameter sets.

4. Create result storage

xgb_tuning_results = []

An empty list is created to store the performance of each parameter configuration.

Eventually it will contain approximately:

20 parameter sets

with their:

Mean MAE
Mean RMSE
Mean R²
Std R²
hyperparameter values

5. Start parameter search

for param_number, params in enumerate(
    xgb_param_candidates,
    start=1
):

This loops through the 20 parameter combinations.

For example, one iteration might contain:

{
    "n_estimators": 300,
    "learning_rate": 0.05,
    "max_depth": 4,
    ...
}

param_number gives the configuration number:

1
2
3
...
20

params contains the actual hyperparameter values.

6. Display current parameter set

print(
    f"\nParameter set "
    f"{param_number}/{len(xgb_param_candidates)}"
)

This gives progress information.

For example:

Parameter set 7/20

It does not affect the model.

7. Storage for fold results

fold_results = []

For the current hyperparameter configuration, this list stores the performance from each of the 5 folds.

So one parameter set produces:

Fold 1
Fold 2
Fold 3
Fold 4
Fold 5

Then those five results are averaged.

8. Stratified 5-Fold Cross-Validation

for fold, (train_idx, valid_idx) in enumerate(
    skf.split(train_df, target_bins),
    start=1
):

This is very important.

For each XGBoost parameter set, you run the model through the same 5 CV folds.

Therefore:

20 parameter sets × 5 folds = 100 XGBoost training runs.

This is the main computational cost of the tuning.

Why use the same folds?

For fair comparison.

Every hyperparameter configuration sees the same training/validation partitions.

Therefore, differences in performance are more likely to come from the hyperparameters rather than different data splits.

9. Create fold datasets

X_fold_train = (
    train_df.iloc[train_idx].copy()
)

X_fold_valid = (
    train_df.iloc[valid_idx].copy()
)

The indices generated by skf are used to separate the development dataset into:

fold training data
fold validation data

.copy() creates independent DataFrames.

The independent test set is still untouched.

10. Prepare training target

y_fold_train_log = (
    y_log.iloc[train_idx].to_numpy()
)

The training target is the log-transformed PCF.

Earlier:

y_log = np.log1p(data[TARGET])

So XGBoost learns:

log1p(PCF)

rather than raw PCF.

This helps reduce the influence of extreme values and skew during model training.

11. Prepare validation target

y_fold_valid_raw = (
    train_df.iloc[valid_idx][TARGET].to_numpy()
)

Notice something important:

The validation target is kept in the original PCF scale.

So:

Training → log scale
Evaluation → original PCF scale

This allows the final metrics to have meaningful PCF units.

12. Build structured features

X_struct_train, X_struct_valid = (
    build_structured_features(
        X_fold_train,
        y_fold_train_log,
        X_fold_valid
    )
)

This creates the structured feature representation.

It includes:

numeric features
one-hot encoded categorical features
country target encoding

The important point is that preprocessing is rebuilt inside every fold.

That helps prevent data leakage.

In particular, Country target encoding uses:

y_fold_train_log

and generates OOF encodings for the training portion.

13. Select SBERT features for the fold

X_sbert_train = (
    X_train_sbert[train_idx]
)

X_sbert_valid = (
    X_train_sbert[valid_idx]
)

The SBERT embeddings corresponding to the current fold are selected.

You already generated SBERT embeddings earlier.

You are not retraining SBERT here.

You are simply selecting the relevant rows.

14. Apply PCA = 50

(
    X_sbert_train_pca,
    X_sbert_valid_pca,
    _
) = apply_pca_to_sbert(
    X_sbert_train,
    X_sbert_valid,
    n_components=50
)

This reduces the SBERT representation to 50 dimensions.

The important part is:

n_components=50

Why 50?

Because your earlier PCA experiment compared:

and 50 performed best among those tested options.

Therefore, 50 is now fixed during XGBoost tuning.

Very important viva point

You are not tuning PCA and XGBoost simultaneously here.

The experimental sequence is:

PCA comparison
       ↓
Select PCA = 50
       ↓
Model comparison
       ↓
Choose/tune XGBoost

This makes the experiment easier to interpret.

15. Hybrid feature fusion

X_hybrid_train = fuse_features(
    X_sbert_train_pca,
    X_struct_train
)

and:

X_hybrid_valid = fuse_features(
    X_sbert_valid_pca,
    X_struct_valid
)

The two feature types are horizontally concatenated:

SBERT/PCA features
        +
Structured features
        ↓
Hybrid features

So XGBoost receives both:

semantic information from text
explicit structured information such as year, weight, industry, protocol, country, etc.

16. Create a fresh XGBoost model

model = XGBRegressor(
    objective="reg:squarederror",
    random_state=42,
    n_jobs=-1,
    **params
)

This creates a new XGBoost model for every parameter configuration and every fold.

`objective="reg:squarederror"`

Specifies that this is a regression problem using squared-error loss.

`random_state=42`

Makes the model's stochastic behaviour reproducible where applicable.

`n_jobs=-1`

Uses all available CPU cores for parallel computation.

`**params`

This is particularly important.

It inserts the sampled hyperparameters into the model.

For example:

**params

might effectively become:

n_estimators=500,
learning_rate=0.03,
max_depth=4,
...

So each parameter set creates a different XGBoost configuration.

17. Train XGBoost

model.fit(
    X_hybrid_train,
    y_fold_train_log
)

XGBoost learns the relationship between:

Hybrid features
        ↓
log1p(PCF)

Only the training fold is used here.

The validation fold is not used for fitting.

18. Predict validation data

y_pred_log = model.predict(
    X_hybrid_valid
)

The model predicts PCF on the log scale.

So these predictions represent:

predicted log1p(PCF)

19. Convert predictions back to original scale

y_pred_raw = np.expm1(
    y_pred_log
)

expm1 is the inverse of:

np.log1p()

So:

log1p(PCF)
      ↓
expm1()
      ↓
PCF

Now predictions are back in the original PCF scale.

20. Prevent negative predictions

y_pred_raw = np.maximum(
    y_pred_raw,
    0
)

PCF cannot logically be negative.

This line replaces any negative prediction with zero.

For example:

-5 → 0
20 → 20
100 → 100

This enforces a physically meaningful non-negative prediction.

21. Calculate MAE

mae = mean_absolute_error(
    y_fold_valid_raw,
    y_pred_raw
)

MAE = Mean Absolute Error.

It measures the average absolute difference between:

Actual PCF
vs
Predicted PCF

Lower is better.

Because this is calculated on the raw scale, its unit is approximately:

kg CO₂e

22. Calculate RMSE

rmse = np.sqrt(
    mean_squared_error(
        y_fold_valid_raw,
        y_pred_raw
    )
)

RMSE = Root Mean Squared Error.

It penalises large errors more strongly than MAE.

Lower is better.

This is especially useful when you want to identify whether the model makes some very large prediction errors.

23. Calculate R²

r2 = r2_score(
    y_fold_valid_raw,
    y_pred_raw
)

R² measures how much of the variation in the target is explained by the model relative to a baseline.

Higher is generally better.

For example:

R² = 0.70

means the model explains substantial variation in the validation data, relative to the baseline, but does not mean 70% of every individual prediction is correct.

24. Store fold performance

fold_results.append({
    "Fold": fold,
    "MAE": mae,
    "RMSE": rmse,
    "R2": r2
})

The results for the current fold are stored.

After five folds, you have five sets of:

MAE
RMSE
R²

for that particular hyperparameter configuration.

25. Average the 5 folds

fold_results_df = pd.DataFrame(
    fold_results
)

Converts the fold results into a DataFrame.

Then:

mean_mae = fold_results_df["MAE"].mean()

calculates average MAE across the five folds.

mean_rmse = fold_results_df["RMSE"].mean()

calculates average RMSE.

mean_r2 = fold_results_df["R2"].mean()

calculates average R².

And:

std_r2 = fold_results_df["R2"].std()

calculates the standard deviation of R² across folds.

Why `Std_R2`?

It gives an indication of performance stability across folds.

For example:

Mean R² = 0.70
Std R²  = 0.02

suggests relatively consistent fold performance.

Whereas:

Mean R² = 0.70
Std R²  = 0.20

suggests much greater variation between folds.

26. Store the parameter-set result

result = {
    "Parameter_Set": param_number,
    "Mean_MAE": mean_mae,
    "Mean_RMSE": mean_rmse,
    "Mean_R2": mean_r2,
    "Std_R2": std_r2
}

This creates one summary record for one hyperparameter configuration.

Then:

result.update(params)

adds the actual hyperparameter values.

So one row eventually looks conceptually like:

Parameter Set	Mean MAE	Mean RMSE	Mean R²	Std R²	n_estimators	learning_rate	max_depth
1	...	...	...	...	300	0.05	4

27. Save result

xgb_tuning_results.append(
    result
)

Adds the current parameter configuration to the overall results list.

After all 20 configurations:

xgb_tuning_results
        ↓
20 parameter-set results

28. Print progress

print(
    f"  Mean R²: {mean_r2:.4f} | "
    f"Mean RMSE: {mean_rmse:.2f}"
)

Displays the average performance for the current configuration.

For example:

Mean R²: 0.7123 | Mean RMSE: 85000.25

This is just progress output.

29. Create final tuning DataFrame

xgb_tuning_results_df = pd.DataFrame(
    xgb_tuning_results
)

Converts all 20 parameter-set results into a DataFrame.

30. Sort by Mean R²

xgb_tuning_results_df = (
    xgb_tuning_results_df
    .sort_values(
        "Mean_R2",
        ascending=False
    )
    .reset_index(drop=True)
)

The configurations are sorted from highest to lowest:

Mean R²
   ↓
highest first

Therefore, the first row represents the configuration with the highest mean R² among the 20 tested configurations.

reset_index(drop=True) simply gives the sorted DataFrame a clean index:

0
1
2
...

31. Display final results

display(
    xgb_tuning_results_df
)

Displays the complete tuning table.

You can now compare all 20 configurations.

The complete workflow

The entire cell can be understood as:

729 possible combinations
          ↓
Randomly sample 20
          ↓
For each parameter set
          ↓
    5-fold CV
          ↓
Build structured features
          ↓
OOF country encoding
          ↓
SBERT features
          ↓
PCA = 50
          ↓
Hybrid fusion
          ↓
Train XGBoost
          ↓
Predict validation fold
          ↓
Convert log prediction → raw PCF
          ↓
MAE / RMSE / R²
          ↓
Average 5 folds
          ↓
Repeat for all 20 sets
          ↓
Rank configurations

The most important viva answer

If your supervisor asks:

“What exactly did you do in this cell?”

Say:

“I performed randomized hyperparameter tuning for XGBoost. I defined a search space containing six hyperparameters, which gives 729 possible combinations, but instead of exhaustively testing all of them, I randomly sampled 20 configurations using ParameterSampler. For each configuration, I used the same stratified 5-fold cross-validation and the same hybrid feature pipeline: OOF country target encoding, SBERT reduced to 50 PCA components, and structured features. Only the XGBoost hyperparameters changed. I then calculated MAE, RMSE, and R² on the original PCF scale and ranked the 20 configurations based on mean R².”

Challenging viva questions

“Why did you test only 20 combinations when there are 729?”

“Testing all 729 combinations with 5-fold CV would require 3,645 model fits. I therefore used randomized search to explore 20 combinations within the predefined search space while keeping the computational cost manageable. I acknowledge that this does not guarantee finding the global optimum.”

“Why is this called tuning?”

“Because I am selecting model hyperparameters based on cross-validation performance. The model's learned parameters are still estimated during training; these are settings controlling how that learning process behaves.”

“Did you use the test set during tuning?”

“No. The independent test set was kept separate. Hyperparameter selection was performed using cross-validation within the development data.”

This is one of the most important answers.

“Why did you keep PCA fixed at 50?”

“I previously compared several PCA dimensions using cross-validation. Among the tested values, 50 produced the best validation performance, so I fixed PCA at 50 before tuning XGBoost. This isolates the effect of XGBoost hyperparameters.”

“Are you guaranteed to find the best XGBoost parameters?”

“No. I only identify the best-performing configuration among the 20 sampled configurations. A larger or exhaustive search could potentially find a better configuration.”

“Why do you calculate the metrics after `expm1()`?”

“Because the model is trained on log1p(PCF) to manage the skewed target distribution, but I want MAE, RMSE and R² to represent performance in the original PCF scale, which is more interpretable.”

“Why do you use OOF target encoding here?”

“Country target encoding uses the target variable, so directly calculating the country mean from the same observations could leak target information into the feature. OOF encoding ensures each training observation is encoded using information from other training folds rather than its own target.”

One important distinction to remember

There are three different things happening:

1. Hyperparameter search

20 XGBoost configurations

2. Cross-validation

5 folds per configuration

3. Independent test

Not touched during tuning

Therefore:

20 × 5 = 100 CV model fits, followed later by one final evaluation on the untouched test set.

That distinction is very likely to matter in your viva.

Cell-11-XGBOOST HYPERPARAMETER SEARCH SPACE

TS — Mon, 14 Sep 2026 08:29:05 +0000

# ============================================================
# XGBOOST HYPERPARAMETER SEARCH SPACE
# ============================================================

xgb_param_grid = {
    "n_estimators": [200, 300, 500],
    "learning_rate": [0.03, 0.05, 0.08],
    "max_depth": [3, 4, 6],
    "min_child_weight": [1, 3, 5],
    "subsample": [0.7, 0.8, 1.0],
    "colsample_bytree": [0.7, 0.8, 1.0]
}

XGBoost Hyperparameter Search Space

This cell defines the hyperparameter search space for XGBoost.

The important distinction is:

```text id="xgbs1"
Model parameters
→ learned automatically during training

Hyperparameters
→ chosen before training
→ control how the model learns




Here, you are defining several possible values for each hyperparameter so that a later search procedure can determine which combination performs best.

### 1. Create the parameter grid



```python id="xgb01"
xgb_param_grid = {

This creates a Python dictionary called xgb_param_grid.

The dictionary has the structure:

```text id="xgb02"
Hyperparameter → possible values




For example:



```text id="xgb03"
n_estimators → [200, 300, 500]

A later search algorithm can try combinations from this space.

2. `n_estimators`

```python id="xgb04"
"n_estimators": [200, 300, 500],




Controls the **number of boosting trees/iterations**.

You are considering:



```text id="xgb05"
200
300
500

Conceptually:

```text id="xgb06"
More trees
↓
potentially more learning capacity
↓
but also more computation




Too many trees can contribute to overfitting depending on the other settings.

#### Viva answer

> “I search over 200, 300 and 500 estimators to investigate whether a smaller or larger boosting ensemble gives better validation performance.”

---

### 3. `learning_rate`



```python id="xgb07"
"learning_rate": [0.03, 0.05, 0.08],

Controls how much each new tree contributes to the overall prediction.

You test:

```text id="xgb08"
0.03
0.05
0.08




A smaller learning rate means the model learns more gradually.

There is generally a trade-off between:



```text id="xgb09"
lower learning rate
+
more trees

vs.

higher learning rate
+
fewer trees

That is why learning_rate should not be considered independently of n_estimators.

4. `max_depth`

```python id="xgb10"
"max_depth": [3, 4, 6],




Controls the maximum depth of each decision tree.

You test:



```text id="xgb11"
3
4
6

Smaller depth

Produces simpler trees and can reduce model complexity.

Larger depth

Allows the model to capture more complex interactions, but can increase overfitting risk.

Viva answer

“max_depth controls tree complexity. I include shallow and deeper trees to evaluate the trade-off between capturing complex relationships and controlling overfitting.”

5. `min_child_weight`

```python id="xgb12"
"min_child_weight": [1, 3, 5],




Controls the minimum amount of weight required in a child node before XGBoost makes a further split.

You test:



```text id="xgb13"
1
3
5

Increasing this value generally makes the model more conservative about creating new branches.

Conceptually:

```text id="xgb14"
Lower min_child_weight
→ easier to split
→ more complex trees

Higher min_child_weight
→ harder to split
→ more conservative trees




This can therefore help control overfitting.

---

### 6. `subsample`



```python id="xgb15"
"subsample": [0.7, 0.8, 1.0],

Controls the proportion of training observations used when constructing each boosting tree.

You test:

```text id="xgb16"
0.7 → 70%
0.8 → 80%
1.0 → 100%




For example:



```text id="xgb17"
subsample = 0.8

means approximately 80% of the available training observations are sampled for a boosting iteration.

Using less than 1.0 introduces randomness and can help reduce overfitting.

7. `colsample_bytree`

```python id="xgb18"
"colsample_bytree": [0.7, 0.8, 1.0]




Controls the proportion of features considered for each tree.

You test:



```text id="xgb19"
0.7 → 70% of features
0.8 → 80% of features
1.0 → 100% of features

This is particularly relevant in your project because your hybrid representation can contain a relatively large number of features:

```text id="xgb20"
PCA-reduced SBERT
+
Structured features




Using feature subsampling can introduce additional randomness and potentially improve generalisation.

---

# How Large Is Your Search Space?

You have:



```text id="xgb21"
n_estimators       → 3
learning_rate      → 3
max_depth          → 3
min_child_weight   → 3
subsample          → 3
colsample_bytree   → 3

Therefore, a full Cartesian grid contains:

```text id="xgb22"
3 × 3 × 3 × 3 × 3 × 3
= 729




possible hyperparameter combinations.

That's an important number for your viva.

### Viva question: How many combinations are there?

> “There are 729 possible combinations because each of the six hyperparameters has three candidate values, giving \(3^6 = 729\) combinations.”

If you use ordinary `GridSearchCV` with 5-fold CV, that could mean:



```text id="xgb23"
729 combinations
× 5 folds
= 3,645 model fits

So this is a fairly large search computationally.

Why These Hyperparameters?

These parameters cover several important aspects of XGBoost behaviour:

Hyperparameter	Controls
`n_estimators`	Number of boosting trees
`learning_rate`	Contribution of each tree
`max_depth`	Tree complexity
`min_child_weight`	Minimum requirement for further splitting
`subsample`	Fraction of observations used
`colsample_bytree`	Fraction of features used

So you are not tuning only one aspect of the model.

You are exploring:

```text id="xgb24"
Learning speed
+
Model complexity
+
Number of trees
+
Row sampling
+
Feature sampling




---

# Very Important: This Cell Does NOT Perform the Search

This is a likely examiner trap.

This cell only defines:



```text id="xgb25"
SEARCH SPACE

It does not actually find the best parameters.

You still need a search method such as:

GridSearchCV(...)

or:

RandomizedSearchCV(...)

or another optimisation procedure.

So if the examiner asks:

“Have you performed hyperparameter tuning here?”

Say:

“This cell defines the candidate search space. The actual tuning is performed in the subsequent hyperparameter-search step.”

Do not say the parameters are already optimised just because you created the grid.

Grid Search vs Random Search

Because you have 729 combinations, an examiner may ask why you chose grid search or random search.

Grid Search

Tests every combination.

```text id="xgb26"
729 combinations
→ all are evaluated




Advantage:

> systematic and exhaustive within the specified grid.

Disadvantage:

> computationally expensive.

### Randomized Search

Samples a specified number of combinations.

For example:



```text id="xgb27"
729 possible combinations
        ↓
randomly sample 50
        ↓
evaluate only 50

Advantage:

much cheaper computationally.

Disadvantage:

does not evaluate every possible combination.

Strong viva answer

“The search space contains 729 combinations, so an exhaustive grid search can be computationally expensive, especially when combined with cross-validation. A randomized search can be more computationally efficient, while grid search is appropriate when exhaustive evaluation of the defined candidate values is required.”

Very Important for Your Project: Avoiding Leakage During Hyperparameter Search

Because your project uses:

target encoding
winsorization
PCA
scaling
cross-validation

you must be careful about where each transformation is fitted.

The ideal principle is:

```text id="xgb28"
Training fold
↓
Fit preprocessing
↓
Fit XGBoost
↓
Validation fold
↓
Evaluate




Not:



```text id="xgb29"
Entire development dataset
   ↓
Fit preprocessing
   ↓
Cross-validation

The second approach can produce optimistic results if the preprocessing learns from validation observations.

Particularly important for your Country Target Encoding

Because country target encoding uses the PCF target, it has a direct leakage risk.

Your OOF approach is therefore especially important during CV.

Why Tune XGBoost After Selecting PCA = 50?

Your experimental sequence is becoming:

```text id="xgb30"
Step 1
Feature engineering
↓
SBERT + structured

Step 2
PCA selection
50 / 100 / 150 / 200
↓
Choose 50

Step 3
Model comparison
Linear / Ridge / SVR / RF / XGBoost / CatBoost
↓
Identify strong candidate

Step 4
XGBoost hyperparameter search
↓
Find better XGBoost configuration




This is a logical progression.

You first establish the feature representation, then compare algorithms, and then tune the selected candidate model.

---

# Presentation Wording

> “This cell defines the hyperparameter search space for XGBoost. I vary six parameters: the number of estimators, learning rate, maximum tree depth, minimum child weight, row subsampling, and feature subsampling. Each parameter has three candidate values, resulting in 729 possible combinations. These parameters control different aspects of model complexity, learning rate, and regularisation. This cell only defines the search space; the actual hyperparameter optimisation is performed in the subsequent search procedure using cross-validation.”

### The 5 viva points to memorise

1. **This is a search space, not the search itself.**
2. **There are 729 combinations.**
3. `n_estimators` + `learning_rate` control boosting capacity and learning speed.
4. `max_depth` + `min_child_weight` control tree complexity.
5. `subsample` + `colsample_bytree` introduce row/feature sampling and can help generalisation.

For your XGBoost tuning

* Explain why 729 trials may be inefficient
* Show leakage-safe tuning structure

Cell-10-PCA: 50 + OOF Country Target Encoding

TS — Mon, 14 Sep 2026 08:24:24 +0000

# ============================================================
# MODEL COMPARISON
# PCA = 50 + OOF Country Target Encoding
# Log Target + Stratified 5-Fold CV
# ============================================================

model_results = []


for model_name, model_template in models.items():

    print(f"\nRunning: {model_name}")

    model_fold_results = []


    for fold, (train_idx, valid_idx) in enumerate(
        skf.split(train_df, target_bins),
        start=1
    ):

        # ----------------------------------------------------
        # 1. Fold data
        # ----------------------------------------------------

        X_fold_train = train_df.iloc[train_idx].copy()
        X_fold_valid = train_df.iloc[valid_idx].copy()

        y_fold_train_log = (
            y_log.iloc[train_idx].to_numpy()
        )

        y_fold_valid_raw = (
            train_df.iloc[valid_idx][TARGET].to_numpy()
        )


        # ----------------------------------------------------
        # 2. Structured features
        #    Country = OOF Target Encoding
        # ----------------------------------------------------

        X_struct_train, X_struct_valid = (
            build_structured_features(
                X_fold_train,
                y_fold_train_log,
                X_fold_valid
            )
        )


        # ----------------------------------------------------
        # 3. SBERT features
        # ----------------------------------------------------

        X_sbert_train = X_train_sbert[train_idx]
        X_sbert_valid = X_train_sbert[valid_idx]


        # ----------------------------------------------------
        # 4. PCA = 50
        # ----------------------------------------------------

        (
            X_sbert_train_pca,
            X_sbert_valid_pca,
            _
        ) = apply_pca_to_sbert(
            X_sbert_train,
            X_sbert_valid,
            n_components=50
        )


        # ----------------------------------------------------
        # 5. Hybrid feature fusion
        # ----------------------------------------------------

        X_hybrid_train = fuse_features(
            X_sbert_train_pca,
            X_struct_train
        )

        X_hybrid_valid = fuse_features(
            X_sbert_valid_pca,
            X_struct_valid
        )


        # ----------------------------------------------------
        # 6. Fresh model instance
        # ----------------------------------------------------

        model = clone(model_template)


        # ----------------------------------------------------
        # 7. Scaling
        # ----------------------------------------------------

        scale_features = model_name in [
            "Linear Regression",
            "Ridge Regression",
            "SVR"
        ]


        # ----------------------------------------------------
        # 8. Evaluate
        # ----------------------------------------------------

        mae, rmse, r2 = evaluate_model(
            model=model,
            X_train=X_hybrid_train,
            y_train_log=y_fold_train_log,
            X_valid=X_hybrid_valid,
            y_valid_raw=y_fold_valid_raw,
            scale_features=scale_features
        )


        model_fold_results.append({
            "Model": model_name,
            "Fold": fold,
            "MAE": mae,
            "RMSE": rmse,
            "R2": r2
        })


        print(
            f"  Fold {fold}: "
            f"MAE={mae:.4f}, "
            f"RMSE={rmse:.4f}, "
            f"R²={r2:.4f}"
        )


    # Store all folds
    model_results.extend(
        model_fold_results
    )


# ============================================================
# MODEL COMPARISON SUMMARY
# ============================================================

model_results_df = pd.DataFrame(
    model_results
)


model_summary = (
    model_results_df
    .groupby("Model")
    .agg(
        Mean_MAE=("MAE", "mean"),
        Mean_RMSE=("RMSE", "mean"),
        Mean_R2=("R2", "mean"),
        Std_R2=("R2", "std")
    )
    .reset_index()
    .sort_values(
        "Mean_R2",
        ascending=False
    )
)


display(model_summary)

Model Comparison

This is the main model-comparison cell in your notebook.

At this stage, you have already decided:

PCA = 50 components
Country = OOF target encoding
Target = log-transformed PCF
Evaluation = stratified 5-fold cross-validation
Features = SBERT + structured features

Now the question becomes:

Which regression algorithm performs best with this final hybrid feature representation?

Your pipeline is:

Development data
      ↓
Stratified 5-Fold CV
      ↓
For each model
      ↓
Build structured features
      +
SBERT features
      ↓
PCA = 50
      ↓
Hybrid features
      ↓
Train model
      ↓
Predict validation fold
      ↓
MAE / RMSE / R²
      ↓
Repeat for all 5 folds
      ↓
Average results
      ↓
Compare models

1. Create result storage

```python id="mc01"
model_results = []




Creates an empty list where all model/fold evaluation results will be stored.

For example, one result might eventually look like:



```text id="x8g5dk"
{
    "Model": "Random Forest",
    "Fold": 1,
    "MAE": ...,
    "RMSE": ...,
    "R2": ...
}

Because you have multiple models and five folds, the list will contain one record for every model × fold combination.

2. Loop through every model

```python id="mc02"
for model_name, model_template in models.items():




This loops through the model dictionary you created earlier.

For example:



```text id="2nqk3c"
Linear Regression
Ridge Regression
SVR
Random Forest
XGBoost
CatBoost

For each model, the entire five-fold evaluation is performed.

`model_name`

The readable model name:

```text id="mc03"
"Random Forest"




#### `model_template`

The corresponding model object/configuration.



```text id="mc04"
RandomForestRegressor(...)

3. Print the current model

```python id="mc05"
print(f"\nRunning: {model_name}")




Displays which model is currently being evaluated.

For example:



```text id="v7d2fa"
Running: Random Forest

The \n adds a blank line before the message to make the output easier to read.

4. Create fold-result storage

```python id="mc06"
model_fold_results = []




Creates an empty list for the current model.

For example, while evaluating Random Forest:



```text id="f8n1x4"
Random Forest
    ↓
Fold 1
Fold 2
Fold 3
Fold 4
Fold 5

The five results are stored in this list.

Five-Fold Cross-Validation

```python id="mc07"
for fold, (train_idx, valid_idx) in enumerate(
skf.split(train_df, target_bins),
start=1
):




This generates the five cross-validation folds.

You previously created:



```python id="mc08"
skf = StratifiedKFold(
    n_splits=5,
    shuffle=True,
    random_state=RANDOM_STATE
)

and:

```python id="mc09"
target_bins = pd.qcut(...)




So here you are actually using them.

The output of each iteration is:



```text id="mc10"
train_idx
valid_idx

`train_idx`

Rows used to train the model in this fold.

`valid_idx`

Rows held out for validation in this fold.

The process looks like:

```text id="q4c7w8"
Fold 1 → 80% train / 20% validation
Fold 2 → 80% train / 20% validation
Fold 3 → 80% train / 20% validation
Fold 4 → 80% train / 20% validation
Fold 5 → 80% train / 20% validation




Each observation is used for validation once.

---

# 1. Fold Data



```python id="mc11"
X_fold_train = train_df.iloc[train_idx].copy()
X_fold_valid = train_df.iloc[valid_idx].copy()

These lines select the actual rows for the current fold.

Training fold

```python id="mc12"
X_fold_train = train_df.iloc[train_idx].copy()




Selects the training observations.

### Validation fold



```python id="mc13"
X_fold_valid = train_df.iloc[valid_idx].copy()

Selects the validation observations.

The .copy() creates independent DataFrames so later operations do not unintentionally modify the original train_df.

5. Training target

```python id="mc14"
y_fold_train_log = (
y_log.iloc[train_idx].to_numpy()
)




Selects the log-transformed PCF values corresponding to the training fold.

So the model receives:



```text id="mc15"
X_fold_train
+
y_fold_train_log

6. Validation target

```python id="mc16"
y_fold_valid_raw = (
train_df.iloc[valid_idx][TARGET].to_numpy()
)




Selects the **original/raw PCF values** for the validation observations.

This is important because you eventually evaluate predictions on the original PCF scale.

So:



```text id="mc17"
Training target
→ log scale

Validation target
→ raw PCF scale

2. Build Structured Features

```python id="mc18"
X_struct_train, X_struct_valid = (
build_structured_features(
X_fold_train,
y_fold_train_log,
X_fold_valid
)
)




This calls your structured-feature builder.

It processes the current fold's structured data.

The function performs:



```text id="mc19"
Year + Weight
       ↓
Numeric features

Industry + Protocol + Stage-level availability
       ↓
One-Hot Encoding

Country + training target
       ↓
OOF Target Encoding

and returns:

```text id="mc20"
X_struct_train
X_struct_valid




### Why is this inside the fold?

This is a **critical leakage-control mechanism**.

The preprocessing is rebuilt for every fold.

For example:



```text id="mc21"
Fold 1
   ↓
Fold-1 training data
   ↓
Fit preprocessing
   ↓
Transform Fold-1 validation


Fold 2
   ↓
Fold-2 training data
   ↓
Fit preprocessing
   ↓
Transform Fold-2 validation

You are not fitting the structured preprocessing once on the entire development dataset.

Viva answer

“I perform structured feature engineering separately inside each fold because some transformations learn information from the data. Fitting them on the entire development dataset before cross-validation could allow information from the validation fold to influence the transformation.”

3. Select SBERT Features

```python id="mc22"
X_sbert_train = X_train_sbert[train_idx]
X_sbert_valid = X_train_sbert[valid_idx]




You already generated SBERT embeddings for the development dataset.

Here, you simply select the embeddings corresponding to the current fold.

### Training



```python id="mc23"
X_sbert_train = X_train_sbert[train_idx]

Gets the training observations' SBERT embeddings.

Validation

```python id="mc24"
X_sbert_valid = X_train_sbert[valid_idx]




Gets the validation observations' SBERT embeddings.

You are not fitting SBERT here.

It is a **pretrained fixed encoder**.

---

# 4. PCA = 50



```python id="mc25"
(
    X_sbert_train_pca,
    X_sbert_valid_pca,
    _
) = apply_pca_to_sbert(
    X_sbert_train,
    X_sbert_valid,
    n_components=50
)

Now you apply the PCA configuration selected in your previous experiment.

You already compared:

```text id="mc26"
50
100
150
200




and found that:



```text id="mc27"
50 components

performed best among those tested.

Therefore, this final model-comparison experiment fixes:

```python id="mc28"
n_components=50




### What happens inside?

The PCA function does:



```text id="mc29"
Fold-training SBERT
      ↓
fit PCA
      ↓
50 components

Fold-validation SBERT
      ↓
transform using same PCA
      ↓
50 components

Why is `_` used?

```python id="mc30"
_




The PCA function returns:



```text id="mc31"
X_train_pca
X_valid_pca
pca

But here you don't need the fitted PCA object after this operation.

So:

```python id="mc32"
_




means:

> “I intentionally don't need this returned value.”

This is a common Python convention.

---

# 5. Hybrid Feature Fusion



```python id="mc33"
X_hybrid_train = fuse_features(
    X_sbert_train_pca,
    X_struct_train
)

Combines:

```text id="mc34"
Reduced SBERT features
+
Structured features




into one feature matrix.

Similarly:



```python id="mc35"
X_hybrid_valid = fuse_features(
    X_sbert_valid_pca,
    X_struct_valid
)

creates the validation hybrid features.

Conceptually:

```text id="mc36"
SBERT
+
Structured
↓
Hybrid
↓
ML algorithm




This is the representation on which all models are compared.

---

# 6. Fresh Model Instance



```python id="mc37"
model = clone(model_template)

This is an important line.

clone() creates a fresh, unfitted copy of the model configuration.

Why is this important?

Because you are repeatedly training the model across five folds.

You do not want the model to carry learned information from a previous fold.

Conceptually:

```text id="mc38"
Fold 1
Fresh model
↓
fit
↓
discard

Fold 2
Fresh model
↓
fit
↓
discard




### Viva question: Why use `clone()`?

> “I use `clone()` to create a fresh unfitted estimator for each fold. This prevents parameters learned in one fold from carrying over into another fold.”

This is a **very good technical point**.

---

# 7. Decide Whether Scaling Is Needed



```python id="mc39"
scale_features = model_name in [
    "Linear Regression",
    "Ridge Regression",
    "SVR"
]

This automatically determines whether the current model should receive scaled features.

If the model is:

```text id="mc40"
Linear Regression
Ridge Regression
SVR




then:



```text id="mc41"
scale_features = True

For:

```text id="mc42"
Random Forest
XGBoost
CatBoost




it becomes:



```text id="mc43"
scale_features = False

Why?

Because feature scaling is more important for models that depend on feature magnitudes, distances, or coefficient regularisation.

Tree-based models generally do not require standardisation.

Strong viva answer

“I enable scaling for the linear and SVR models because these algorithms can be sensitive to feature scale, particularly Ridge and RBF-SVR. Tree-based ensemble models generally do not require standardisation, so I leave scaling disabled for them.”

8. Evaluate the Model

```python id="mc44"
mae, rmse, r2 = evaluate_model(
model=model,
X_train=X_hybrid_train,
y_train_log=y_fold_train_log,
X_valid=X_hybrid_valid,
y_valid_raw=y_fold_valid_raw,
scale_features=scale_features
)




This calls your previously defined `evaluate_model()` function.

The function:



```text id="mc45"
1. Optionally scales features
2. Fits the model
3. Predicts log PCF
4. Converts predictions back to raw PCF
5. Prevents negative predictions
6. Calculates MAE
7. Calculates RMSE
8. Calculates R²

The returned values are:

```text id="mc46"
mae
rmse
r2




---

# Store Fold Results



```python id="mc47"
model_fold_results.append({
    "Model": model_name,
    "Fold": fold,
    "MAE": mae,
    "RMSE": rmse,
    "R2": r2
})

A dictionary containing the current fold's results is added to the list.

For example:

```text id="mc48"
Model = Random Forest
Fold = 1
MAE = ...
RMSE = ...
R2 = ...




After five folds, you have five records for that model.

---

# Print Fold Performance



```python id="mc49"
print(
    f"  Fold {fold}: "
    f"MAE={mae:.4f}, "
    f"RMSE={rmse:.4f}, "
    f"R²={r2:.4f}"
)

Displays the performance for the current fold.

`:.4f`

Means the number is displayed with four decimal places.

For example:

```text id="mc50"
R²=0.6547




This is mainly for readable output.

---

# Store All Folds for the Model



```python id="mc51"
model_results.extend(
    model_fold_results
)

Once all five folds for a model have finished, their results are added to the main model_results list.

The process is:

```text id="mc52"
Linear Regression
→ 5 results
↓
model_results

Ridge
→ 5 results
↓
model_results

SVR
→ 5 results
↓
model_results

...




---

# MODEL COMPARISON SUMMARY

Now you convert all those fold-level results into a summary table.

### 1. Convert results into DataFrame



```python id="mc53"
model_results_df = pd.DataFrame(
    model_results
)

This converts the list of dictionaries into a pandas DataFrame.

Conceptually:

Model	Fold	MAE	RMSE	R²
Linear Regression	1	...	...	...
Linear Regression	2	...	...	...
...	...	...	...	...
Random Forest	5	...	...	...

2. Group by Model

```python id="mc54"
model_summary = (
model_results_df
.groupby("Model")




Groups all five folds belonging to the same model.

For example:



```text id="mc55"
Random Forest
→ Fold 1
→ Fold 2
→ Fold 3
→ Fold 4
→ Fold 5

3. Calculate Mean MAE

```python id="mc56"
Mean_MAE=("MAE", "mean"),




Calculates the average MAE across the five folds.

Lower is better.

---

# 4. Calculate Mean RMSE



```python id="mc57"
Mean_RMSE=("RMSE", "mean"),

Calculates average RMSE across the five folds.

Lower is better.

5. Calculate Mean R²

```python id="mc58"
Mean_R2=("R2", "mean"),




Calculates average R² across the five folds.

Higher is better.

---

# 6. Calculate Standard Deviation of R²



```python id="mc59"
Std_R2=("R2", "std")

This is an especially useful addition.

It measures how much the R² varies between folds.

For example:

```text id="mc60"
Mean R² = 0.65
Std R² = 0.02




suggests relatively stable fold performance.

But:



```text id="mc61"
Mean R² = 0.65
Std R²  = 0.20

would indicate much greater variability.

So Std_R2 helps you understand stability, not just average performance.

7. Reset Index

```python id="mc62"
.reset_index()




Converts the grouped model name back into a normal DataFrame column.

---

# 8. Sort by R²



```python id="mc63"
.sort_values(
    "Mean_R2",
    ascending=False
)

Sorts models according to mean R².

```text id="mc64"
Highest R²
↓
Best position




because:



```text id="mc65"
ascending=False

means descending order.

Important

This sorting does not mean R² is the only metric that matters.

You should still examine:

Mean MAE
Mean RMSE
Mean R²
Std R²

A model with the highest R² but dramatically worse MAE/RMSE should be investigated rather than automatically declared best.

9. Display the Summary

```python id="mc66"
display(model_summary)




Displays the final model-comparison table.

Conceptually:

| Model   | Mean MAE ↓ | Mean RMSE ↓ | Mean R² ↑ | Std R² ↓ |
| ------- | ---------: | ----------: | --------: | -------: |
| Model A |        ... |         ... |       ... |      ... |
| Model B |        ... |         ... |       ... |      ... |
| Model C |        ... |         ... |       ... |      ... |

This becomes the basis for your model selection.

---

# The Most Important Concept in This Cell

Your model comparison is **fairer than simply training all models once** because every model goes through essentially the same evaluation structure:



```text id="mc67"
Same development data
        ↓
Same 5 folds
        ↓
Same PCA = 50
        ↓
Same hybrid representation
        ↓
Same target transformation
        ↓
Same evaluation metrics
        ↓
Different ML algorithm

Therefore, the major experimental difference is the model algorithm, rather than changing the dataset or evaluation strategy for each model.

That is exactly what you want in a model comparison.

Very Important Viva Question: Why Compare Models on the Same Features?

“To make the comparison fair. I fixed the feature representation at PCA-50 hybrid features and used the same cross-validation folds, target transformation and evaluation metrics for each algorithm. Therefore, differences in performance are primarily attributable to the modelling algorithm rather than different preprocessing choices.”

Excellent answer.

Very Important Viva Question: Why Is PCA Fixed at 50 Here?

“I previously evaluated 50, 100, 150 and 200 components using five-fold cross-validation. Since 50 produced the strongest average performance among those candidates, I fixed PCA at 50 for the subsequent model comparison. This avoids simultaneously changing the PCA dimension and the regression algorithm.”

This is a very important experimental-design answer.

Very Important Viva Question: Is This Data Leakage?

Your answer should distinguish the different stages.

“The independent test set is not used in this model comparison. Within each cross-validation fold, the structured preprocessing and PCA are fitted only on the fold-training data and then applied to the validation fold. The regression model is also freshly fitted for each fold. This prevents validation information from influencing model training or learned preprocessing.”

Your leakage-control pipeline

```text id="mc68"
Fold training
↓
Fit winsorization
Fit OHE
Fit target encoding
Fit PCA
Fit scaler if required
Fit ML model
↓
Validation
↓
Transform only
↓
Predict




That is one of the strongest methodological points in your notebook.

---

# Very Important Viva Question: Why `clone()`?

> “Because each fold must start with an unfitted model. `clone()` creates a fresh estimator with the same hyperparameter configuration, preventing learned parameters from one fold carrying into another fold.”

---

# Very Important Viva Question: Why Do You Use Raw Target for Metrics?

> “Although the models are trained on the log-transformed target, I inverse-transform the predictions back to the original PCF scale before calculating MAE, RMSE and R². This makes the evaluation directly interpretable in the original target scale.”

---

# Very Important Viva Question: Why Not Use the Independent Test Set Here?

> “Because the test set should remain untouched during model and feature-selection decisions. I use cross-validation within the development data for model comparison and reserve the independent test set for the final unbiased evaluation.”

This is perhaps one of the **most important answers in your whole viva**.

---

# One Subtle Point You Should Understand

Your code uses:



```python id="mc69"
y_fold_train_log

for model training, while:

```python id="mc70"
target_bins




is used for **stratification**.

These are not the same thing.



```text id="mc71"
y_fold_train_log
→ actual training target

target_bins
→ temporary labels used only to create balanced folds

The model never learns to predict target_bins.

The prediction task remains:

```text id="mc72"
Hybrid features
↓
Continuous PCF prediction




---

# What This Experiment Ultimately Answers

This cell answers:

> **“Given the same PCA-50 hybrid representation and the same evaluation procedure, which regression algorithm performs best for PCF prediction?”**

Your previous PCA experiment answered:

> **“How many SBERT components should I retain?”**

So your experimental logic is:



```text id="mc73"
Experiment 1
PCA = 50 / 100 / 150 / 200
        ↓
Select best PCA dimension
        ↓
50


Experiment 2
Fix PCA = 50
        ↓
Compare ML algorithms
        ↓
Select best model

That is a much stronger explanation than simply saying:

“I tried different models.”

Presentation Script

“After selecting 50 PCA components from the previous cross-validation experiment, I use that configuration for the final model comparison. I evaluate multiple regression algorithms using stratified five-fold cross-validation on the development data. For each fold, I first separate the training and validation observations and build the structured features using only the fold-training information. I then select the corresponding SBERT embeddings, fit PCA with 50 components only on the fold-training embeddings, and transform the validation embeddings using the same PCA transformation. The reduced SBERT features are then fused with the structured features to form the hybrid representation.

For each model, I create a fresh estimator using clone() so that models do not carry learned information between folds. Feature scaling is enabled for Linear Regression, Ridge and SVR, while it is disabled for the tree-based ensemble models. Each model is trained on the log-transformed PCF target, and predictions are converted back to the original PCF scale before calculating MAE, RMSE and R². Finally, I average the metrics across the five folds and also calculate the standard deviation of R² to assess performance stability. The models are then ranked by mean R² while considering MAE and RMSE as complementary performance measures.”

The 10 lines you should know perfectly for viva

```text id="mc74"

skf.split(...)
→ creates the five train/validation folds
build_structured_features(...)
→ creates leakage-controlled structured features
X_train_sbert[train_idx]
→ selects SBERT features for the training fold
apply_pca_to_sbert(..., n_components=50)
→ reduces SBERT dimensions using training-fold-fitted PCA
fuse_features(...)
→ combines SBERT + structured features
clone(model_template)
→ creates a fresh model for each fold
scale_features = ...
→ scaling depends on the algorithm
evaluate_model(...)
→ trains, predicts and calculates metrics
groupby("Model").agg(...)
→ averages performance across five folds
sort_values("Mean_R2", ascending=False)
→ ranks models by average R²

The one-sentence summary

“I fixed the selected PCA dimension at 50 and compared different regression algorithms using the same leakage-controlled hybrid features and stratified five-fold development evaluation, selecting the strongest model based on cross-validated predictive performance.”

cell-09-MODEL EVALUATION

TS — Mon, 14 Sep 2026 08:22:44 +0000

# ============================================================
# MODEL EVALUATION
# ============================================================

from sklearn.preprocessing import StandardScaler


def evaluate_model(
    model,
    X_train,
    y_train_log,
    X_valid,
    y_valid_raw,
    scale_features=False
):
    """
    Train on log-transformed PCF and evaluate
    predictions in the original PCF scale.
    """

    # Feature scaling
    if scale_features:

        scaler = StandardScaler()

        X_train_model = scaler.fit_transform(X_train)
        X_valid_model = scaler.transform(X_valid)

    else:

        X_train_model = X_train
        X_valid_model = X_valid


    # Train model
    model.fit(
        X_train_model,
        y_train_log
    )


    # Predict log(PCF)
    y_pred_log = model.predict(
        X_valid_model
    )


    # Convert predictions back to PCF scale
    y_pred_raw = np.expm1(
        y_pred_log
    )

    # Prevent negative PCF predictions
    y_pred_raw = np.maximum(
        y_pred_raw,
        0
    )


    # Evaluation metrics
    mae = mean_absolute_error(
        y_valid_raw,
        y_pred_raw
    )

    rmse = np.sqrt(
        mean_squared_error(
            y_valid_raw,
            y_pred_raw
        )
    )

    r2 = r2_score(
        y_valid_raw,
        y_pred_raw
    )


    return mae, rmse, r2

Model Evaluation

This function creates a reusable evaluation pipeline for all of your regression models.

The main purpose is to make sure every model is evaluated in the same way:

Training features
      ↓
Optional scaling
      ↓
Train using log-transformed PCF
      ↓
Predict log-transformed PCF
      ↓
Convert prediction back to raw PCF
      ↓
Calculate MAE, RMSE, R²

This is an important cell because it standardises your model comparison.

1. Import StandardScaler

```python id="scaler01"
from sklearn.preprocessing import StandardScaler




This imports `StandardScaler`.

StandardScaler standardises numerical feature values approximately to:



```text
mean = 0
standard deviation = 1

The transformation is learned from the training data.

It is particularly relevant for models such as:

Linear Regression
Ridge Regression
SVR

Tree-based models such as Random Forest generally do not require feature scaling.

2. Define the evaluation function

```python id="eval02"
def evaluate_model(
model,
X_train,
y_train_log,
X_valid,
y_valid_raw,
scale_features=False
):




You define a reusable function called:



```text
evaluate_model()

It accepts six inputs.

`model`

The regression model you want to evaluate.

For example:

Linear Regression
Ridge
SVR
Random Forest
XGBoost
CatBoost

`X_train`

Training feature matrix.

This may contain your hybrid features:

PCA-reduced SBERT
+
structured features

`y_train_log`

The log-transformed PCF target used for model training.

`X_valid`

Validation feature matrix.

`y_valid_raw`

The actual PCF values in their original/raw scale.

`scale_features=False`

This controls whether feature scaling is applied.

By default:

scale_features = False

So no scaling happens unless you explicitly request it.

3. Function documentation

```python id="eval03"
"""
Train on log-transformed PCF and evaluate
predictions in the original PCF scale.
"""




This is a **docstring**.

It documents what the function does.

The key information is:



```text
Training → log-transformed PCF

Evaluation → original PCF scale

That distinction is important in your project.

Feature Scaling

```python id="eval04"
if scale_features:




This checks whether the function was called with:



```python
scale_features=True

If yes, scaling is performed.

If false, the original feature matrices are used.

4. Create scaler

```python id="eval05"
scaler = StandardScaler()




Creates a StandardScaler object.

The scaler will learn:

* feature means
* feature standard deviations

from the training data.

---

### 5. Fit and transform training features



```python id="eval06"
X_train_model = scaler.fit_transform(X_train)

This does two operations:

fit
+
transform

`fit`

Learns the mean and standard deviation from training data only.

`transform`

Uses those learned values to standardise the training features.

Conceptually:

X_train
   ↓
learn mean/std
   ↓
standardise
   ↓
X_train_model

6. Transform validation features

```python id="eval07"
X_valid_model = scaler.transform(X_valid)




This applies the **same training-derived scaling parameters** to the validation data.

Notice:



```text
Training:
fit_transform()

Validation:
transform()

This is very important for avoiding leakage.

Why not use `fit_transform()` on validation?

Because then the validation data would be used to calculate its own mean and standard deviation.

That means information from validation would influence the preprocessing.

Your approach is correct:

Training
 ↓
fit scaler
 ↓
learn parameters

Validation
 ↓
use same parameters

Viva answer

“I fit the scaler only on the training data and then apply the learned transformation to validation data. This prevents information from the validation set influencing the preprocessing.”

If Scaling Is Not Requested

```python id="eval08"
else:

X_train_model = X_train
X_valid_model = X_valid




If:



```python
scale_features=False

the original feature matrices are used without scaling.

So:

scale_features=True
→ StandardScaler

scale_features=False
→ original features

This allows the same evaluation function to work with models that do and do not benefit from scaling.

Train the Model

```python id="eval09"
model.fit(
X_train_model,
y_train_log
)




This trains the selected regression model.

The model receives:



```text
X_train_model
+
y_train_log

The target is the log-transformed PCF.

So the model learns to predict approximately:

log(PCF + 1)

rather than the raw PCF.

Predict in Log Space

```python id="eval10"
y_pred_log = model.predict(
X_valid_model
)




The trained model predicts the validation observations.

Because the model was trained using `y_train_log`, these predictions are also in log space.

Therefore:



```text
y_pred_log

is not yet the final PCF value in the original units.

Convert Predictions Back to PCF Scale

```python id="eval11"
y_pred_raw = np.expm1(
y_pred_log
)




This reverses the earlier log transformation.

If your target transformation was:



```text
log(PCF + 1)

then:

expm1(prediction)

returns the prediction to approximately the original PCF scale.

Conceptually:

Raw PCF
   ↓
log1p()
   ↓
Log PCF
   ↓
ML model
   ↓
Predicted Log PCF
   ↓
expm1()
   ↓
Predicted Raw PCF

Why do this before calculating metrics?

Because you want your MAE and RMSE to be expressed in the original PCF units, making them easier to interpret.

Prevent Negative PCF Predictions

```python id="eval12"
y_pred_raw = np.maximum(
y_pred_raw,
0
)




This prevents negative predicted PCF values.

For every prediction:



```text
if prediction < 0
    → 0

if prediction ≥ 0
    → keep it

This is a domain-informed constraint because PCF is non-negative.

Viva question: Can your model produce negative predictions?

Depending on the regression algorithm and transformation, yes, a model may produce a log-space value that back-transforms to a value that needs domain checking. This line ensures the final reported prediction is not negative.

A safe answer:

“I apply a non-negativity constraint after inverse transformation because negative PCF values are not physically meaningful.”

Evaluation Metrics

Now the function calculates three metrics.

```python id="eval13"
mae = mean_absolute_error(
y_valid_raw,
y_pred_raw
)




### MAE

Mean Absolute Error measures the average absolute difference between:



```text
actual PCF
vs.
predicted PCF

Lower is better.

For example, an MAE of 100 means the average absolute prediction error is 100 PCF units, assuming the target's unit is kg CO₂e.

RMSE

```python id="eval14"
rmse = np.sqrt(
mean_squared_error(
y_valid_raw,
y_pred_raw
)
)




First:



```python
mean_squared_error(...)

calculates Mean Squared Error.

Then:

np.sqrt(...)

takes the square root.

This produces RMSE.

RMSE penalises larger errors more heavily than MAE because the errors are squared before averaging.

Lower is better.

R²

```python id="eval15"
r2 = r2_score(
y_valid_raw,
y_pred_raw
)




Calculates the coefficient of determination, R².

It evaluates how well the predictions explain variation in the target relative to a baseline.

Generally:



```text
Higher R² → better

But R² should always be interpreted together with MAE and RMSE.

Return the Results

```python id="eval16"
return mae, rmse, r2




The function returns three values:



```text
MAE
RMSE
R²

This makes it easy to store and compare the performance of different models.

For example, conceptually:

Model              MAE       RMSE       R²
------------------------------------------------
Linear Regression  ...       ...        ...
Ridge              ...       ...        ...
SVR                ...       ...        ...
Random Forest      ...       ...        ...
XGBoost            ...       ...        ...
CatBoost           ...       ...        ...

The Most Important Leakage Principle in This Function

Your preprocessing follows:

TRAIN
 ↓
fit scaler
 ↓
transform train


VALIDATION
 ↓
transform using training scaler

Not:

TRAIN + VALIDATION
 ↓
fit scaler

This same principle applies to your other learned transformations:

Winsorization
→ fit on training only

One-Hot Encoder
→ fit on training only

Target Encoder
→ training/OOF strategy

PCA
→ fit on training only

Scaler
→ fit on training only

That is a very strong viva point because it shows you understand preprocessing leakage rather than simply applying preprocessing mechanically.

Why Scale Some Models but Not Others?

This is another likely examiner question.

Linear Regression

Scaling can make coefficients numerically more comparable, although ordinary linear regression predictions are invariant to simple feature rescaling under standard conditions.

Ridge

Scaling is important because Ridge applies regularisation to coefficients. Different feature scales can otherwise cause the penalty to affect features unevenly.

SVR

Scaling is particularly important for SVR because distance calculations and the RBF kernel are sensitive to feature scale.

Random Forest

Usually does not require scaling because tree splits are based on feature thresholds rather than distances or coefficient magnitudes.

Strong viva answer

“I made feature scaling optional because its importance depends on the algorithm. Models such as Ridge and especially RBF-SVR are sensitive to feature scale, whereas tree-based models such as Random Forest generally do not require standardisation. The function therefore allows scaling to be enabled when appropriate.”

Very Important Question: Why Train on Log Target but Evaluate on Raw Target?

Answer this exactly:

“The log transformation helps manage the skewness and extreme values of the PCF target during model training. After prediction, I apply the inverse transformation so that the predictions return to the original PCF scale. I then calculate MAE, RMSE and R² against the raw validation target, making the reported performance interpretable in the original PCF units.”

Very Important Question: Why Use Three Metrics?

You can say:

“I use MAE, RMSE and R² because they provide complementary information. MAE represents the average absolute error, RMSE gives greater weight to large errors, and R² indicates how well the predictions explain variation in the target. Using multiple metrics reduces the risk of judging a model from a single perspective.”

One Important Technical Point

Your function itself does not perform cross-validation.

It evaluates:

one training set
+
one validation set

Cross-validation happens outside this function when you repeatedly call it for different folds.

So:

evaluate_model()
→ one train/validation evaluation

5-fold CV loop
→ calls evaluate_model() five times

This distinction may come up in your viva.

Presentation wording

“This function standardises the evaluation process across my regression models. It optionally applies feature scaling, fitting the scaler only on the training data and then transforming the validation data. The selected model is trained using the log-transformed PCF target. Predictions are generated in log space and then converted back to the original PCF scale using the inverse transformation. I apply a non-negativity constraint because negative PCF values are not physically meaningful. Finally, I calculate MAE, RMSE and R² on the original PCF scale and return these metrics for model comparison.”

Viva checklist for this cell

fit_transform() → training data
transform() → validation data
Scaling → model-dependent
Training target → log scale
Prediction → log scale
expm1() → back to raw PCF
maximum(..., 0) → no negative PCF
MAE → average absolute error
RMSE → emphasises large errors
R² → explained variation
Metrics → calculated on raw PCF scale
Function → evaluates one train/validation split; CV is handled outside

For this evaluation function

Compare scaled and unscaled models
Check raw-scale metric bias

Cell-08-MODEL IMPORTS

TS — Mon, 14 Sep 2026 08:06:40 +0000

# ============================================================
# MODEL IMPORTS
# ============================================================

from sklearn.linear_model import LinearRegression, Ridge
from sklearn.svm import SVR
from sklearn.ensemble import RandomForestRegressor

# Optional boosting models
try:
    from xgboost import XGBRegressor
    XGB_AVAILABLE = True
except ImportError:
    XGB_AVAILABLE = False

try:
    from catboost import CatBoostRegressor
    CATBOOST_AVAILABLE = True
except ImportError:
    CATBOOST_AVAILABLE = False


# ============================================================
# MODEL CONFIGURATION
# ============================================================

models = {

    "Linear Regression": LinearRegression(),

    "Ridge Regression": Ridge(
        alpha=1.0
    ),

    "SVR": SVR(
        kernel="rbf",
        C=10.0,
        epsilon=0.1
    ),

    "Random Forest": RandomForestRegressor(
        n_estimators=300,
        random_state=42,
        n_jobs=-1
    )
}


# XGBoost
if XGB_AVAILABLE:

    models["XGBoost"] = XGBRegressor(
        n_estimators=300,
        learning_rate=0.05,
        max_depth=6,
        subsample=0.8,
        colsample_bytree=0.8,
        objective="reg:squarederror",
        random_state=42,
        n_jobs=-1
    )


# CatBoost
if CATBOOST_AVAILABLE:

    models["CatBoost"] = CatBoostRegressor(
        iterations=300,
        learning_rate=0.05,
        depth=6,
        loss_function="RMSE",
        random_seed=42,
        verbose=False
    )


print("Models selected:")
for name in models:
    print(" -", name)

Model Imports

This cell defines the candidate regression models that will be compared for PCF prediction.

The key idea is that you are not relying on only one algorithm. You are testing different model families:

Linear model
     ↓
Linear Regression

Regularised linear model
     ↓
Ridge Regression

Kernel-based model
     ↓
SVR

Tree ensemble
     ↓
Random Forest

Boosting ensembles
     ↓
XGBoost / CatBoost

This gives you a broader comparison of modelling approaches.

1. Import Linear Regression and Ridge

from sklearn.linear_model import LinearRegression, Ridge

This imports two linear regression algorithms.

`LinearRegression`

Ordinary linear regression assumes the target can be represented as a linear combination of the input features.

In simplified form:

Prediction = intercept + weighted features

It is useful as a simple baseline model.

Viva question: Why include Linear Regression?

“I included Linear Regression as a simple baseline. It provides a reference point against which the more complex nonlinear models can be compared.”

`Ridge`

Ridge Regression is a regularised version of linear regression.

It adds a penalty to large coefficients.

This can be useful when you have many correlated features, which is particularly relevant when using high-dimensional embedding features.

Viva question: Why Ridge?

“Ridge provides a regularised linear baseline. Since the hybrid representation can contain many correlated features, Ridge can reduce the influence of excessively large coefficients and provide a useful comparison with ordinary linear regression.”

2. Import SVR

from sklearn.svm import SVR

This imports Support Vector Regression.

Your configuration later uses:

kernel="rbf"

The RBF kernel allows SVR to model nonlinear relationships between the features and PCF.

So your model comparison includes a nonlinear kernel-based approach rather than only linear models.

3. Import Random Forest

from sklearn.ensemble import RandomForestRegressor

Random Forest is an ensemble of decision trees.

It can model nonlinear relationships and interactions between features.

This is particularly useful for your hybrid feature representation because relationships between:

SBERT features
+
structured features

may not be purely linear.

Optional XGBoost Import

try:
    from xgboost import XGBRegressor
    XGB_AVAILABLE = True
except ImportError:
    XGB_AVAILABLE = False

This is a safe optional import.

`try`

Python first attempts:

from xgboost import XGBRegressor

If XGBoost is installed, the import succeeds.

Then:

XGB_AVAILABLE = True

means the model is available.

`except ImportError`

If XGBoost is not installed, Python catches the import error.

Instead of stopping the entire notebook, you set:

XGB_AVAILABLE = False

This means the rest of the notebook can continue without XGBoost.

Viva question: Why use `try/except`?

“XGBoost and CatBoost are optional dependencies. I used try/except so that the notebook remains executable even if one of these external libraries is unavailable.”

That's a good software-engineering decision.

Optional CatBoost Import

try:
    from catboost import CatBoostRegressor
    CATBOOST_AVAILABLE = True
except ImportError:
    CATBOOST_AVAILABLE = False

This does the same thing for CatBoost.

If installed:

CATBOOST_AVAILABLE = True

If not:

CATBOOST_AVAILABLE = False

Therefore, your core scikit-learn models can still run.

Model Configuration

models = {

You create a Python dictionary called models.

The dictionary stores:

Model name → Model object

For example:

"Random Forest" → RandomForestRegressor(...)

This is convenient because later you can iterate through all models using something like:

for name, model in models.items():

instead of writing separate training code for every model.

1. Linear Regression

"Linear Regression": LinearRegression(),

The key:

"Linear Regression"

is simply the human-readable model name.

The value:

LinearRegression()

creates the regression model.

This serves as a baseline.

2. Ridge Regression

"Ridge Regression": Ridge(
    alpha=1.0
),

Creates a Ridge Regression model.

`alpha=1.0`

alpha controls the strength of the regularisation.

Conceptually:

Higher alpha
→ stronger coefficient penalty

Lower alpha
→ weaker coefficient penalty

Here you use:

alpha = 1.0

as the selected configuration.

Viva question: Is `alpha=1.0` automatically optimal?

Answer:

“No. It is a chosen hyperparameter configuration. A more extensive experiment could tune alpha using cross-validation.”

This is safer than claiming that 1.0 is universally optimal.

3. SVR

"SVR": SVR(
    kernel="rbf",
    C=10.0,
    epsilon=0.1
),

Creates the Support Vector Regression model.

`kernel="rbf"`

RBF stands for Radial Basis Function.

It allows SVR to model nonlinear relationships.

This is useful because PCF relationships may not be linear.

`C=10.0`

C controls the trade-off between:

fitting the training data closely
allowing some prediction error

A larger C generally puts more emphasis on reducing training errors.

Again, don't claim 10 is universally optimal.

`epsilon=0.1`

Defines an epsilon-insensitive region around the regression prediction.

Conceptually:

Prediction
     ↓
small errors inside epsilon
     ↓
not penalised in the same way

Viva question: Why RBF SVR?

“I used the RBF kernel because it allows SVR to capture nonlinear relationships between the hybrid features and the continuous PCF target.”

4. Random Forest

"Random Forest": RandomForestRegressor(
    n_estimators=300,
    random_state=42,
    n_jobs=-1
)

Creates a Random Forest regression model.

`n_estimators=300`

Creates an ensemble of 300 trees.

The final prediction is based on the combined predictions of these trees.

`random_state=42`

Makes the stochastic parts reproducible.

`n_jobs=-1`

Uses all available CPU cores for parallel processing.

Viva question: Why Random Forest?

“Random Forest can capture nonlinear relationships and interactions without requiring the feature-target relationship to be linear. It is therefore a useful tree-based model for evaluating the hybrid representation.”

XGBoost

if XGB_AVAILABLE:

This checks whether XGBoost was successfully imported.

If:

XGB_AVAILABLE = True

then the following model is added.

models["XGBoost"] = XGBRegressor(

This adds XGBoost to the dictionary.

`n_estimators=300`

n_estimators=300

Uses 300 boosting iterations/trees.

`learning_rate=0.05`

learning_rate=0.05

Controls how strongly each new tree contributes to the overall model.

A smaller learning rate generally means the model learns more gradually.

`max_depth=6`

max_depth=6

Controls the maximum depth of individual trees.

Greater depth can capture more complex relationships but can also increase overfitting risk.

`subsample=0.8`

subsample=0.8

Uses approximately 80% of the training observations for each boosting stage.

This introduces randomness and can help reduce overfitting.

`colsample_bytree=0.8`

colsample_bytree=0.8

Uses approximately 80% of the features for each tree.

This introduces feature-level randomness.

`objective="reg:squarederror"`

objective="reg:squarederror"

Specifies that XGBoost is solving a regression problem using squared error.

This is appropriate because PCF is a continuous target.

`random_state=42`

Provides reproducibility.

`n_jobs=-1`

Uses available CPU resources for parallel processing.

CatBoost

if CATBOOST_AVAILABLE:

Checks whether CatBoost is available.

If it is:

models["CatBoost"] = CatBoostRegressor(

adds a CatBoost regression model.

`iterations=300`

iterations=300

Uses 300 boosting iterations.

`learning_rate=0.05`

Controls how strongly each boosting iteration contributes.

`depth=6`

Controls tree depth.

`loss_function="RMSE"`

loss_function="RMSE"

Specifies RMSE as the optimisation loss.

`random_seed=42`

Provides reproducibility.

`verbose=False`

Suppresses CatBoost's detailed training output.

This keeps the notebook output cleaner.

Print Selected Models

print("Models selected:")

Prints a heading.

Then:

for name in models:

loops through the dictionary keys.

Each key is a model name.

For example:

Linear Regression
Ridge Regression
SVR
Random Forest
XGBoost
CatBoost

Finally:

print(" -", name)

prints each model name.

So the output might look like:

Models selected:
 - Linear Regression
 - Ridge Regression
 - SVR
 - Random Forest
 - XGBoost
 - CatBoost

The exact final two depend on whether those libraries are installed.

Why Compare These Models?

This is a very likely viva question.

You are deliberately comparing different model families.

Linear Regression
→ simple linear baseline

Ridge
→ regularised linear model

SVR
→ nonlinear kernel model

Random Forest
→ bagging/tree ensemble

XGBoost
→ gradient boosting

CatBoost
→ gradient boosting

This gives you a broader experimental comparison.

Strong viva answer

“I selected models from different algorithmic families so that I could evaluate whether the hybrid PCF features work better with linear, regularised, kernel-based, bagging, or boosting approaches. Linear Regression provides a simple baseline, Ridge addresses regularisation, SVR captures nonlinear relationships through an RBF kernel, and Random Forest, XGBoost and CatBoost provide tree-based ensemble alternatives.”

Important Question: Why Not Just Use Random Forest?

If your examiner asks:

“If Random Forest works well, why did you test all these models?”

Answer:

“Because selecting a model based only on prior expectation would introduce unnecessary assumptions. Comparing multiple model families allows me to empirically determine which algorithm works best with the engineered hybrid representation.”

Important Question: Are XGBoost and CatBoost Necessary?

Answer:

“They are optional comparison models rather than mandatory components of the pipeline. I included them to broaden the model comparison. If the libraries are unavailable, the core scikit-learn models can still be evaluated.”

Important Question: Are these hyperparameters tuned?

Based only on this code, these are predefined configurations, not a comprehensive hyperparameter search.

So say:

“These are selected model configurations used for comparison. They are not the result of an exhaustive hyperparameter optimisation procedure.”

This is a very defensible answer.

Important Question: Which model is best?

You cannot answer that from this cell alone.

This cell only defines the models.

The actual answer must come from your later evaluation results.

The process is:

Models defined
      ↓
Train models
      ↓
Generate validation/test predictions
      ↓
Calculate MAE, RMSE, R²
      ↓
Compare models
      ↓
Select best-performing model

So don't say:

“Random Forest is the best model.”

unless your later results actually demonstrate that.

One Critical Point for Your Viva

Your modelling target is:

log-transformed PCF

but your evaluation metrics are calculated after converting predictions back to:

raw PCF scale

So your complete modelling process is:

Raw PCF
   ↓
log1p transformation
   ↓
Model training
   ↓
Prediction in log space
   ↓
expm1
   ↓
Raw PCF prediction
   ↓
MAE / RMSE / R²

This is worth remembering because an examiner may ask:

“Why are you training on log PCF but evaluating on raw PCF?”

Answer:

“The log transformation helps manage the skewed target distribution during modelling, while converting predictions back to the original PCF scale makes the evaluation metrics directly interpretable in the original target units.”

Presentation wording

“In this cell, I define the regression models used for comparative evaluation. I include Linear Regression as a simple baseline, Ridge Regression as a regularised linear model, SVR with an RBF kernel to capture nonlinear relationships, and Random Forest as a tree-based ensemble. I also include XGBoost and CatBoost as optional boosting models when their libraries are available. I store all models in a dictionary so that the same evaluation pipeline can be applied consistently across algorithms. The hyperparameters shown here are predefined configurations rather than the result of exhaustive hyperparameter optimisation. The final model selection is therefore based on the comparative validation or test performance obtained in the subsequent evaluation stage.”

For your model comparison

Explain why raw-scale metrics can differ
Check whether scaling changes SVR and Ridge

Cell-07-PCA COMPONENT SELECTION — 5-FOLD CV

TS — Mon, 14 Sep 2026 08:04:49 +0000

# ============================================================
# PCA COMPONENT SELECTION — 5-FOLD CV
# ============================================================

PCA_COMPONENTS = [50, 100, 150, 200]

pca_results = []

for n_components in PCA_COMPONENTS:

    fold_results = []

    for fold, (train_idx, valid_idx) in enumerate(
        skf.split(train_df, target_bins),
        start=1
    ):

        # 1. Split development data
        X_fold_train = train_df.iloc[train_idx].copy()
        X_fold_valid = train_df.iloc[valid_idx].copy()

        y_fold_train_log = (
            y_log.iloc[train_idx].to_numpy()
        )

        y_fold_valid_raw = (
            train_df.iloc[valid_idx][TARGET].to_numpy()
        )

        # 2. Structured features
        #    Country → OOF Target Encoding
        X_struct_train, X_struct_valid = (
            build_structured_features(
                X_fold_train,
                y_fold_train_log,
                X_fold_valid
            )
        )

        # 3. SBERT features
        X_sbert_train = X_train_sbert[train_idx]
        X_sbert_valid = X_train_sbert[valid_idx]

        # 4. PCA — fit on training fold only
        X_sbert_train_pca, X_sbert_valid_pca, pca = (
            apply_pca_to_sbert(
                X_sbert_train,
                X_sbert_valid,
                n_components=n_components
            )
        )

        # 5. Hybrid feature fusion
        X_hybrid_train = fuse_features(
            X_sbert_train_pca,
            X_struct_train
        )

        X_hybrid_valid = fuse_features(
            X_sbert_valid_pca,
            X_struct_valid
        )

        # 6. Random Forest
        model = RandomForestRegressor(
            n_estimators=300,
            random_state=42,
            n_jobs=-1
        )

        model.fit(
            X_hybrid_train,
            y_fold_train_log
        )

        # 7. Prediction
        y_pred_log = model.predict(
            X_hybrid_valid
        )

        y_pred_raw = np.maximum(
            np.expm1(y_pred_log),
            0
        )

        # 8. Evaluation
        fold_results.append({
            "PCA": n_components,
            "Fold": fold,
            "MAE": mean_absolute_error(
                y_fold_valid_raw,
                y_pred_raw
            ),
            "RMSE": np.sqrt(
                mean_squared_error(
                    y_fold_valid_raw,
                    y_pred_raw
                )
            ),
            "R2": r2_score(
                y_fold_valid_raw,
                y_pred_raw
            )
        })

    pca_results.extend(fold_results)


# ============================================================
# PCA RESULTS
# ============================================================

pca_results_df = pd.DataFrame(
    pca_results
)

pca_summary = (
    pca_results_df
    .groupby("PCA")
    .agg(
        Mean_MAE=("MAE", "mean"),
        Mean_RMSE=("RMSE", "mean"),
        Mean_R2=("R2", "mean")
    )
    .reset_index()
)

display(pca_summary)

PCA Component Selection — 5-Fold CV

This is an important experimental cell because you are not choosing the PCA dimension arbitrarily. You test several PCA dimensions using five-fold cross-validation and compare their predictive performance.

Your tested values are:

PCA_COMPONENTS = [50, 100, 150, 200]

The result shows that 50 components performed best among the tested options.

The overall pipeline is:

Development data
       ↓
5-fold stratified CV
       ↓
For each PCA size:
       ↓
Structured features
       +
SBERT features
       ↓
PCA
       ↓
Hybrid features
       ↓
Random Forest
       ↓
Prediction
       ↓
MAE / RMSE / R²
       ↓
Average across 5 folds
       ↓
Choose PCA dimension

1. Define PCA candidates

PCA_COMPONENTS = [50, 100, 150, 200]

You are testing four different dimensionalities:

384 original SBERT dimensions
        ↓
50
100
150
200

You are asking:

“Which PCA dimension gives the best downstream PCF prediction performance?”

This is essentially a hyperparameter/model-selection experiment.

You are not assuming that retaining more components automatically gives better results.

2. Create storage for results

pca_results = []

This creates an empty Python list.

You will store the evaluation results from every fold and every PCA configuration.

Since you have:

4 PCA choices × 5 folds

you should get:

20 fold-level result records

Each record contains:

PCA components
Fold number
MAE
RMSE
R²

3. Loop through PCA choices

for n_components in PCA_COMPONENTS:

This starts the outer loop.

It means:

First → PCA = 50
Second → PCA = 100
Third → PCA = 150
Fourth → PCA = 200

For every choice, you perform the complete five-fold evaluation.

Viva question: Why test several PCA values?

“Because the optimal dimensionality is data-dependent. Too few components may discard useful information, while too many components may retain redundancy and increase model complexity. I therefore evaluate several candidate dimensions empirically.”

4. Create fold-level storage

fold_results = []

For each PCA setting, you create a new list to store the results from its five folds.

Conceptually:

PCA = 50
    ↓
Fold 1 result
Fold 2 result
Fold 3 result
Fold 4 result
Fold 5 result

Then the same process is repeated for 100, 150 and 200.

5. Generate the five folds

for fold, (train_idx, valid_idx) in enumerate(
    skf.split(train_df, target_bins),
    start=1
):

This is a very important line.

skf.split() generates the indices for each cross-validation fold.

It uses:

train_df
+
target_bins

The target_bins are the quantile-based bins you created earlier from the continuous target.

The result for each fold is:

train_idx
valid_idx

`train_idx`

Indices of observations used for training in that fold.

`valid_idx`

Indices of observations used for validation in that fold.

Why use `target_bins`?

Because your problem is regression.

The target is continuous, so you created quantile bins earlier:

Continuous y_log
      ↓
5 quantile bins
      ↓
StratifiedKFold

This attempts to keep the distribution of lower and higher target values reasonably balanced across folds.

6. `enumerate(..., start=1)`

enumerate(..., start=1)

This gives each fold a human-readable number:

Fold 1
Fold 2
Fold 3
Fold 4
Fold 5

Without start=1, Python would normally number them:

0, 1, 2, 3, 4

1. Split Development Data

X_fold_train = train_df.iloc[train_idx].copy()
X_fold_valid = train_df.iloc[valid_idx].copy()

These lines use the indices generated by StratifiedKFold.

Training fold

X_fold_train = train_df.iloc[train_idx].copy()

Selects the observations belonging to the training portion of the current fold.

Validation fold

X_fold_valid = train_df.iloc[valid_idx].copy()

Selects the observations belonging to the validation portion.

Remember:

This is not the final test set.

You are still working inside the development data.

Full dataset
   ↓
Development data + independent test
   ↓
Development data
   ↓
5-fold CV
   ├── Fold training
   └── Fold validation

The independent test set remains untouched during this experiment.

7. Get training target

y_fold_train_log = (
    y_log.iloc[train_idx].to_numpy()
)

This selects the log-transformed target values corresponding to the current training fold.

.iloc[train_idx] selects the same observations as X_fold_train.

.to_numpy() converts the pandas Series into a NumPy array.

So the model will train using:

X_fold_train
+
y_fold_train_log

8. Get validation target in raw scale

y_fold_valid_raw = (
    train_df.iloc[valid_idx][TARGET].to_numpy()
)

This retrieves the original, raw PCF target for the validation observations.

This is important because later you convert the model's log predictions back to the raw PCF scale before calculating the metrics.

So:

Training:
y → log scale

Validation evaluation:
actual y → raw scale

This allows your MAE, RMSE and R² to be interpreted in the original PCF units.

2. Structured Features

X_struct_train, X_struct_valid = (
    build_structured_features(
        X_fold_train,
        y_fold_train_log,
        X_fold_valid
    )
)

This calls your previously defined build_structured_features() function.

It creates the structured representation for this specific fold.

The inputs are:

X_fold_train
y_fold_train_log
X_fold_valid

The function performs:

Training fold
    ↓
Winsorization
    ↓
Numeric features

Categorical features
    ↓
One-hot encoding

Country + training target
    ↓
OOF target encoding

and returns:

X_struct_train
X_struct_valid

Critical leakage point

This is one of the strongest parts of your methodology.

For every fold, the structured preprocessing is re-fitted using only the fold-training data.

For example:

Fold 1

Fold training
   ↓
fit winsorization
fit OHE
fit target encoding
   ↓
transform validation

The validation fold's target is not used to learn these transformations.

Viva question: Why rebuild structured features inside every fold?

“Because preprocessing must be learned independently within each training fold. If I fitted preprocessing once using the entire development dataset before cross-validation, information from the validation folds could influence the preprocessing and lead to optimistic performance estimates.”

This is an excellent answer to remember.

3. SBERT Features

X_sbert_train = X_train_sbert[train_idx]
X_sbert_valid = X_train_sbert[valid_idx]

Here you select the SBERT embeddings corresponding to the current fold.

Remember:

X_train_sbert

contains SBERT embeddings for the entire development dataset.

You use the fold indices to select the relevant rows.

Training SBERT

X_sbert_train = X_train_sbert[train_idx]

Gets SBERT representations for the fold-training observations.

Validation SBERT

X_sbert_valid = X_train_sbert[valid_idx]

Gets SBERT representations for the fold-validation observations.

Important subtlety

You are not fitting SBERT here.

SBERT was already used as a pretrained fixed encoder.

You are simply selecting the appropriate embeddings for each fold.

4. PCA — Fit on Training Fold Only

X_sbert_train_pca, X_sbert_valid_pca, pca = (
    apply_pca_to_sbert(
        X_sbert_train,
        X_sbert_valid,
        n_components=n_components
    )
)

This calls your PCA function.

This is another major leakage-control step.

Inside the function:

pca.fit_transform(X_train_sbert)

is applied to the fold-training data.

Then:

pca.transform(X_valid_sbert)

is applied to the validation data.

So:

Fold training SBERT
       ↓
    FIT PCA
       ↓
Learn components
       ↓
Transform training


Fold validation SBERT
       ↓
Use SAME PCA
       ↓
Transform validation

You do not fit PCA on the validation fold.

Why is this important?

Because PCA learns from the feature distribution.

If validation data were included when fitting PCA, the validation data would influence the representation used to evaluate the model.

Strong viva answer

“For every fold, PCA is fitted only on the fold-training SBERT embeddings and then applied to the validation embeddings. This prevents information from the validation fold influencing the dimensionality-reduction step.”

5. Hybrid Feature Fusion

X_hybrid_train = fuse_features(
    X_sbert_train_pca,
    X_struct_train
)

This combines:

Reduced SBERT features
+
Structured features

to create the training hybrid representation.

Similarly:

X_hybrid_valid = fuse_features(
    X_sbert_valid_pca,
    X_struct_valid
)

creates the validation hybrid representation.

The result is:

SBERT semantic features
          +
Structured features
          ↓
Hybrid feature matrix

Why hybrid?

Because:

SBERT
→ semantic information from text

Structured features
→ explicit numerical/categorical information

Hybrid
→ both information sources

This is the core idea of your hybrid modelling approach.

6. Random Forest

model = RandomForestRegressor(
    n_estimators=300,
    random_state=42,
    n_jobs=-1
)

You create a Random Forest regression model.

`RandomForestRegressor`

Because your target is continuous PCF, you use the regression version rather than classification.

`n_estimators=300`

The Random Forest contains 300 decision trees.

Conceptually:

Hybrid features
      ↓
Tree 1
Tree 2
Tree 3
...
Tree 300
      ↓
Average predictions
      ↓
Final prediction

More trees generally make the ensemble more stable, although they increase computation.

`random_state=42`

Makes the random elements of the Random Forest reproducible.

`n_jobs=-1`

Tells scikit-learn to use all available CPU cores for parallel processing.

It affects computational speed, not the underlying modelling objective.

7. Train the Random Forest

model.fit(
    X_hybrid_train,
    y_fold_train_log
)

This trains the Random Forest.

The inputs are:

X_hybrid_train
       +
y_fold_train_log

So the model learns:

Hybrid features
      ↓
Predict log(PCF + 1)

The important point is that the model is trained on the log-transformed target.

8. Generate Predictions

y_pred_log = model.predict(
    X_hybrid_valid
)

The model predicts the PCF values for the validation fold.

But because the model was trained on y_fold_train_log, these predictions are also in log space.

So:

y_pred_log
=
prediction in log scale

9. Convert Predictions Back to Raw PCF

y_pred_raw = np.maximum(
    np.expm1(y_pred_log),
    0
)

This line contains two operations.

`np.expm1()`

np.expm1(x) calculates:

exp(x) - 1

This reverses the transformation produced by:

log(PCF + 1)

So:

log scale
   ↓
expm1
   ↓
original PCF scale

Why `expm1` instead of `np.exp(x) - 1`?

expm1 is designed specifically to calculate exp(x)-1 accurately, particularly for values close to zero.

`np.maximum(..., 0)`

np.maximum(
    np.expm1(y_pred_log),
    0
)

This ensures that predicted PCF values cannot become negative.

Conceptually:

Predicted PCF
     ↓
if negative → 0
if positive → keep it

This is consistent with the physical interpretation of PCF as a non-negative quantity.

Viva question: Why force predictions to zero?

“PCF is physically non-negative, so I constrain negative back-transformed predictions to zero. This is a domain-informed post-processing step.”

Be careful: don't claim that Random Forest itself cannot produce negative predictions.

10. Evaluation

fold_results.append({

You create a dictionary containing the performance results for this fold.

Store PCA size

"PCA": n_components,

Records whether this result came from:

Store fold number

"Fold": fold,

Records:

Fold 1
Fold 2
...
Fold 5

MAE

"MAE": mean_absolute_error(
    y_fold_valid_raw,
    y_pred_raw
),

MAE = Mean Absolute Error.

It measures the average absolute difference between actual and predicted PCF.

Conceptually:

actual PCF
     -
predicted PCF
     ↓
absolute error
     ↓
average
     ↓
MAE

Lower is better.

Your best PCA configuration has the lowest MAE.

RMSE

"RMSE": np.sqrt(
    mean_squared_error(
        y_fold_valid_raw,
        y_pred_raw
    )
),

RMSE = Root Mean Squared Error.

It squares the errors before averaging and taking the square root.

Because of the squaring, large prediction errors have greater influence.

Lower is better.

R²

"R2": r2_score(
    y_fold_valid_raw,
    y_pred_raw
)

R² measures how much variation in the target is explained by the predictions relative to a baseline.

Higher is generally better.

Your result:

PCA 50 → R² = 0.654748

means the model achieved an R² of approximately 0.655 in the five-fold average.

Do not say:

“The model predicts 65.5% of every PCF correctly.”

That is incorrect.

Instead say:

“The average cross-validated R² was approximately 0.655.”

Store Fold Results

pca_results.extend(fold_results)

After completing the five folds for a particular PCA configuration, the fold results are added to the main pca_results list.

Eventually you have:

50 components
→ 5 results

100 components
→ 5 results

150 components
→ 5 results

200 components
→ 5 results

Total:

20 fold-level results

PCA Results

Now the second part of the cell summarises those results.

Convert results to DataFrame

pca_results_df = pd.DataFrame(
    pca_results
)

Converts the list of dictionaries into a pandas DataFrame.

Conceptually:

PCA	Fold	MAE	RMSE	R²
50	1	...	...	...
50	2	...	...	...
50	3	...	...	...
50	4	...	...	...
50	5	...	...	...
100	1	...	...	...

and so on.

Group by PCA

pca_summary = (
    pca_results_df
    .groupby("PCA")

This groups all five fold results according to PCA dimension.

So:

PCA = 50
→ Fold 1–5

PCA = 100
→ Fold 1–5

...

Calculate mean MAE

.agg(
    Mean_MAE=("MAE", "mean"),

Calculates the average MAE across the five folds.

Calculate mean RMSE

Mean_RMSE=("RMSE", "mean"),

Calculates the average RMSE across the five folds.

Calculate mean R²

Mean_R2=("R2", "mean")

Calculates the average R² across the five folds.

Reset index

.reset_index()

Converts PCA from a grouping index back into a normal DataFrame column.

Display summary

display(pca_summary)

Displays the final comparison table.

Your results are:

PCA Components	Mean MAE	Mean RMSE	Mean R²
50	11,095.20	93,823.13	0.6547
100	12,920.35	114,964.07	0.5302
150	12,901.37	114,008.82	0.5352
200	13,032.94	115,111.65	0.5363

What does the result tell you?

The 50-component PCA configuration is clearly the best among the tested choices.

It has:

Lowest MAE
Lowest RMSE
Highest R²

Specifically:

50 components
MAE  = 11,095.20
RMSE = 93,823.13
R²   = 0.6547

Whereas 100, 150 and 200 components perform worse on all three average metrics.

Your conclusion

You can say:

“Among the candidate PCA dimensions tested, 50 components produced the strongest cross-validated performance, achieving the lowest mean MAE and RMSE and the highest mean R². Therefore, I selected 50 PCA components for the subsequent modelling stage.”

Very important examiner question: Why did 50 perform better than 200?

Don't say:

“Because 50 contains more information.”

It actually contains less dimensional information than 200.

A better answer:

“The results suggest that retaining more components did not improve predictive performance in this experiment. The additional components may contain redundant or less useful variation for the downstream Random Forest. However, I would describe this as an empirical result rather than claiming that 50 components is universally optimal.”

That last sentence is important because your experiment only tested:

50, 100, 150, 200

You cannot conclude that 50 is globally optimal.

You can only conclude:

50 was the best among the tested candidates.

Very important examiner question: Is this test set?

Answer:

“No. This is five-fold cross-validation within the development data. The independent test set remains separate and should only be used for the final evaluation after the modelling choices have been made.”

This distinction is extremely important for your viva.

Very important examiner question: Why not choose PCA based on variance explained alone?

Strong answer:

“Explained variance measures how much variation in the original features is retained, but my ultimate objective is PCF prediction. Therefore, I selected the PCA dimension based on downstream cross-validated predictive performance rather than PCA variance alone.”

Most important leakage question

Examiner: “You already created SBERT embeddings for the whole training dataset. Isn't that leakage?”

Your defensible answer:

“SBERT was used as a pretrained fixed encoder and was not fitted or fine-tuned on the PCF dataset. Therefore, generating embeddings for the development observations does not use their PCF targets. For PCA, however, I explicitly fit the transformation separately inside each training fold and only transform the corresponding validation fold.”

This distinction is very important:

SBERT
→ pretrained externally
→ no PCF target used

PCA
→ learned from our dataset
→ must be fitted inside each training fold

One more important point about your experiment

Your code is actually doing two levels of selection:

Outer independent test set
        ↑
   kept untouched

Development data
        ↓
5-fold CV
        ↓
Compare PCA = 50, 100, 150, 200
        ↓
Select 50
        ↓
Final model development
        ↓
Independent test evaluation

This is a strong experimental structure because you are not using the final test set to decide the PCA dimension.

Presentation script

“In this experiment, I selected the PCA dimensionality using five-fold cross-validation on the development data. I tested 50, 100, 150 and 200 components. For each PCA setting, I generated five stratified folds. Within every fold, I independently created the structured features, selected the corresponding SBERT embeddings, and fitted PCA only on the fold-training data. I then fused the reduced SBERT representation with the structured features and trained a Random Forest regressor using the log-transformed PCF target. Predictions were transformed back to the original PCF scale, and I evaluated each fold using MAE, RMSE and R². Finally, I averaged the five fold results for each PCA configuration. The 50-component configuration achieved the lowest mean MAE and RMSE and the highest mean R², so I selected 50 components for the subsequent modelling stage. This selection was made only within the development data, keeping the independent test set untouched.”

For your PCA selection

Check fold-to-fold stability
Compare variance explained

CELL-06-FEATURE COLUMN DEFINITIONS-&-PCA FOR SBERT FEATURES

TS — Mon, 14 Sep 2026 08:02:07 +0000

# ============================================================
# FEATURE COLUMN DEFINITIONS
# ============================================================

YEAR_COL = "Year of reporting"

WEIGHT_COL = "Product weight (kg)"

COUNTRY_COL = "Country (where company is incorporated)"

OTHER_CATEGORICAL_FEATURES = [
    "Company's GICS Industry",
    "Protocol used for PCF",
    "*Stage-level CO2e available"
]

Feature Column Definitions

This cell defines the column names that will be used later in the structured feature engineering pipeline.

The main purpose is to avoid repeatedly writing long dataset column names throughout the notebook and to make the code easier to read and maintain.

1. Year column

YEAR_COL = "Year of reporting"

This creates a variable called YEAR_COL.

It points to the dataset column:

Year of reporting

This is treated as a numeric structured feature.

For example:

The model can potentially learn whether reporting year is associated with differences in PCF values.

Viva question: Why include reporting year?

“Reporting year can capture temporal differences in products, reporting practices, technologies, or emission factors that may be associated with PCF values.”

Be careful not to claim that year causes PCF changes. It is simply a predictive feature.

2. Product weight column

WEIGHT_COL = "Product weight (kg)"

This assigns the dataset's product-weight column to WEIGHT_COL.

This is another numeric feature.

For example:

0.5 kg
2 kg
10 kg
100 kg

You later apply winsorization to this feature.

The reason is that product weight can have extreme values, and extreme values can disproportionately influence some models.

3. Country column

COUNTRY_COL = "Country (where company is incorporated)"

This defines the country feature.

It is treated differently from ordinary categorical variables because you later apply target encoding to it.

The transformation is approximately:

Country
   ↓
Country-specific mean PCF
   ↓
Numerical feature

For example, conceptually:

Germany → 12.4
USA     → 18.7
Japan   → 10.9

These values are learned from the training data.

Because target encoding uses the target variable, you specifically introduced out-of-fold target encoding to reduce target leakage during development.

Viva question: Why target-encode country instead of one-hot encoding it?

A good answer is:

“Country can have many categories, and one-hot encoding would create a separate binary feature for each country. Target encoding provides a compact numerical representation of the relationship between country and the target. However, because it uses target values, I use out-of-fold encoding for the training data to reduce target leakage.”

4. Other categorical features

```python id="8s5x5x"
OTHER_CATEGORICAL_FEATURES = [
"Company's GICS Industry",
"Protocol used for PCF",
"*Stage-level CO2e available"
]




This creates a list containing three categorical variables.

#### `Company's GICS Industry`

Represents the company's industry classification.

Different industries can have substantially different production processes and emission profiles.

#### `Protocol used for PCF`

Represents the protocol or methodology used to calculate the PCF.

This can be relevant because different calculation/reporting approaches may affect the resulting PCF data.

#### `*Stage-level CO2e available`

Indicates whether stage-level carbon-emission information is available.

This is a categorical/indicator-type feature representing the availability of more detailed PCF information.

---

### Why are these three features grouped together?

Because you process them using **One-Hot Encoding** later.

Your structured-feature pipeline effectively separates the variables into:



```text
Structured Features
│
├── Numeric
│   ├── Year
│   └── Product Weight
│
├── One-Hot Categorical
│   ├── GICS Industry
│   ├── PCF Protocol
│   └── Stage-level CO2e availability
│
└── Target Encoded
    └── Country

This is an important design decision.

Why not process all categorical features in the same way?

Because different representations can be appropriate for different characteristics.

For the three ordinary categorical features:

Industry
Protocol
Stage-level availability
        ↓
One-Hot Encoding

For country:

Country
   ↓
Target Encoding

The target encoding gives a compact numerical representation, but because it uses the target, it requires additional leakage control.

Why create variables instead of directly using column names?

Instead of repeatedly writing:

X_train["Product weight (kg)"]

you can write:

X_train[WEIGHT_COL]

This improves:

Readability
Maintainability
Consistency
Reduced risk of spelling mistakes

If the original dataset column name changes, you can update the definition in one place.

Strong viva answer

If the examiner asks “What is the purpose of this cell?”, say:

“This cell defines the feature groups used in my structured feature engineering pipeline. I identify year and product weight as numeric features, country as a target-encoded feature, and industry, PCF protocol, and stage-level CO2e availability as categorical features that will be one-hot encoded. Using variables for the column names also makes the later preprocessing functions more readable and maintainable.”

One important connection to your previous cells

This cell provides the column definitions used by your build_structured_features() function:

YEAR_COL
WEIGHT_COL
     ↓
Numeric processing

OTHER_CATEGORICAL_FEATURES
     ↓
One-Hot Encoding

COUNTRY_COL
     ↓
OOF Target Encoding

So this cell itself does not transform any data. It simply tells the later feature-engineering code which columns belong to which processing strategy.

For your feature pipeline

Explain country encoding leakage
Check feature availability before splitting

# ============================================================
# PCA FOR SBERT FEATURES
# ============================================================

from sklearn.decomposition import PCA


def apply_pca_to_sbert(
    X_train_sbert,
    X_valid_sbert,
    n_components
):

    pca = PCA(
        n_components=n_components,
        random_state=42
    )

    X_train_pca = pca.fit_transform(
        X_train_sbert
    )

    X_valid_pca = pca.transform(
        X_valid_sbert
    )

    return (
        X_train_pca,
        X_valid_pca,
        pca
    )

PCA FOR SBERT FEATURES

This function applies Principal Component Analysis (PCA) to the SBERT embeddings.

The main reason is dimensionality reduction: SBERT produces a relatively high-dimensional vector, and PCA can compress that representation while retaining as much variance/information as possible.

Your flow is:

Combined Text
      ↓
SBERT
      ↓
High-dimensional embeddings
      ↓
PCA
      ↓
Reduced-dimensional embeddings
      ↓
ML model

1. Import PCA

```python id="pca001"
from sklearn.decomposition import PCA




This imports the `PCA` class from scikit-learn.

PCA stands for **Principal Component Analysis**.

It transforms the original features into a new set of features called **principal components**.

The first principal component captures the largest amount of variance, the second captures the next largest amount, and so on.

---

### 2. Define the PCA function



```python id="pca002"
def apply_pca_to_sbert(
    X_train_sbert,
    X_valid_sbert,
    n_components
):

You define a reusable function called:

apply_pca_to_sbert()

It takes three inputs.

`X_train_sbert`

The SBERT embeddings for the training/development portion.

For example:

800 observations × 384 SBERT features

`X_valid_sbert`

The SBERT embeddings for the validation portion.

For example:

200 observations × 384 features

`n_components`

The number of dimensions you want after PCA.

For example:

384 → 50

This means PCA reduces the SBERT representation from 384 dimensions to 50 dimensions.

3. Create the PCA object

```python id="pca003"
pca = PCA(
n_components=n_components,
random_state=42
)




This creates the PCA transformation.

#### `n_components=n_components`

This tells PCA how many principal components to retain.

If:



```python
n_components = 50

then:

Original SBERT
384 dimensions

        ↓ PCA

Reduced SBERT
50 dimensions

The important point is that PCA does not simply select the first 50 original SBERT features.

Instead, it creates new combinations of the original features.

4. Why reduce the SBERT dimensions?

This is a likely viva question.

SBERT gives you a dense embedding with many dimensions.

High-dimensional features can:

increase computational cost
increase memory requirements
introduce redundant information
make downstream ML models more expensive

PCA provides a way to create a smaller representation while retaining major patterns in the original embeddings.

Strong viva answer

“I applied PCA to reduce the dimensionality of the SBERT embeddings. SBERT produces a high-dimensional representation, and PCA allows me to create a more compact representation while retaining the principal sources of variance. This can reduce computational cost and potentially remove redundant information for the downstream ML models.”

5. Fit PCA on training data

```python id="pca004"
X_train_pca = pca.fit_transform(
X_train_sbert
)




This line performs **two operations**:



```text
fit
+
transform

`fit()`

PCA learns the transformation from the training data.

It determines the principal directions/components based on the training embeddings.

`transform()`

The training embeddings are then projected onto those learned principal components.

So:

X_train_sbert
      ↓
PCA learns components
      ↓
Training data transformed
      ↓
X_train_pca

Very important leakage point

PCA must be fitted only on training data.

You correctly do:

pca.fit_transform(X_train_sbert)

rather than fitting PCA on both training and validation data.

Why?

Because PCA learns information about the distribution of the data.

If you did:

pca.fit_transform(
    np.vstack([X_train_sbert, X_valid_sbert])
)

then the validation distribution would influence the learned PCA components.

That would introduce information from the validation set into the preprocessing stage.

Viva question: Why fit PCA only on training data?

“Because PCA learns its components from the data distribution. If I fitted it using both training and validation data, information from the validation set would influence the transformation. Therefore, I fit PCA only on the training data and apply the learned transformation to validation data.”

6. Transform validation data

```python id="pca005"
X_valid_pca = pca.transform(
X_valid_sbert
)




This applies the **already learned PCA transformation** to the validation embeddings.

Notice the difference:



```text
Training:
pca.fit_transform()

Validation:
pca.transform()

This distinction is extremely important.

`fit_transform`

Used when the transformation needs to learn parameters from training data and then transform it.

`transform`

Used when the transformation has already been learned and you want to apply the same transformation to new data.

So your pipeline is:

Training SBERT
      ↓
fit PCA
      ↓
learn principal components
      ↓
transform training


Validation SBERT
      ↓
use existing PCA
      ↓
transform validation

You do not calculate new PCA components for validation.

7. Return the results

```python id="pca006"
return (
X_train_pca,
X_valid_pca,
pca
)




The function returns three things.

#### `X_train_pca`

The reduced-dimensional training SBERT features.

#### `X_valid_pca`

The reduced-dimensional validation SBERT features.

#### `pca`

The fitted PCA object itself.

Returning the PCA object is useful because it contains the learned transformation and can be reused later.

For example, the same fitted PCA can be applied to another unseen dataset:



```text
New SBERT embeddings
        ↓
fitted PCA
        ↓
same reduced feature space

The most important concept: `fit_transform` vs `transform`

Remember this for your viva:

TRAINING
pca.fit_transform(X_train_sbert)
        ↓
Learn + transform


VALIDATION / TEST
pca.transform(X_valid_sbert)
        ↓
Transform only

If the examiner asks “Why don't you use fit_transform on validation?”, say:

“Because that would learn a separate transformation from the validation data. I want validation to remain unseen during preprocessing, so I reuse the PCA transformation learned from the training data.”

Example

Suppose SBERT produces:

Training:   800 × 384
Validation: 200 × 384

and you select:

n_components = 50

After PCA:

Training PCA:   800 × 50
Validation PCA: 200 × 50

So you reduce:

384 → 50

while keeping the observations unchanged.

The number of rows does not change.

Only the number of feature columns changes.

How this connects to your hybrid features

This is particularly important in your project because later you combine the text representation with structured features.

Your pipeline can be understood as:

Combined_Text
      ↓
    SBERT
      ↓
384-dimensional embeddings
      ↓
     PCA
      ↓
Reduced SBERT features
      +
Structured features
      ↓
Hybrid feature representation
      ↓
ML model

So PCA is applied to the SBERT part, not to the entire hybrid feature matrix at this stage.

Strong presentation wording

“Here, I define a reusable PCA function for reducing the dimensionality of the SBERT embeddings. The function receives training embeddings, validation embeddings, and the desired number of components. I initialise PCA with that number of components and fit it only on the training SBERT features. The training embeddings are therefore used to learn the principal components and are transformed into the reduced representation. I then apply the same fitted PCA transformation to the validation embeddings without refitting it. Finally, I return both reduced feature matrices and the fitted PCA object. This ensures dimensionality reduction while avoiding information leakage from the validation data.”

Examiner trap questions

“Does PCA select the most important original SBERT features?”

Answer:

“No. PCA creates new orthogonal components as linear combinations of the original features. It does not simply select a subset of the original dimensions.”

“Does PCA guarantee better prediction?”

Answer:

“No. PCA is a dimensionality-reduction technique, not a prediction-improvement guarantee. It may reduce computational cost and redundancy, but the effect on predictive performance needs to be evaluated empirically.”

“Why not remove PCA completely?”

“That is a valid alternative. PCA is a modelling choice that can be compared experimentally. If the original SBERT dimensions provide better validation performance, retaining them may be preferable.”

“Does PCA use the target variable?”

Answer:

“No. Standard PCA is unsupervised. It learns the principal directions from the feature matrix and does not use the PCF target.”

“Why is that useful for leakage control?”

“Because PCA itself does not use the target, but it still learns from the feature distribution. Therefore, I must still fit it only on the training data and not on validation or test data.”

For your SBERT PCA step

Explain how to choose components
Check PCA assumptions

cell-05

TS — Mon, 14 Sep 2026 07:52:52 +0000

# 5. Five-Fold Development Evaluation
from sklearn.model_selection import StratifiedKFold

target_bins = pd.qcut(
    y_log,
    q=5,
    labels=False,
    duplicates="drop"
)

skf = StratifiedKFold(
    n_splits=5,
    shuffle=True,
    random_state=RANDOM_STATE
)

print("Stratified folds:", skf.n_splits)

5. Five-Fold Development Evaluation

This cell prepares five-fold cross-validation for the development/training data.

The important point is that your target is a continuous PCF value, so you cannot directly use ordinary stratification on the raw target. Instead, you first divide the continuous target into bins and then use those bins for stratification.

The overall idea is:

Continuous PCF target
        ↓
Log-transformed target (y_log)
        ↓
Divide into 5 target ranges
        ↓
Create stratified 5 folds
        ↓
Each fold has a similar target distribution

1. Import StratifiedKFold

from sklearn.model_selection import StratifiedKFold

This imports StratifiedKFold from scikit-learn.

StratifiedKFold creates multiple train/validation folds while trying to preserve the distribution of a categorical class label across the folds.

The problem is that PCF is a continuous regression target, not a class label.

Therefore, you create target bins first.

2. Create target bins

target_bins = pd.qcut(
    y_log,
    q=5,
    labels=False,
    duplicates="drop"
)

This is the most important part of the cell.

`y_log`

This is your log-transformed PCF target.

You are using the transformed target rather than the original PCF values.

A log transformation is commonly used when a continuous target is highly right-skewed, because it can reduce the influence of extremely large values and make the distribution more manageable for modelling.

3. `pd.qcut()`

pd.qcut(...)

qcut divides the values into quantile-based bins.

Unlike a normal fixed-width binning method, it tries to put approximately the same number of observations into each bin.

For example, conceptually:

y_log values
     ↓
 ┌───────────────┐
 │ Lowest 20%    │ → Bin 0
 ├───────────────┤
 │ 20–40%        │ → Bin 1
 ├───────────────┤
 │ 40–60%        │ → Bin 2
 ├───────────────┤
 │ 60–80%        │ → Bin 3
 ├───────────────┤
 │ Highest 20%   │ → Bin 4
 └───────────────┘

So although the original target is continuous, you temporarily create approximately five groups representing different levels of PCF.

4. `q=5`

q=5

This asks qcut to create five quantile groups.

Approximately:

Bin 0 → lowest 20%
Bin 1 → next 20%
Bin 2 → middle 20%
Bin 3 → next 20%
Bin 4 → highest 20%

This does not mean that you are converting the regression problem into a classification problem.

That distinction is very important.

Viva question: Are you turning PCF prediction into classification?

Answer:

“No. The actual task remains regression because the original target is continuous. I only use quantile bins to create stratification labels for cross-validation, so that the folds have a more balanced distribution of target values.”

5. `labels=False`

labels=False

This tells qcut to return integer labels instead of interval descriptions.

So instead of something like:

(1.2, 2.8]
(2.8, 4.1]

you get:

These integer values can then be used by StratifiedKFold.

6. `duplicates="drop"`

duplicates="drop"

This handles situations where quantile boundaries are not unique.

For example, if many observations have exactly the same target value, qcut may not be able to create exactly five distinct intervals.

duplicates="drop" tells pandas to remove duplicate bin boundaries instead of throwing an error.

Viva question: Why did you use `duplicates="drop"`?

Answer:

“Because repeated target values can produce identical quantile boundaries. duplicates='drop' makes the binning more robust by allowing duplicate boundaries to be removed instead of causing an error.”

7. Create StratifiedKFold

skf = StratifiedKFold(
    n_splits=5,
    shuffle=True,
    random_state=RANDOM_STATE
)

This creates the five-fold cross-validation strategy.

`n_splits=5`

n_splits=5

The development data is divided into five folds.

Conceptually:

Fold 1 → validation
Fold 2 → training
Fold 3 → training
Fold 4 → training
Fold 5 → training

Then it rotates:

Fold 1 → training
Fold 2 → validation
Fold 3 → training
Fold 4 → training
Fold 5 → training

and so on.

Eventually, every observation gets used as validation data exactly once.

8. `shuffle=True`

shuffle=True

This randomly shuffles the observations before creating the folds.

This helps avoid having the folds determined by the original ordering of the dataset.

For example, if your dataset happened to be ordered by country, year, or PCF magnitude, not shuffling could produce unrepresentative folds.

Because you are using stratification, the shuffled data is then distributed while attempting to preserve the target-bin proportions.

9. `random_state`

random_state=RANDOM_STATE

This makes the random shuffling reproducible.

If you run the notebook again with the same data and the same RANDOM_STATE, you should obtain the same fold assignment.

Viva question: Why is reproducibility important?

Answer:

“It ensures that the same data partitioning can be reproduced when I rerun the experiment, making model comparison and evaluation more consistent.”

10. Print number of folds

print("Stratified folds:", skf.n_splits)

skf.n_splits returns the number of folds.

So this should print:

Stratified folds: 5

This is simply a verification step.

Why use StratifiedKFold for a regression problem?

This is very likely to be asked in your viva.

Normally:

Classification
     ↓
StratifiedKFold
     ↓
Preserve class proportions

But your problem is:

PCF prediction
     ↓
Regression
     ↓
Continuous target

So you cannot directly stratify the raw continuous target.

Your solution is:

Continuous y_log
      ↓
Quantile binning
      ↓
Target bins
      ↓
StratifiedKFold

This allows you to approximately preserve the distribution of the target across the five development folds.

Important distinction from ordinary KFold

With ordinary:

KFold(n_splits=5)

the folds are created without explicitly trying to preserve the target distribution.

With your approach:

StratifiedKFold(n_splits=5)

you provide target_bins as the stratification labels, so each fold attempts to contain a similar proportion of low, medium, and high target observations.

One subtle point the examiner may challenge

q=5 and n_splits=5 are two different things.

q=5
↓
creates 5 target bins

n_splits=5
↓
creates 5 cross-validation folds

They happen to both be five in your code, but they serve different purposes.

You could theoretically have:

q = 5
n_splits = 10

and that would still be conceptually valid if the sample size and bin frequencies support it.

One thing to remember for the next cell

This cell only creates the cross-validation strategy.

You have not actually performed cross-validation yet.

The actual fold generation will happen later when you do something like:

for train_idx, valid_idx in skf.split(X, target_bins):

At that point:

X
+
target_bins
      ↓
skf.split()
      ↓
training indices + validation indices

Strong presentation explanation

“Because PCF is a continuous regression target, I cannot directly apply stratification to the raw target values. Therefore, I first use pd.qcut on the log-transformed target to create five quantile-based target bins. These bins act as temporary stratification labels. I then create a five-fold StratifiedKFold with shuffling and a fixed random state. This allows the development folds to have approximately similar distributions of low and high target values, providing a more balanced and reproducible cross-validation evaluation. The bins are only used for splitting; the actual prediction task remains regression.”

For your five-fold evaluation

Check bin sizes before splitting
Explain fold-level leakage risks

cell-04-SBERT TEXT REPRESENTATION

TS — Mon, 14 Sep 2026 06:51:52 +0000

# ============================================================
# SBERT TEXT REPRESENTATION
# ============================================================

from sentence_transformers import SentenceTransformer

SBERT_MODEL_NAME = "all-MiniLM-L6-v2"

sbert_model = SentenceTransformer(SBERT_MODEL_NAME)

train_texts = train_df["Combined_Text"].fillna("").astype(str).tolist()
test_texts = test_df["Combined_Text"].fillna("").astype(str).tolist()

X_train_sbert = sbert_model.encode(
    train_texts,
    show_progress_bar=True,
    convert_to_numpy=True
)

X_test_sbert = sbert_model.encode(
    test_texts,
    show_progress_bar=True,
    convert_to_numpy=True
)

print("SBERT dimension:", X_train_sbert.shape[1])
print("Training embeddings:", X_train_sbert.shape)
print("Test embeddings:", X_test_sbert.shape)


# ============================================================
# HYBRID FEATURE FUSION
# ============================================================

def fuse_features(sbert_features, structured_features):
    return np.hstack([
        sbert_features,
        structured_features
    ])

SBERT TEXT REPRESENTATION

This cell converts the combined text feature into numerical semantic embeddings using SBERT. These embeddings will later be combined with the structured features to create the hybrid feature representation.

1. Import SBERT

from sentence_transformers import SentenceTransformer

Imports the SentenceTransformer class from the sentence_transformers library.
SBERT stands for Sentence-BERT.
It converts text into dense numerical vectors called embeddings.
These embeddings capture the semantic meaning and relationships between words/text, rather than simply representing individual words.

Viva question: Why did you use SBERT?

Answer:

“I used SBERT to convert the combined textual information into dense semantic embeddings. This allows the ML models to capture relationships between product names, companies, and countries that may not be captured by simple categorical or numerical encoding.”

2. Select the SBERT model

SBERT_MODEL_NAME = "all-MiniLM-L6-v2"

Stores the name of the pretrained SBERT model in a variable.
"all-MiniLM-L6-v2" is a lightweight sentence-transformer model.
It provides a good balance between semantic representation, computational cost, and embedding size.

The important point is that you are using a pretrained model, rather than training SBERT from scratch on your PCF dataset.

Viva question: Why this particular model?

Answer:

“I selected all-MiniLM-L6-v2 because it is a relatively lightweight pretrained sentence-transformer that provides useful semantic embeddings while keeping computational requirements manageable.”

If examiner asks: Why not a larger model?

“A larger transformer could potentially provide richer representations, but it would increase computational cost. For this project, I prioritised a practical balance between semantic representation and efficiency.”

3. Load the pretrained model

sbert_model = SentenceTransformer(SBERT_MODEL_NAME)

This creates the SBERT model object.

The process is:

"all-MiniLM-L6-v2"
        ↓
SentenceTransformer()
        ↓
Pretrained SBERT model loaded

The model is now ready to convert text into embeddings.

Important viva point

You are not fitting SBERT on the PCF dataset here.

You are using an already pretrained language model to generate representations.

That is different from:

model.fit(...)

used by conventional ML models.

4. Get training text

train_texts = train_df["Combined_Text"].fillna("").astype(str).tolist()

Let's break this line into four operations.

`train_df["Combined_Text"]`

Selects the Combined_Text column from the training dataset.

Earlier, you created something like:

Product: Laptop | Company: Dell | Country: USA

So each row contains a combined textual representation.

`.fillna("")`

.fillna("")

Replaces missing text values with an empty string.

For example:

NaN

becomes:

""

This prevents missing values from causing problems during text encoding.

`.astype(str)`

.astype(str)

Converts every value into a string.

SBERT expects textual input, so this ensures the input has the appropriate type.

`.tolist()`

.tolist()

Converts the pandas Series into a Python list.

For example:

Pandas Series
     ↓
Python list
     ↓
["Product: Laptop | Company: Dell | Country: USA",
 "Product: Phone | Company: Apple | Country: USA",
 ...]

This list is then passed to SBERT.

5. Get test text

test_texts = test_df["Combined_Text"].fillna("").astype(str).tolist()

This performs exactly the same preparation for the test dataset.

The important difference is:

train_df → train_texts
test_df  → test_texts

The two datasets remain separate.

Viva question: Why do you encode train and test separately?

Answer:

“I encode the training and test observations separately so that each observation receives its embedding without mixing the datasets. The same pretrained SBERT model is used for both, ensuring that the representations are generated in a consistent way.”

6. Generate training embeddings

X_train_sbert = sbert_model.encode(
    train_texts,
    show_progress_bar=True,
    convert_to_numpy=True
)

This is the main SBERT operation.

The model takes the training text and converts every text record into a numerical vector.

Conceptually:

Combined_Text
      ↓
     SBERT
      ↓
Dense numerical embedding

For example, conceptually:

"Product: Laptop | Company: Dell | Country: USA"
                     ↓
        [0.12, -0.34, 0.56, ..., 0.08]

The actual vector contains many numerical dimensions.

7. `show_progress_bar=True`

show_progress_bar=True

Displays the encoding progress while SBERT processes the observations.

For example:

Encoding: 100% |████████████| ...

This does not affect the model's predictions.

It is simply useful for monitoring a potentially time-consuming operation.

Viva question: Does this parameter affect accuracy?

Answer:

“No. It only controls whether a progress bar is displayed during encoding.”

8. `convert_to_numpy=True`

convert_to_numpy=True

Requests the embeddings as a NumPy array.

This is important because later you use:

np.hstack(...)

to combine the SBERT features with structured features.

So the pipeline becomes:

SBERT
 ↓
NumPy embedding matrix
 ↓
np.hstack()
 ↓
Hybrid features

9. Generate test embeddings

X_test_sbert = sbert_model.encode(
    test_texts,
    show_progress_bar=True,
    convert_to_numpy=True
)

The same pretrained SBERT model converts the test texts into embeddings.

Notice that you do not train or refit SBERT using the test data.

You simply apply the already loaded model.

This is important for maintaining a clean evaluation process.

Viva question: Is this data leakage?

Answer:

“No, because the SBERT model is pretrained independently and is not fitted on the PCF test data. The test text is only passed through the pretrained encoder to generate representations.”

A more cautious answer if the examiner is strict:

“The encoder itself is pretrained externally rather than learned from this dataset. I use it only as a fixed representation model and do not fine-tune it using the PCF test set.”

10. Print SBERT dimension

print("SBERT dimension:", X_train_sbert.shape[1])

X_train_sbert.shape contains:

(number of observations, number of embedding dimensions)

For example:

(800, 384)

Then:

.shape[1]

selects the second dimension:

So this prints the number of features generated by SBERT.

For all-MiniLM-L6-v2, the embedding dimension is typically 384.

11. Print training embedding shape

print("Training embeddings:", X_train_sbert.shape)

This shows:

(number of training observations, embedding dimensions)

For example:

Training embeddings: (800, 384)

This allows you to verify that the number of embeddings corresponds to the number of training observations.

12. Print test embedding shape

print("Test embeddings:", X_test_sbert.shape)

Similarly, this verifies the dimensions of the test embeddings.

For example:

Test embeddings: (200, 384)

The important requirement is:

number of training embeddings = number of training rows

number of test embeddings = number of test rows

HYBRID FEATURE FUSION

Now you combine the semantic text representation with the structured representation.

def fuse_features(sbert_features, structured_features):

This defines a function called fuse_features.

It accepts two inputs:

sbert_features
structured_features

`sbert_features`

These are the SBERT embeddings.

For example:

384 dimensions

`structured_features`

These are the features you previously created from:

Year
Product weight
One-hot encoded categorical variables
Country target encoding

So you have two different types of information:

SBERT
→ semantic information

Structured features
→ explicit numerical/categorical information

13. Horizontally concatenate the features

return np.hstack([
    sbert_features,
    structured_features
])

np.hstack() means horizontal stacking.

Suppose:

SBERT = 384 features
Structured = 10 features

Then:

384 + 10 = 394 features

The result becomes:

[ SBERT features | Structured features ]

For every observation, the two feature vectors are placed side by side.

Conceptually:

SBERT
[0.12, -0.34, 0.56, ..., 0.08]
              +
Structured
[2023, 1.5, 0, 1, 0, ...]
              ↓
Hybrid
[0.12, -0.34, 0.56, ..., 0.08, 2023, 1.5, 0, 1, 0, ...]

Why use hybrid features?

This is one of the most important viva questions.

Your reasoning is:

Text alone
   ↓
captures semantic information

Structured features alone
   ↓
captures explicit numerical/categorical information

Both together
   ↓
hybrid representation
   ↓
potentially richer information for ML

Strong viva answer

“I used hybrid feature fusion because the two representations capture different types of information. SBERT captures semantic relationships within the combined text, while the structured features explicitly represent numerical and categorical information such as reporting year, product weight, industry, protocol, and country. By horizontally concatenating them, the downstream ML models can use both sources of information simultaneously.”

Very important distinction

Do not say:

“SBERT understands the exact carbon footprint.”

That would overclaim.

Instead say:

“SBERT provides a semantic representation of the textual information, which may contain information useful for predicting PCF.”

The PCF prediction itself is performed by your downstream ML model.

Overall pipeline for this section

Product
Company
Country
   ↓
Combined_Text
   ↓
Pretrained SBERT
   ↓
384-dimensional semantic embeddings
   ↓
              ┌────────────────────┐
              │ Structured features│
              │ Year               │
              │ Weight             │
              │ Industry           │
              │ Protocol           │
              │ Country encoding   │
              └────────────────────┘
                         ↓
SBERT embeddings ──→ Feature Fusion ←── Structured features
                         ↓
                  Hybrid Features
                         ↓
                    ML Model
                         ↓
                  PCF Prediction

Most likely examiner questions

Why SBERT instead of TF-IDF?

“TF-IDF mainly represents lexical importance based on word frequency, whereas SBERT produces dense contextual semantic embeddings. Since product and company descriptions can contain meaningful relationships beyond exact word matching, SBERT provides a richer semantic representation.”

Why combine SBERT with structured features?

“Because SBERT and structured features contain complementary information. SBERT captures semantic text information, while structured features preserve explicit numerical and categorical information.”

Why use the same SBERT model for train and test?

“To maintain a consistent feature space. The same pretrained encoder transforms both datasets, but the test data is not used to fit or fine-tune the encoder.”

What does `np.hstack()` do?

“It horizontally concatenates the two feature matrices, placing the SBERT and structured features side by side for each observation.”

What is the biggest limitation of this approach?

“SBERT was pretrained for general language representation, not specifically for PCF prediction. Therefore, its embeddings may not capture all domain-specific carbon-footprint relationships. The hybrid approach helps by adding explicit structured information, but it does not guarantee that every relevant PCF relationship is represented.”

Presentation wording

“In this stage, I convert the combined textual representation into semantic embeddings using the pretrained all-MiniLM-L6-v2 SBERT model. First, I prepare the training and test text by handling missing values and converting them into lists. I then pass both datasets through the same pretrained SBERT encoder and obtain numerical embeddings. I verify the dimensions of these embeddings to ensure that the transformation was successful. Finally, I define a feature-fusion function that horizontally combines the SBERT embeddings with the structured features. This creates a hybrid representation containing both semantic textual information and explicit numerical and categorical information, which is then used by the downstream machine-learning models.”

For your SBERT pipeline

Explain how to verify row alignment
Explain feature scaling after fusion

PCA	Fold	MAE	RMSE	R²
50	1	...	...	...
50	2	...	...	...
50	3	...	...	...
50	4	...	...	...
50	5	...	...	...
100	1	...	...	...

PCA	Fold	MAE	RMSE	R²
50	1	...	...	...
50	2	...	...	...
50	3	...	...	...
50	4	...	...	...
50	5	...	...	...
100	1	...	...	...

PCA	Fold	MAE	RMSE	R²
50	1	...	...	...
50	2	...	...	...
50	3	...	...	...
50	4	...	...	...
50	5	...	...	...
100	1	...	...	...