Posted on Sep 14

cell-09-MODEL EVALUATION

# ============================================================
# MODEL EVALUATION
# ============================================================

from sklearn.preprocessing import StandardScaler


def evaluate_model(
    model,
    X_train,
    y_train_log,
    X_valid,
    y_valid_raw,
    scale_features=False
):
    """
    Train on log-transformed PCF and evaluate
    predictions in the original PCF scale.
    """

    # Feature scaling
    if scale_features:

        scaler = StandardScaler()

        X_train_model = scaler.fit_transform(X_train)
        X_valid_model = scaler.transform(X_valid)

    else:

        X_train_model = X_train
        X_valid_model = X_valid


    # Train model
    model.fit(
        X_train_model,
        y_train_log
    )


    # Predict log(PCF)
    y_pred_log = model.predict(
        X_valid_model
    )


    # Convert predictions back to PCF scale
    y_pred_raw = np.expm1(
        y_pred_log
    )

    # Prevent negative PCF predictions
    y_pred_raw = np.maximum(
        y_pred_raw,
        0
    )


    # Evaluation metrics
    mae = mean_absolute_error(
        y_valid_raw,
        y_pred_raw
    )

    rmse = np.sqrt(
        mean_squared_error(
            y_valid_raw,
            y_pred_raw
        )
    )

    r2 = r2_score(
        y_valid_raw,
        y_pred_raw
    )


    return mae, rmse, r2

Model Evaluation

This function creates a reusable evaluation pipeline for all of your regression models.

The main purpose is to make sure every model is evaluated in the same way:

Training features
      ↓
Optional scaling
      ↓
Train using log-transformed PCF
      ↓
Predict log-transformed PCF
      ↓
Convert prediction back to raw PCF
      ↓
Calculate MAE, RMSE, R²

This is an important cell because it standardises your model comparison.

1. Import StandardScaler

```python id="scaler01"
from sklearn.preprocessing import StandardScaler




This imports `StandardScaler`.

StandardScaler standardises numerical feature values approximately to:



```text
mean = 0
standard deviation = 1

The transformation is learned from the training data.

It is particularly relevant for models such as:

Linear Regression
Ridge Regression
SVR

Tree-based models such as Random Forest generally do not require feature scaling.

2. Define the evaluation function

```python id="eval02"
def evaluate_model(
model,
X_train,
y_train_log,
X_valid,
y_valid_raw,
scale_features=False
):




You define a reusable function called:



```text
evaluate_model()

It accepts six inputs.

`model`

The regression model you want to evaluate.

For example:

Linear Regression
Ridge
SVR
Random Forest
XGBoost
CatBoost

`X_train`

Training feature matrix.

This may contain your hybrid features:

PCA-reduced SBERT
+
structured features

`y_train_log`

The log-transformed PCF target used for model training.

`X_valid`

Validation feature matrix.

`y_valid_raw`

The actual PCF values in their original/raw scale.

`scale_features=False`

This controls whether feature scaling is applied.

By default:

scale_features = False

So no scaling happens unless you explicitly request it.

3. Function documentation

```python id="eval03"
"""
Train on log-transformed PCF and evaluate
predictions in the original PCF scale.
"""




This is a **docstring**.

It documents what the function does.

The key information is:



```text
Training → log-transformed PCF

Evaluation → original PCF scale

That distinction is important in your project.

Feature Scaling

```python id="eval04"
if scale_features:




This checks whether the function was called with:



```python
scale_features=True

If yes, scaling is performed.

If false, the original feature matrices are used.

4. Create scaler

```python id="eval05"
scaler = StandardScaler()




Creates a StandardScaler object.

The scaler will learn:

* feature means
* feature standard deviations

from the training data.

---

### 5. Fit and transform training features



```python id="eval06"
X_train_model = scaler.fit_transform(X_train)

This does two operations:

fit
+
transform

`fit`

Learns the mean and standard deviation from training data only.

`transform`

Uses those learned values to standardise the training features.

Conceptually:

X_train
   ↓
learn mean/std
   ↓
standardise
   ↓
X_train_model

6. Transform validation features

```python id="eval07"
X_valid_model = scaler.transform(X_valid)




This applies the **same training-derived scaling parameters** to the validation data.

Notice:



```text
Training:
fit_transform()

Validation:
transform()

This is very important for avoiding leakage.

Why not use `fit_transform()` on validation?

Because then the validation data would be used to calculate its own mean and standard deviation.

That means information from validation would influence the preprocessing.

Your approach is correct:

Training
 ↓
fit scaler
 ↓
learn parameters

Validation
 ↓
use same parameters

Viva answer

“I fit the scaler only on the training data and then apply the learned transformation to validation data. This prevents information from the validation set influencing the preprocessing.”

If Scaling Is Not Requested

```python id="eval08"
else:

X_train_model = X_train
X_valid_model = X_valid




If:



```python
scale_features=False

the original feature matrices are used without scaling.

So:

scale_features=True
→ StandardScaler

scale_features=False
→ original features

This allows the same evaluation function to work with models that do and do not benefit from scaling.

Train the Model

```python id="eval09"
model.fit(
X_train_model,
y_train_log
)




This trains the selected regression model.

The model receives:



```text
X_train_model
+
y_train_log

The target is the log-transformed PCF.

So the model learns to predict approximately:

log(PCF + 1)

rather than the raw PCF.

Predict in Log Space

```python id="eval10"
y_pred_log = model.predict(
X_valid_model
)




The trained model predicts the validation observations.

Because the model was trained using `y_train_log`, these predictions are also in log space.

Therefore:



```text
y_pred_log

is not yet the final PCF value in the original units.

Convert Predictions Back to PCF Scale

```python id="eval11"
y_pred_raw = np.expm1(
y_pred_log
)




This reverses the earlier log transformation.

If your target transformation was:



```text
log(PCF + 1)

then:

expm1(prediction)

returns the prediction to approximately the original PCF scale.

Conceptually:

Raw PCF
   ↓
log1p()
   ↓
Log PCF
   ↓
ML model
   ↓
Predicted Log PCF
   ↓
expm1()
   ↓
Predicted Raw PCF

Why do this before calculating metrics?

Because you want your MAE and RMSE to be expressed in the original PCF units, making them easier to interpret.

Prevent Negative PCF Predictions

```python id="eval12"
y_pred_raw = np.maximum(
y_pred_raw,
0
)




This prevents negative predicted PCF values.

For every prediction:



```text
if prediction < 0
    → 0

if prediction ≥ 0
    → keep it

This is a domain-informed constraint because PCF is non-negative.

Viva question: Can your model produce negative predictions?

Depending on the regression algorithm and transformation, yes, a model may produce a log-space value that back-transforms to a value that needs domain checking. This line ensures the final reported prediction is not negative.

A safe answer:

“I apply a non-negativity constraint after inverse transformation because negative PCF values are not physically meaningful.”

Evaluation Metrics

Now the function calculates three metrics.

```python id="eval13"
mae = mean_absolute_error(
y_valid_raw,
y_pred_raw
)




### MAE

Mean Absolute Error measures the average absolute difference between:



```text
actual PCF
vs.
predicted PCF

Lower is better.

For example, an MAE of 100 means the average absolute prediction error is 100 PCF units, assuming the target's unit is kg CO₂e.

RMSE

```python id="eval14"
rmse = np.sqrt(
mean_squared_error(
y_valid_raw,
y_pred_raw
)
)




First:



```python
mean_squared_error(...)

calculates Mean Squared Error.

Then:

np.sqrt(...)

takes the square root.

This produces RMSE.

RMSE penalises larger errors more heavily than MAE because the errors are squared before averaging.

Lower is better.

R²

```python id="eval15"
r2 = r2_score(
y_valid_raw,
y_pred_raw
)




Calculates the coefficient of determination, R².

It evaluates how well the predictions explain variation in the target relative to a baseline.

Generally:



```text
Higher R² → better

But R² should always be interpreted together with MAE and RMSE.

Return the Results

```python id="eval16"
return mae, rmse, r2




The function returns three values:



```text
MAE
RMSE
R²

This makes it easy to store and compare the performance of different models.

For example, conceptually:

Model              MAE       RMSE       R²
------------------------------------------------
Linear Regression  ...       ...        ...
Ridge              ...       ...        ...
SVR                ...       ...        ...
Random Forest      ...       ...        ...
XGBoost            ...       ...        ...
CatBoost           ...       ...        ...

The Most Important Leakage Principle in This Function

Your preprocessing follows:

TRAIN
 ↓
fit scaler
 ↓
transform train


VALIDATION
 ↓
transform using training scaler

Not:

TRAIN + VALIDATION
 ↓
fit scaler

This same principle applies to your other learned transformations:

Winsorization
→ fit on training only

One-Hot Encoder
→ fit on training only

Target Encoder
→ training/OOF strategy

PCA
→ fit on training only

Scaler
→ fit on training only

That is a very strong viva point because it shows you understand preprocessing leakage rather than simply applying preprocessing mechanically.

Why Scale Some Models but Not Others?

This is another likely examiner question.

Linear Regression

Scaling can make coefficients numerically more comparable, although ordinary linear regression predictions are invariant to simple feature rescaling under standard conditions.

Ridge

Scaling is important because Ridge applies regularisation to coefficients. Different feature scales can otherwise cause the penalty to affect features unevenly.

SVR

Scaling is particularly important for SVR because distance calculations and the RBF kernel are sensitive to feature scale.

Random Forest

Usually does not require scaling because tree splits are based on feature thresholds rather than distances or coefficient magnitudes.

Strong viva answer

“I made feature scaling optional because its importance depends on the algorithm. Models such as Ridge and especially RBF-SVR are sensitive to feature scale, whereas tree-based models such as Random Forest generally do not require standardisation. The function therefore allows scaling to be enabled when appropriate.”

Very Important Question: Why Train on Log Target but Evaluate on Raw Target?

Answer this exactly:

“The log transformation helps manage the skewness and extreme values of the PCF target during model training. After prediction, I apply the inverse transformation so that the predictions return to the original PCF scale. I then calculate MAE, RMSE and R² against the raw validation target, making the reported performance interpretable in the original PCF units.”

Very Important Question: Why Use Three Metrics?

You can say:

“I use MAE, RMSE and R² because they provide complementary information. MAE represents the average absolute error, RMSE gives greater weight to large errors, and R² indicates how well the predictions explain variation in the target. Using multiple metrics reduces the risk of judging a model from a single perspective.”

One Important Technical Point

Your function itself does not perform cross-validation.

It evaluates:

one training set
+
one validation set

Cross-validation happens outside this function when you repeatedly call it for different folds.

So:

evaluate_model()
→ one train/validation evaluation

5-fold CV loop
→ calls evaluate_model() five times

This distinction may come up in your viva.

Presentation wording

“This function standardises the evaluation process across my regression models. It optionally applies feature scaling, fitting the scaler only on the training data and then transforming the validation data. The selected model is trained using the log-transformed PCF target. Predictions are generated in log space and then converted back to the original PCF scale using the inverse transformation. I apply a non-negativity constraint because negative PCF values are not physically meaningful. Finally, I calculate MAE, RMSE and R² on the original PCF scale and return these metrics for model comparison.”

Viva checklist for this cell

fit_transform() → training data
transform() → validation data
Scaling → model-dependent
Training target → log scale
Prediction → log scale
expm1() → back to raw PCF
maximum(..., 0) → no negative PCF
MAE → average absolute error
RMSE → emphasises large errors
R² → explained variation
Metrics → calculated on raw PCF scale
Function → evaluates one train/validation split; CV is handled outside

For this evaluation function

Compare scaled and unscaled models
Check raw-scale metric bias