Cell-07-PCA COMPONENT SELECTION — 5-FOLD CV

# ============================================================
# PCA COMPONENT SELECTION — 5-FOLD CV
# ============================================================

PCA_COMPONENTS = [50, 100, 150, 200]

pca_results = []

for n_components in PCA_COMPONENTS:

    fold_results = []

    for fold, (train_idx, valid_idx) in enumerate(
        skf.split(train_df, target_bins),
        start=1
    ):

        # 1. Split development data
        X_fold_train = train_df.iloc[train_idx].copy()
        X_fold_valid = train_df.iloc[valid_idx].copy()

        y_fold_train_log = (
            y_log.iloc[train_idx].to_numpy()
        )

        y_fold_valid_raw = (
            train_df.iloc[valid_idx][TARGET].to_numpy()
        )

        # 2. Structured features
        #    Country → OOF Target Encoding
        X_struct_train, X_struct_valid = (
            build_structured_features(
                X_fold_train,
                y_fold_train_log,
                X_fold_valid
            )
        )

        # 3. SBERT features
        X_sbert_train = X_train_sbert[train_idx]
        X_sbert_valid = X_train_sbert[valid_idx]

        # 4. PCA — fit on training fold only
        X_sbert_train_pca, X_sbert_valid_pca, pca = (
            apply_pca_to_sbert(
                X_sbert_train,
                X_sbert_valid,
                n_components=n_components
            )
        )

        # 5. Hybrid feature fusion
        X_hybrid_train = fuse_features(
            X_sbert_train_pca,
            X_struct_train
        )

        X_hybrid_valid = fuse_features(
            X_sbert_valid_pca,
            X_struct_valid
        )

        # 6. Random Forest
        model = RandomForestRegressor(
            n_estimators=300,
            random_state=42,
            n_jobs=-1
        )

        model.fit(
            X_hybrid_train,
            y_fold_train_log
        )

        # 7. Prediction
        y_pred_log = model.predict(
            X_hybrid_valid
        )

        y_pred_raw = np.maximum(
            np.expm1(y_pred_log),
            0
        )

        # 8. Evaluation
        fold_results.append({
            "PCA": n_components,
            "Fold": fold,
            "MAE": mean_absolute_error(
                y_fold_valid_raw,
                y_pred_raw
            ),
            "RMSE": np.sqrt(
                mean_squared_error(
                    y_fold_valid_raw,
                    y_pred_raw
                )
            ),
            "R2": r2_score(
                y_fold_valid_raw,
                y_pred_raw
            )
        })

    pca_results.extend(fold_results)


# ============================================================
# PCA RESULTS
# ============================================================

pca_results_df = pd.DataFrame(
    pca_results
)

pca_summary = (
    pca_results_df
    .groupby("PCA")
    .agg(
        Mean_MAE=("MAE", "mean"),
        Mean_RMSE=("RMSE", "mean"),
        Mean_R2=("R2", "mean")
    )
    .reset_index()
)

display(pca_summary)

PCA Component Selection — 5-Fold CV

This is an important experimental cell because you are not choosing the PCA dimension arbitrarily. You test several PCA dimensions using five-fold cross-validation and compare their predictive performance.

Your tested values are:

PCA_COMPONENTS = [50, 100, 150, 200]

The result shows that 50 components performed best among the tested options.

The overall pipeline is:

Development data
       ↓
5-fold stratified CV
       ↓
For each PCA size:
       ↓
Structured features
       +
SBERT features
       ↓
PCA
       ↓
Hybrid features
       ↓
Random Forest
       ↓
Prediction
       ↓
MAE / RMSE / R²
       ↓
Average across 5 folds
       ↓
Choose PCA dimension

1. Define PCA candidates

PCA_COMPONENTS = [50, 100, 150, 200]

You are testing four different dimensionalities:

384 original SBERT dimensions
        ↓
50
100
150
200

You are asking:

“Which PCA dimension gives the best downstream PCF prediction performance?”

This is essentially a hyperparameter/model-selection experiment.

You are not assuming that retaining more components automatically gives better results.

---

2. Create storage for results

pca_results = []

This creates an empty Python list.

You will store the evaluation results from every fold and every PCA configuration.

Since you have:

4 PCA choices × 5 folds

you should get:

20 fold-level result records

Each record contains:

• PCA components
• Fold number
• MAE
• RMSE
• R²

---

3. Loop through PCA choices

for n_components in PCA_COMPONENTS:

This starts the outer loop.

It means:

First → PCA = 50
Second → PCA = 100
Third → PCA = 150
Fourth → PCA = 200

For every choice, you perform the complete five-fold evaluation.

Viva question: Why test several PCA values?

“Because the optimal dimensionality is data-dependent. Too few components may discard useful information, while too many components may retain redundancy and increase model complexity. I therefore evaluate several candidate dimensions empirically.”

---

4. Create fold-level storage

fold_results = []

For each PCA setting, you create a new list to store the results from its five folds.

Conceptually:

PCA = 50
    ↓
Fold 1 result
Fold 2 result
Fold 3 result
Fold 4 result
Fold 5 result

Then the same process is repeated for 100, 150 and 200.

---

5. Generate the five folds

for fold, (train_idx, valid_idx) in enumerate(
    skf.split(train_df, target_bins),
    start=1
):

This is a very important line.

skf.split() generates the indices for each cross-validation fold.

It uses:

train_df
+
target_bins

The target_bins are the quantile-based bins you created earlier from the continuous target.

The result for each fold is:

train_idx
valid_idx

train_idx

Indices of observations used for training in that fold.

valid_idx

Indices of observations used for validation in that fold.

---

Why use target_bins?

Because your problem is regression.

The target is continuous, so you created quantile bins earlier:

Continuous y_log
      ↓
5 quantile bins
      ↓
StratifiedKFold

This attempts to keep the distribution of lower and higher target values reasonably balanced across folds.

---

6. enumerate(..., start=1)

enumerate(..., start=1)

This gives each fold a human-readable number:

Fold 1
Fold 2
Fold 3
Fold 4
Fold 5

Without start=1, Python would normally number them:

0, 1, 2, 3, 4

---

1. Split Development Data

X_fold_train = train_df.iloc[train_idx].copy()
X_fold_valid = train_df.iloc[valid_idx].copy()

These lines use the indices generated by StratifiedKFold.

Training fold

X_fold_train = train_df.iloc[train_idx].copy()

Selects the observations belonging to the training portion of the current fold.

Validation fold

X_fold_valid = train_df.iloc[valid_idx].copy()

Selects the observations belonging to the validation portion.

Remember:

This is not the final test set.

You are still working inside the development data.

Full dataset
   ↓
Development data + independent test
   ↓
Development data
   ↓
5-fold CV
   ├── Fold training
   └── Fold validation

The independent test set remains untouched during this experiment.

---

7. Get training target

y_fold_train_log = (
    y_log.iloc[train_idx].to_numpy()
)

This selects the log-transformed target values corresponding to the current training fold.

.iloc[train_idx] selects the same observations as X_fold_train.

.to_numpy() converts the pandas Series into a NumPy array.

So the model will train using:

X_fold_train
+
y_fold_train_log

---

8. Get validation target in raw scale

y_fold_valid_raw = (
    train_df.iloc[valid_idx][TARGET].to_numpy()
)

This retrieves the original, raw PCF target for the validation observations.

This is important because later you convert the model's log predictions back to the raw PCF scale before calculating the metrics.

So:

Training:
y → log scale

Validation evaluation:
actual y → raw scale

This allows your MAE, RMSE and R² to be interpreted in the original PCF units.

---

2. Structured Features

X_struct_train, X_struct_valid = (
    build_structured_features(
        X_fold_train,
        y_fold_train_log,
        X_fold_valid
    )
)

This calls your previously defined build_structured_features() function.

It creates the structured representation for this specific fold.

The inputs are:

X_fold_train
y_fold_train_log
X_fold_valid

The function performs:

Training fold
    ↓
Winsorization
    ↓
Numeric features

Categorical features
    ↓
One-hot encoding

Country + training target
    ↓
OOF target encoding

and returns:

X_struct_train
X_struct_valid

Critical leakage point

This is one of the strongest parts of your methodology.

For every fold, the structured preprocessing is re-fitted using only the fold-training data.

For example:

Fold 1

Fold training
   ↓
fit winsorization
fit OHE
fit target encoding
   ↓
transform validation

The validation fold's target is not used to learn these transformations.

Viva question: Why rebuild structured features inside every fold?

“Because preprocessing must be learned independently within each training fold. If I fitted preprocessing once using the entire development dataset before cross-validation, information from the validation folds could influence the preprocessing and lead to optimistic performance estimates.”

This is an excellent answer to remember.

---

3. SBERT Features

X_sbert_train = X_train_sbert[train_idx]
X_sbert_valid = X_train_sbert[valid_idx]

Here you select the SBERT embeddings corresponding to the current fold.

Remember:

X_train_sbert

contains SBERT embeddings for the entire development dataset.

You use the fold indices to select the relevant rows.

Training SBERT

X_sbert_train = X_train_sbert[train_idx]

Gets SBERT representations for the fold-training observations.

Validation SBERT

X_sbert_valid = X_train_sbert[valid_idx]

Gets SBERT representations for the fold-validation observations.

Important subtlety

You are not fitting SBERT here.

SBERT was already used as a pretrained fixed encoder.

You are simply selecting the appropriate embeddings for each fold.

---

4. PCA — Fit on Training Fold Only

X_sbert_train_pca, X_sbert_valid_pca, pca = (
    apply_pca_to_sbert(
        X_sbert_train,
        X_sbert_valid,
        n_components=n_components
    )
)

This calls your PCA function.

This is another major leakage-control step.

Inside the function:

pca.fit_transform(X_train_sbert)

is applied to the fold-training data.

Then:

pca.transform(X_valid_sbert)

is applied to the validation data.

So:

Fold training SBERT
       ↓
    FIT PCA
       ↓
Learn components
       ↓
Transform training


Fold validation SBERT
       ↓
Use SAME PCA
       ↓
Transform validation

You do not fit PCA on the validation fold.

Why is this important?

Because PCA learns from the feature distribution.

If validation data were included when fitting PCA, the validation data would influence the representation used to evaluate the model.

Strong viva answer

“For every fold, PCA is fitted only on the fold-training SBERT embeddings and then applied to the validation embeddings. This prevents information from the validation fold influencing the dimensionality-reduction step.”

---

5. Hybrid Feature Fusion

X_hybrid_train = fuse_features(
    X_sbert_train_pca,
    X_struct_train
)

This combines:

Reduced SBERT features
+
Structured features

to create the training hybrid representation.

Similarly:

X_hybrid_valid = fuse_features(
    X_sbert_valid_pca,
    X_struct_valid
)

creates the validation hybrid representation.

The result is:

SBERT semantic features
          +
Structured features
          ↓
Hybrid feature matrix

Why hybrid?

Because:

SBERT
→ semantic information from text

Structured features
→ explicit numerical/categorical information

Hybrid
→ both information sources

This is the core idea of your hybrid modelling approach.

---

6. Random Forest

model = RandomForestRegressor(
    n_estimators=300,
    random_state=42,
    n_jobs=-1
)

You create a Random Forest regression model.

RandomForestRegressor

Because your target is continuous PCF, you use the regression version rather than classification.

n_estimators=300

The Random Forest contains 300 decision trees.

Conceptually:

Hybrid features
      ↓
Tree 1
Tree 2
Tree 3
...
Tree 300
      ↓
Average predictions
      ↓
Final prediction

More trees generally make the ensemble more stable, although they increase computation.

random_state=42

Makes the random elements of the Random Forest reproducible.

n_jobs=-1

Tells scikit-learn to use all available CPU cores for parallel processing.

It affects computational speed, not the underlying modelling objective.

---

7. Train the Random Forest

model.fit(
    X_hybrid_train,
    y_fold_train_log
)

This trains the Random Forest.

The inputs are:

X_hybrid_train
       +
y_fold_train_log

So the model learns:

Hybrid features
      ↓
Predict log(PCF + 1)

The important point is that the model is trained on the log-transformed target.

---

8. Generate Predictions

y_pred_log = model.predict(
    X_hybrid_valid
)

The model predicts the PCF values for the validation fold.

But because the model was trained on y_fold_train_log, these predictions are also in log space.

So:

y_pred_log
=
prediction in log scale

---

9. Convert Predictions Back to Raw PCF

y_pred_raw = np.maximum(
    np.expm1(y_pred_log),
    0
)

This line contains two operations.

np.expm1()

np.expm1(x) calculates:

exp(x) - 1

This reverses the transformation produced by:

log(PCF + 1)

So:

log scale
   ↓
expm1
   ↓
original PCF scale

Why expm1 instead of np.exp(x) - 1?

expm1 is designed specifically to calculate exp(x)-1 accurately, particularly for values close to zero.

---

np.maximum(..., 0)

np.maximum(
    np.expm1(y_pred_log),
    0
)

This ensures that predicted PCF values cannot become negative.

Conceptually:

Predicted PCF
     ↓
if negative → 0
if positive → keep it

This is consistent with the physical interpretation of PCF as a non-negative quantity.

Viva question: Why force predictions to zero?

“PCF is physically non-negative, so I constrain negative back-transformed predictions to zero. This is a domain-informed post-processing step.”

Be careful: don't claim that Random Forest itself cannot produce negative predictions.

---

10. Evaluation

fold_results.append({

You create a dictionary containing the performance results for this fold.

---

Store PCA size

"PCA": n_components,

Records whether this result came from:

50
100
150
200

---

Store fold number

"Fold": fold,

Records:

Fold 1
Fold 2
...
Fold 5

---

MAE

"MAE": mean_absolute_error(
    y_fold_valid_raw,
    y_pred_raw
),

MAE = Mean Absolute Error.

It measures the average absolute difference between actual and predicted PCF.

Conceptually:

actual PCF
     -
predicted PCF
     ↓
absolute error
     ↓
average
     ↓
MAE

Lower is better.

Your best PCA configuration has the lowest MAE.

---

RMSE

"RMSE": np.sqrt(
    mean_squared_error(
        y_fold_valid_raw,
        y_pred_raw
    )
),

RMSE = Root Mean Squared Error.

It squares the errors before averaging and taking the square root.

Because of the squaring, large prediction errors have greater influence.

Lower is better.

---

R²

"R2": r2_score(
    y_fold_valid_raw,
    y_pred_raw
)

R² measures how much variation in the target is explained by the predictions relative to a baseline.

Higher is generally better.

Your result:

PCA 50 → R² = 0.654748

means the model achieved an R² of approximately 0.655 in the five-fold average.

Do not say:

“The model predicts 65.5% of every PCF correctly.”

That is incorrect.

Instead say:

“The average cross-validated R² was approximately 0.655.”

---

Store Fold Results

pca_results.extend(fold_results)

After completing the five folds for a particular PCA configuration, the fold results are added to the main pca_results list.

Eventually you have:

50 components
→ 5 results

100 components
→ 5 results

150 components
→ 5 results

200 components
→ 5 results

Total:

20 fold-level results

---

PCA Results

Now the second part of the cell summarises those results.

Convert results to DataFrame

pca_results_df = pd.DataFrame(
    pca_results
)

Converts the list of dictionaries into a pandas DataFrame.

Conceptually:

PCA	Fold	MAE	RMSE	R²
50	1	...	...	...
50	2	...	...	...
50	3	...	...	...
50	4	...	...	...
50	5	...	...	...
100	1	...	...	...

and so on.

---

Group by PCA

pca_summary = (
    pca_results_df
    .groupby("PCA")

This groups all five fold results according to PCA dimension.

So:

PCA = 50
→ Fold 1–5

PCA = 100
→ Fold 1–5

...

---

Calculate mean MAE

.agg(
    Mean_MAE=("MAE", "mean"),

Calculates the average MAE across the five folds.

---

Calculate mean RMSE

Mean_RMSE=("RMSE", "mean"),

Calculates the average RMSE across the five folds.

---

Calculate mean R²

Mean_R2=("R2", "mean")

Calculates the average R² across the five folds.

---

Reset index

.reset_index()

Converts PCA from a grouping index back into a normal DataFrame column.

---

Display summary

display(pca_summary)

Displays the final comparison table.

Your results are:

PCA Components	Mean MAE	Mean RMSE	Mean R²
50	11,095.20	93,823.13	0.6547
100	12,920.35	114,964.07	0.5302
150	12,901.37	114,008.82	0.5352
200	13,032.94	115,111.65	0.5363

What does the result tell you?

The 50-component PCA configuration is clearly the best among the tested choices.

It has:

Lowest MAE
Lowest RMSE
Highest R²

Specifically:

50 components
MAE  = 11,095.20
RMSE = 93,823.13
R²   = 0.6547

Whereas 100, 150 and 200 components perform worse on all three average metrics.

Your conclusion

You can say:

“Among the candidate PCA dimensions tested, 50 components produced the strongest cross-validated performance, achieving the lowest mean MAE and RMSE and the highest mean R². Therefore, I selected 50 PCA components for the subsequent modelling stage.”

Very important examiner question: Why did 50 perform better than 200?

Don't say:

“Because 50 contains more information.”

It actually contains less dimensional information than 200.

A better answer:

“The results suggest that retaining more components did not improve predictive performance in this experiment. The additional components may contain redundant or less useful variation for the downstream Random Forest. However, I would describe this as an empirical result rather than claiming that 50 components is universally optimal.”

That last sentence is important because your experiment only tested:

50, 100, 150, 200

You cannot conclude that 50 is globally optimal.

You can only conclude:

50 was the best among the tested candidates.

Very important examiner question: Is this test set?

Answer:

“No. This is five-fold cross-validation within the development data. The independent test set remains separate and should only be used for the final evaluation after the modelling choices have been made.”

This distinction is extremely important for your viva.

Very important examiner question: Why not choose PCA based on variance explained alone?

Strong answer:

“Explained variance measures how much variation in the original features is retained, but my ultimate objective is PCF prediction. Therefore, I selected the PCA dimension based on downstream cross-validated predictive performance rather than PCA variance alone.”

Most important leakage question

Examiner: “You already created SBERT embeddings for the whole training dataset. Isn't that leakage?”

Your defensible answer:

“SBERT was used as a pretrained fixed encoder and was not fitted or fine-tuned on the PCF dataset. Therefore, generating embeddings for the development observations does not use their PCF targets. For PCA, however, I explicitly fit the transformation separately inside each training fold and only transform the corresponding validation fold.”

This distinction is very important:

SBERT
→ pretrained externally
→ no PCF target used

PCA
→ learned from our dataset
→ must be fitted inside each training fold

One more important point about your experiment

Your code is actually doing two levels of selection:

Outer independent test set
        ↑
   kept untouched

Development data
        ↓
5-fold CV
        ↓
Compare PCA = 50, 100, 150, 200
        ↓
Select 50
        ↓
Final model development
        ↓
Independent test evaluation

This is a strong experimental structure because you are not using the final test set to decide the PCA dimension.

Presentation script

“In this experiment, I selected the PCA dimensionality using five-fold cross-validation on the development data. I tested 50, 100, 150 and 200 components. For each PCA setting, I generated five stratified folds. Within every fold, I independently created the structured features, selected the corresponding SBERT embeddings, and fitted PCA only on the fold-training data. I then fused the reduced SBERT representation with the structured features and trained a Random Forest regressor using the log-transformed PCF target. Predictions were transformed back to the original PCF scale, and I evaluated each fold using MAE, RMSE and R². Finally, I averaged the five fold results for each PCA configuration. The 50-component configuration achieved the lowest mean MAE and RMSE and the highest mean R², so I selected 50 components for the subsequent modelling stage. This selection was made only within the development data, keeping the independent test set untouched.”

For your PCA selection

• Check fold-to-fold stability
• Compare variance explained