cell-05

# 5. Five-Fold Development Evaluation
from sklearn.model_selection import StratifiedKFold

target_bins = pd.qcut(
    y_log,
    q=5,
    labels=False,
    duplicates="drop"
)

skf = StratifiedKFold(
    n_splits=5,
    shuffle=True,
    random_state=RANDOM_STATE
)

print("Stratified folds:", skf.n_splits)

5. Five-Fold Development Evaluation

This cell prepares five-fold cross-validation for the development/training data.

The important point is that your target is a continuous PCF value, so you cannot directly use ordinary stratification on the raw target. Instead, you first divide the continuous target into bins and then use those bins for stratification.

The overall idea is:

Continuous PCF target
        ↓
Log-transformed target (y_log)
        ↓
Divide into 5 target ranges
        ↓
Create stratified 5 folds
        ↓
Each fold has a similar target distribution

1. Import StratifiedKFold

from sklearn.model_selection import StratifiedKFold

This imports StratifiedKFold from scikit-learn.

StratifiedKFold creates multiple train/validation folds while trying to preserve the distribution of a categorical class label across the folds.

The problem is that PCF is a continuous regression target, not a class label.

Therefore, you create target bins first.

---

2. Create target bins

target_bins = pd.qcut(
    y_log,
    q=5,
    labels=False,
    duplicates="drop"
)

This is the most important part of the cell.

y_log

This is your log-transformed PCF target.

You are using the transformed target rather than the original PCF values.

A log transformation is commonly used when a continuous target is highly right-skewed, because it can reduce the influence of extremely large values and make the distribution more manageable for modelling.

---

3. pd.qcut()

pd.qcut(...)

qcut divides the values into quantile-based bins.

Unlike a normal fixed-width binning method, it tries to put approximately the same number of observations into each bin.

For example, conceptually:

y_log values
     ↓
 ┌───────────────┐
 │ Lowest 20%    │ → Bin 0
 ├───────────────┤
 │ 20–40%        │ → Bin 1
 ├───────────────┤
 │ 40–60%        │ → Bin 2
 ├───────────────┤
 │ 60–80%        │ → Bin 3
 ├───────────────┤
 │ Highest 20%   │ → Bin 4
 └───────────────┘

So although the original target is continuous, you temporarily create approximately five groups representing different levels of PCF.

---

4. q=5

q=5

This asks qcut to create five quantile groups.

Approximately:

Bin 0 → lowest 20%
Bin 1 → next 20%
Bin 2 → middle 20%
Bin 3 → next 20%
Bin 4 → highest 20%

This does not mean that you are converting the regression problem into a classification problem.

That distinction is very important.

Viva question: Are you turning PCF prediction into classification?

Answer:

“No. The actual task remains regression because the original target is continuous. I only use quantile bins to create stratification labels for cross-validation, so that the folds have a more balanced distribution of target values.”

---

5. labels=False

labels=False

This tells qcut to return integer labels instead of interval descriptions.

So instead of something like:

(1.2, 2.8]
(2.8, 4.1]

you get:

0
1
2
3
4

These integer values can then be used by StratifiedKFold.

---

6. duplicates="drop"

duplicates="drop"

This handles situations where quantile boundaries are not unique.

For example, if many observations have exactly the same target value, qcut may not be able to create exactly five distinct intervals.

duplicates="drop" tells pandas to remove duplicate bin boundaries instead of throwing an error.

Viva question: Why did you use duplicates="drop"?

Answer:

“Because repeated target values can produce identical quantile boundaries. duplicates='drop' makes the binning more robust by allowing duplicate boundaries to be removed instead of causing an error.”

---

7. Create StratifiedKFold

skf = StratifiedKFold(
    n_splits=5,
    shuffle=True,
    random_state=RANDOM_STATE
)

This creates the five-fold cross-validation strategy.

n_splits=5

n_splits=5

The development data is divided into five folds.

Conceptually:

Fold 1 → validation
Fold 2 → training
Fold 3 → training
Fold 4 → training
Fold 5 → training

Then it rotates:

Fold 1 → training
Fold 2 → validation
Fold 3 → training
Fold 4 → training
Fold 5 → training

and so on.

Eventually, every observation gets used as validation data exactly once.

---

8. shuffle=True

shuffle=True

This randomly shuffles the observations before creating the folds.

This helps avoid having the folds determined by the original ordering of the dataset.

For example, if your dataset happened to be ordered by country, year, or PCF magnitude, not shuffling could produce unrepresentative folds.

Because you are using stratification, the shuffled data is then distributed while attempting to preserve the target-bin proportions.

---

9. random_state

random_state=RANDOM_STATE

This makes the random shuffling reproducible.

If you run the notebook again with the same data and the same RANDOM_STATE, you should obtain the same fold assignment.

Viva question: Why is reproducibility important?

Answer:

“It ensures that the same data partitioning can be reproduced when I rerun the experiment, making model comparison and evaluation more consistent.”

---

10. Print number of folds

print("Stratified folds:", skf.n_splits)

skf.n_splits returns the number of folds.

So this should print:

Stratified folds: 5

This is simply a verification step.

---

Why use StratifiedKFold for a regression problem?

This is very likely to be asked in your viva.

Normally:

Classification
     ↓
StratifiedKFold
     ↓
Preserve class proportions

But your problem is:

PCF prediction
     ↓
Regression
     ↓
Continuous target

So you cannot directly stratify the raw continuous target.

Your solution is:

Continuous y_log
      ↓
Quantile binning
      ↓
Target bins
      ↓
StratifiedKFold

This allows you to approximately preserve the distribution of the target across the five development folds.

Important distinction from ordinary KFold

With ordinary:

KFold(n_splits=5)

the folds are created without explicitly trying to preserve the target distribution.

With your approach:

StratifiedKFold(n_splits=5)

you provide target_bins as the stratification labels, so each fold attempts to contain a similar proportion of low, medium, and high target observations.

One subtle point the examiner may challenge

q=5 and n_splits=5 are two different things.

q=5
↓
creates 5 target bins

n_splits=5
↓
creates 5 cross-validation folds

They happen to both be five in your code, but they serve different purposes.

You could theoretically have:

q = 5
n_splits = 10

and that would still be conceptually valid if the sample size and bin frequencies support it.

One thing to remember for the next cell

This cell only creates the cross-validation strategy.

You have not actually performed cross-validation yet.

The actual fold generation will happen later when you do something like:

for train_idx, valid_idx in skf.split(X, target_bins):

At that point:

X
+
target_bins
      ↓
skf.split()
      ↓
training indices + validation indices

Strong presentation explanation

“Because PCF is a continuous regression target, I cannot directly apply stratification to the raw target values. Therefore, I first use pd.qcut on the log-transformed target to create five quantile-based target bins. These bins act as temporary stratification labels. I then create a five-fold StratifiedKFold with shuffling and a fixed random state. This allows the development folds to have approximately similar distributions of low and high target values, providing a more balanced and reproducible cross-validation evaluation. The bins are only used for splitting; the actual prediction task remains regression.”

For your five-fold evaluation

• Check bin sizes before splitting
• Explain fold-level leakage risks