Posted on Sep 14

CELL-06-FEATURE COLUMN DEFINITIONS-&-PCA FOR SBERT FEATURES

# ============================================================
# FEATURE COLUMN DEFINITIONS
# ============================================================

YEAR_COL = "Year of reporting"

WEIGHT_COL = "Product weight (kg)"

COUNTRY_COL = "Country (where company is incorporated)"

OTHER_CATEGORICAL_FEATURES = [
    "Company's GICS Industry",
    "Protocol used for PCF",
    "*Stage-level CO2e available"
]

Feature Column Definitions

This cell defines the column names that will be used later in the structured feature engineering pipeline.

The main purpose is to avoid repeatedly writing long dataset column names throughout the notebook and to make the code easier to read and maintain.

1. Year column

YEAR_COL = "Year of reporting"

This creates a variable called YEAR_COL.

It points to the dataset column:

Year of reporting

This is treated as a numeric structured feature.

For example:

The model can potentially learn whether reporting year is associated with differences in PCF values.

Viva question: Why include reporting year?

“Reporting year can capture temporal differences in products, reporting practices, technologies, or emission factors that may be associated with PCF values.”

Be careful not to claim that year causes PCF changes. It is simply a predictive feature.

2. Product weight column

WEIGHT_COL = "Product weight (kg)"

This assigns the dataset's product-weight column to WEIGHT_COL.

This is another numeric feature.

For example:

0.5 kg
2 kg
10 kg
100 kg

You later apply winsorization to this feature.

The reason is that product weight can have extreme values, and extreme values can disproportionately influence some models.

3. Country column

COUNTRY_COL = "Country (where company is incorporated)"

This defines the country feature.

It is treated differently from ordinary categorical variables because you later apply target encoding to it.

The transformation is approximately:

Country
   ↓
Country-specific mean PCF
   ↓
Numerical feature

For example, conceptually:

Germany → 12.4
USA     → 18.7
Japan   → 10.9

These values are learned from the training data.

Because target encoding uses the target variable, you specifically introduced out-of-fold target encoding to reduce target leakage during development.

Viva question: Why target-encode country instead of one-hot encoding it?

A good answer is:

“Country can have many categories, and one-hot encoding would create a separate binary feature for each country. Target encoding provides a compact numerical representation of the relationship between country and the target. However, because it uses target values, I use out-of-fold encoding for the training data to reduce target leakage.”

4. Other categorical features

```python id="8s5x5x"
OTHER_CATEGORICAL_FEATURES = [
"Company's GICS Industry",
"Protocol used for PCF",
"*Stage-level CO2e available"
]




This creates a list containing three categorical variables.

#### `Company's GICS Industry`

Represents the company's industry classification.

Different industries can have substantially different production processes and emission profiles.

#### `Protocol used for PCF`

Represents the protocol or methodology used to calculate the PCF.

This can be relevant because different calculation/reporting approaches may affect the resulting PCF data.

#### `*Stage-level CO2e available`

Indicates whether stage-level carbon-emission information is available.

This is a categorical/indicator-type feature representing the availability of more detailed PCF information.

---

### Why are these three features grouped together?

Because you process them using **One-Hot Encoding** later.

Your structured-feature pipeline effectively separates the variables into:



```text
Structured Features
│
├── Numeric
│   ├── Year
│   └── Product Weight
│
├── One-Hot Categorical
│   ├── GICS Industry
│   ├── PCF Protocol
│   └── Stage-level CO2e availability
│
└── Target Encoded
    └── Country

This is an important design decision.

Why not process all categorical features in the same way?

Because different representations can be appropriate for different characteristics.

For the three ordinary categorical features:

Industry
Protocol
Stage-level availability
        ↓
One-Hot Encoding

For country:

Country
   ↓
Target Encoding

The target encoding gives a compact numerical representation, but because it uses the target, it requires additional leakage control.

Why create variables instead of directly using column names?

Instead of repeatedly writing:

X_train["Product weight (kg)"]

you can write:

X_train[WEIGHT_COL]

This improves:

Readability
Maintainability
Consistency
Reduced risk of spelling mistakes

If the original dataset column name changes, you can update the definition in one place.

Strong viva answer

If the examiner asks “What is the purpose of this cell?”, say:

“This cell defines the feature groups used in my structured feature engineering pipeline. I identify year and product weight as numeric features, country as a target-encoded feature, and industry, PCF protocol, and stage-level CO2e availability as categorical features that will be one-hot encoded. Using variables for the column names also makes the later preprocessing functions more readable and maintainable.”

One important connection to your previous cells

This cell provides the column definitions used by your build_structured_features() function:

YEAR_COL
WEIGHT_COL
     ↓
Numeric processing

OTHER_CATEGORICAL_FEATURES
     ↓
One-Hot Encoding

COUNTRY_COL
     ↓
OOF Target Encoding

So this cell itself does not transform any data. It simply tells the later feature-engineering code which columns belong to which processing strategy.

For your feature pipeline

Explain country encoding leakage
Check feature availability before splitting

# ============================================================
# PCA FOR SBERT FEATURES
# ============================================================

from sklearn.decomposition import PCA


def apply_pca_to_sbert(
    X_train_sbert,
    X_valid_sbert,
    n_components
):

    pca = PCA(
        n_components=n_components,
        random_state=42
    )

    X_train_pca = pca.fit_transform(
        X_train_sbert
    )

    X_valid_pca = pca.transform(
        X_valid_sbert
    )

    return (
        X_train_pca,
        X_valid_pca,
        pca
    )

PCA FOR SBERT FEATURES

This function applies Principal Component Analysis (PCA) to the SBERT embeddings.

The main reason is dimensionality reduction: SBERT produces a relatively high-dimensional vector, and PCA can compress that representation while retaining as much variance/information as possible.

Your flow is:

Combined Text
      ↓
SBERT
      ↓
High-dimensional embeddings
      ↓
PCA
      ↓
Reduced-dimensional embeddings
      ↓
ML model

1. Import PCA

```python id="pca001"
from sklearn.decomposition import PCA




This imports the `PCA` class from scikit-learn.

PCA stands for **Principal Component Analysis**.

It transforms the original features into a new set of features called **principal components**.

The first principal component captures the largest amount of variance, the second captures the next largest amount, and so on.

---

### 2. Define the PCA function



```python id="pca002"
def apply_pca_to_sbert(
    X_train_sbert,
    X_valid_sbert,
    n_components
):

You define a reusable function called:

apply_pca_to_sbert()

It takes three inputs.

`X_train_sbert`

The SBERT embeddings for the training/development portion.

For example:

800 observations × 384 SBERT features

`X_valid_sbert`

The SBERT embeddings for the validation portion.

For example:

200 observations × 384 features

`n_components`

The number of dimensions you want after PCA.

For example:

384 → 50

This means PCA reduces the SBERT representation from 384 dimensions to 50 dimensions.

3. Create the PCA object

```python id="pca003"
pca = PCA(
n_components=n_components,
random_state=42
)




This creates the PCA transformation.

#### `n_components=n_components`

This tells PCA how many principal components to retain.

If:



```python
n_components = 50

then:

Original SBERT
384 dimensions

        ↓ PCA

Reduced SBERT
50 dimensions

The important point is that PCA does not simply select the first 50 original SBERT features.

Instead, it creates new combinations of the original features.

4. Why reduce the SBERT dimensions?

This is a likely viva question.

SBERT gives you a dense embedding with many dimensions.

High-dimensional features can:

increase computational cost
increase memory requirements
introduce redundant information
make downstream ML models more expensive

PCA provides a way to create a smaller representation while retaining major patterns in the original embeddings.

Strong viva answer

“I applied PCA to reduce the dimensionality of the SBERT embeddings. SBERT produces a high-dimensional representation, and PCA allows me to create a more compact representation while retaining the principal sources of variance. This can reduce computational cost and potentially remove redundant information for the downstream ML models.”

5. Fit PCA on training data

```python id="pca004"
X_train_pca = pca.fit_transform(
X_train_sbert
)




This line performs **two operations**:



```text
fit
+
transform

`fit()`

PCA learns the transformation from the training data.

It determines the principal directions/components based on the training embeddings.

`transform()`

The training embeddings are then projected onto those learned principal components.

So:

X_train_sbert
      ↓
PCA learns components
      ↓
Training data transformed
      ↓
X_train_pca

Very important leakage point

PCA must be fitted only on training data.

You correctly do:

pca.fit_transform(X_train_sbert)

rather than fitting PCA on both training and validation data.

Why?

Because PCA learns information about the distribution of the data.

If you did:

pca.fit_transform(
    np.vstack([X_train_sbert, X_valid_sbert])
)

then the validation distribution would influence the learned PCA components.

That would introduce information from the validation set into the preprocessing stage.

Viva question: Why fit PCA only on training data?

“Because PCA learns its components from the data distribution. If I fitted it using both training and validation data, information from the validation set would influence the transformation. Therefore, I fit PCA only on the training data and apply the learned transformation to validation data.”

6. Transform validation data

```python id="pca005"
X_valid_pca = pca.transform(
X_valid_sbert
)




This applies the **already learned PCA transformation** to the validation embeddings.

Notice the difference:



```text
Training:
pca.fit_transform()

Validation:
pca.transform()

This distinction is extremely important.

`fit_transform`

Used when the transformation needs to learn parameters from training data and then transform it.

`transform`

Used when the transformation has already been learned and you want to apply the same transformation to new data.

So your pipeline is:

Training SBERT
      ↓
fit PCA
      ↓
learn principal components
      ↓
transform training


Validation SBERT
      ↓
use existing PCA
      ↓
transform validation

You do not calculate new PCA components for validation.

7. Return the results

```python id="pca006"
return (
X_train_pca,
X_valid_pca,
pca
)




The function returns three things.

#### `X_train_pca`

The reduced-dimensional training SBERT features.

#### `X_valid_pca`

The reduced-dimensional validation SBERT features.

#### `pca`

The fitted PCA object itself.

Returning the PCA object is useful because it contains the learned transformation and can be reused later.

For example, the same fitted PCA can be applied to another unseen dataset:



```text
New SBERT embeddings
        ↓
fitted PCA
        ↓
same reduced feature space

The most important concept: `fit_transform` vs `transform`

Remember this for your viva:

TRAINING
pca.fit_transform(X_train_sbert)
        ↓
Learn + transform


VALIDATION / TEST
pca.transform(X_valid_sbert)
        ↓
Transform only

If the examiner asks “Why don't you use fit_transform on validation?”, say:

“Because that would learn a separate transformation from the validation data. I want validation to remain unseen during preprocessing, so I reuse the PCA transformation learned from the training data.”

Example

Suppose SBERT produces:

Training:   800 × 384
Validation: 200 × 384

and you select:

n_components = 50

After PCA:

Training PCA:   800 × 50
Validation PCA: 200 × 50

So you reduce:

384 → 50

while keeping the observations unchanged.

The number of rows does not change.

Only the number of feature columns changes.

How this connects to your hybrid features

This is particularly important in your project because later you combine the text representation with structured features.

Your pipeline can be understood as:

Combined_Text
      ↓
    SBERT
      ↓
384-dimensional embeddings
      ↓
     PCA
      ↓
Reduced SBERT features
      +
Structured features
      ↓
Hybrid feature representation
      ↓
ML model

So PCA is applied to the SBERT part, not to the entire hybrid feature matrix at this stage.

Strong presentation wording

“Here, I define a reusable PCA function for reducing the dimensionality of the SBERT embeddings. The function receives training embeddings, validation embeddings, and the desired number of components. I initialise PCA with that number of components and fit it only on the training SBERT features. The training embeddings are therefore used to learn the principal components and are transformed into the reduced representation. I then apply the same fitted PCA transformation to the validation embeddings without refitting it. Finally, I return both reduced feature matrices and the fitted PCA object. This ensures dimensionality reduction while avoiding information leakage from the validation data.”

Examiner trap questions

“Does PCA select the most important original SBERT features?”

Answer:

“No. PCA creates new orthogonal components as linear combinations of the original features. It does not simply select a subset of the original dimensions.”

“Does PCA guarantee better prediction?”

Answer:

“No. PCA is a dimensionality-reduction technique, not a prediction-improvement guarantee. It may reduce computational cost and redundancy, but the effect on predictive performance needs to be evaluated empirically.”

“Why not remove PCA completely?”

“That is a valid alternative. PCA is a modelling choice that can be compared experimentally. If the original SBERT dimensions provide better validation performance, retaining them may be preferable.”

“Does PCA use the target variable?”

Answer:

“No. Standard PCA is unsupervised. It learns the principal directions from the feature matrix and does not use the PCF target.”

“Why is that useful for leakage control?”

“Because PCA itself does not use the target, but it still learns from the feature distribution. Therefore, I must still fit it only on the training data and not on validation or test data.”

For your SBERT PCA step

Explain how to choose components
Check PCA assumptions

Feature Column Definitions

1. Year column

Viva question: Why include reporting year?

2. Product weight column

3. Country column

Viva question: Why target-encode country instead of one-hot encoding it?

4. Other categorical features

Why not process all categorical features in the same way?

Why create variables instead of directly using column names?

Strong viva answer

One important connection to your previous cells

PCA FOR SBERT FEATURES

1. Import PCA

X_train_sbert

X_valid_sbert

n_components

3. Create the PCA object

4. Why reduce the SBERT dimensions?

Strong viva answer

5. Fit PCA on training data

fit()

transform()

Very important leakage point

Viva question: Why fit PCA only on training data?

6. Transform validation data

fit_transform

transform

7. Return the results

The most important concept: fit_transform vs transform

Example

How this connects to your hybrid features

Strong presentation wording

Examiner trap questions

“Does PCA select the most important original SBERT features?”

“Does PCA guarantee better prediction?”

“Why not remove PCA completely?”

“Does PCA use the target variable?”

“Why is that useful for leakage control?”

`X_train_sbert`

`X_valid_sbert`

`n_components`

`fit()`

`transform()`

`fit_transform`

`transform`

The most important concept: `fit_transform` vs `transform`