DEV Community: Talha Bin Rais

step-01

Talha Bin Rais — Wed, 19 Aug 2026 13:52:59 +0000

Yes. Let's restart the ML/hybrid notebook from Step 01, but keep the decisions that are already sound. We will not redesign BEACON; this is only the ML prediction stage.

I recommend we build it one step at a time and run/check each step before moving to the next.

New notebook structure

01. Load Carbon Catalogue
02. Define target and input features
03. Fixed 80/20 train-test split
04. Create inner training / validation split
05. Structured-feature preprocessing
06. Text preparation
07. Sentence-BERT embeddings
08. Optional PCA experiment
09. Feature fusion
10. Baseline ML models
11. Hybrid ML models
12. Validation comparison
13. Select best configuration
14. Freeze final model
15. Final 20% test evaluation
16. Baseline vs Hybrid comparison

Step 01 — Load and inspect the dataset

Do not do feature engineering, PCA, Sentence-BERT, or modelling yet.

The first notebook cell should simply establish the dataset.

# ============================================================
# STEP 01 — LOAD AND INSPECT CARBON CATALOGUE
# ============================================================

import pandas as pd
import numpy as np

# Load dataset
df = pd.read_csv("YOUR_CARBON_CATALOGUE_FILE.csv")

print("Dataset shape:", df.shape)

print("\nColumns:")
print(df.columns.tolist())

print("\nFirst 5 rows:")
display(df.head())

print("\nData types:")
display(df.dtypes)

print("\nMissing values:")
display(df.isnull().sum())

print("\nDuplicate rows:", df.duplicated().sum())

What we want to establish here

After running Step 01, we should know:

Number of rows
Number of columns
Exact column names
Data types
Missing values
Duplicate rows

Most importantly, we need to confirm the exact PCF target column and the exact text columns you intend to use.

Then Step 02 will define the modelling inputs

Conceptually:

Target
   │
   └── Product Carbon Footprint (PCF)

Structured branch
   │
   ├── Year
   ├── Weight
   ├── Industry information
   ├── Protocol information
   └── other selected structured features

Text branch
   │
   ├── Product Name
   ├── Company
   └── Country

We should not guess these columns. We will take them from your existing notebook/dataset so that the new implementation remains consistent with the work you have already done.

Step 03 — Split the data

We will retain the good part of the previous design:

                 ALL DATA
                 866 rows
                    │
             ┌──────┴──────┐
             │             │
           80%           20%
        Development       Test
          ~692            ~174
             │
        ┌────┴────┐
        │         │
      80%       20%
     ~553       ~139
     Train    Validation

The 174 test rows remain untouched until the very end.

That is important and I do not recommend changing it.

Step 04 onwards — the hybrid experiment

The central part will eventually become:

                   TEXT
                     │
                     ▼
              Sentence-BERT
                     │
                     ▼
                  384-D
                     │
              ┌──────┴──────┐
              │             │
          No PCA          PCA
              │             │
              │       200 / 100 / 50
              │             │
              └──────┬──────┘
                     │
                     ▼
              TEXT FEATURES
                     │
                     │
STRUCTURED ──────────┘
FEATURES
     │
     ▼
FEATURE FUSION
     │
     ▼
HYBRID FEATURES
     │
     ▼
ML REGRESSOR
     │
     ▼
PCF

The 384-dimensional version is our natural starting point. PCA is then an experiment to determine whether reducing those dimensions improves or maintains performance.

So we are not assuming that 384 is too high.

One important rule for the restart

Let's not write the entire notebook in one go.

We should work like this:

Step 01 → run → inspect result → confirm → Step 02 → run → inspect → ...

That will prevent the previous problem where the notebook became technically complicated before we had established exactly what each part was doing.

Start now with Step 01.

Run the Step 01 cell using your actual Carbon Catalogue file.

Send me the output of:

Dataset shape
Columns
Data types
Missing values
Duplicate rows

Then I will give you Step 02 only, using the actual structure of your dataset.

confusion-10

Talha Bin Rais — Wed, 19 Aug 2026 13:51:53 +0000

Yes — if you want to restart the ML part from the beginning, this is actually a good point to simplify it properly. We do not need to redesign the whole project; we can rebuild the hybrid experiment cleanly around the idea you originally intended.

And yes, your understanding of PCA is correct:

Sentence-BERT gives a 384-dimensional embedding. PCA can be used to reduce those 384 dimensions to a smaller number of dimensions.

But 384 dimensions is not automatically "too high". With your dataset size, the question is whether keeping all 384 dimensions is useful and stable.

1. Start from the simplest possible architecture

I recommend starting with this:

                 CARBON CATALOGUE
                       │
              Fixed 80/20 split
                       │
          ┌────────────┴────────────┐
          │                         │
          ▼                         ▼
   STRUCTURED FEATURES          TEXT FEATURES
          │                         │
   Year, Weight, etc.       Product + Company
          │                   + Country
          │                         │
          │                         ▼
          │                  Sentence-BERT
          │                         │
          │                    384-D embedding
          │                         │
          └────────────┬────────────┘
                       ▼
                 FEATURE FUSION
                       │
                       ▼
             HYBRID FEATURE VECTOR
                       │
                       ▼
                  ML REGRESSOR
                       │
                       ▼
                  PCF prediction

That is the core hybrid model.

Don't introduce PCA yet.

First establish that this model works.

2. What exactly happens to the text?

Suppose one row contains:

Product Name:
Organic cotton T-shirt

Company:
Example Textile Ltd

Country:
Bangladesh

You combine those text fields:

Organic cotton T-shirt | Example Textile Ltd | Bangladesh

Then:

                  Combined text
                       │
                       ▼
               Sentence-BERT
                       │
                       ▼
          [e1, e2, e3, ..., e384]

So each row gets 384 numerical features.

For example, conceptually:

e1       = 0.034
e2       = -0.127
e3       = 0.082
...
e384     = 0.041

These are the embedding features.

3. Is 384-D too high?

Short answer: No.

384 dimensions is actually quite modest for a text embedding.

The bigger issue is your dataset size.

You have approximately:

866 total rows

and therefore roughly:

553 training
139 validation
174 test

So you have around 553 inner-training observations.

If you directly use:

384 embedding features

you have a relatively high feature-to-sample ratio.

It is not impossible, but it creates a reasonable research question:

Does dimensionality reduction improve generalisation?

That's exactly where PCA becomes useful.

4. PCA's role becomes very clear now

Think of PCA like this:

Sentence-BERT
     │
     ▼
384 dimensions
     │
     │ PCA
     ▼
smaller representation

For example:

PCA attempts to preserve as much of the variation/information in the original embedding as possible while representing it using fewer dimensions.

So PCA is not creating the hybrid model.

Instead:

Sentence-BERT creates the text representation. PCA optionally compresses that representation. Feature fusion creates the hybrid representation.

This distinction is very important.

5. Therefore, don't immediately test 50, 75, 100, 150, 200

This is where I would change the previous notebook.

If we start again, I recommend a two-step experiment.

Experiment 1 — Full embedding

Text
 ↓
Sentence-BERT
 ↓
384-D
 ↓
Fusion
 ↓
ML

This establishes your baseline hybrid model.

Experiment 2 — PCA

Then investigate whether PCA improves it:

Text
 ↓
Sentence-BERT
 ↓
384-D
 ↓
PCA
 ↓
Fusion
 ↓
ML

Now PCA has a clear scientific purpose.

6. What should PCA dimensions be?

Don't arbitrarily choose many values simply because they are available.

I would use a small, justified set such as:

where:

384 = no PCA
200 = moderate reduction
100 = substantial reduction
50 = strong reduction

Then compare validation performance.

You could also select the PCA dimension based on explained variance, which is arguably more principled.

For example:

PCA → retain 90% variance

PCA → retain 95% variance

This is actually preferable academically because you are not saying:

"I randomly decided to use 50 dimensions."

Instead:

"The number of principal components was selected according to the amount of variance retained."

However, because your dissertation has a practical time constraint, I would first run 384/200/100/50 and inspect the results. We don't need to over-engineer this.

7. The important leakage issue

This is very important when you rebuild it.

PCA must be fitted only on the training data.

Not:

Train + Validation + Test
        ↓
      PCA

That would leak information.

Instead:

Inner training embeddings
        │
        ▼
    FIT PCA
        │
        ├──────────────► transform training
        │
        └──────────────► transform validation

Then, after selecting the configuration, the final model is evaluated on the untouched test set.

So:

TRAIN
 ↓
fit Sentence-BERT? → no training required if using pre-trained frozen SBERT
fit PCA
fit ML model
 ↓
VALIDATION
 ↓
select PCA + model
 ↓
FREEZE
 ↓
TEST

This is the clean experimental protocol.

8. Why keep the 553 / 139 / 174 split?

I would keep it.

It is actually one of the stronger parts of the previous notebook.

866 total
│
├── 692 development = 80%
│   │
│   ├── 553 inner training
│   └── 139 validation
│
└── 174 final test = 20%

The roles are:

Data	Purpose
553	Train models
139	Select model/PCA configuration
174	Final unbiased evaluation

You should not use the 174 test rows to decide whether PCA=50 or PCA=384 is better.

9. Then compare structured vs hybrid

This is where your dissertation becomes much stronger.

Don't only report:

"Our hybrid model achieved X."

Instead ask:

Baseline model

Structured features
        ↓
Random Forest / XGBoost / etc.
        ↓
PCF

Hybrid model

Structured features
        +
Sentence-BERT text representation
        ↓
Feature fusion
        ↓
Same ML model
        ↓
PCF

Then:

             Baseline       Hybrid
RMSE            X             Y
MAE             X             Y
R²              X             Y

Now you can answer the real research question:

Does incorporating semantic information from textual PCF metadata improve PCF prediction compared with structured features alone?

That is much more academically meaningful than simply testing 25 combinations.

10. And the "best ML model" issue

I recommend we handle this in a controlled way.

For example:

                 STRUCTURED
                     │
          ┌──────────┼──────────┐
          ▼          ▼          ▼
        Ridge       RF       XGBoost
          │          │          │
          └──────────┼──────────┘
                     ▼
              Best baseline

Then:

             HYBRID FEATURES
                    │
          ┌─────────┼─────────┐
          ▼         ▼         ▼
        Ridge      RF      XGBoost
          │         │         │
          └─────────┼─────────┘
                    ▼
              Best hybrid

You can retain LightGBM and CatBoost if those are already part of your project methodology, but don't let the number of models obscure the central experiment.

11. So the complete experiment becomes

I would freeze this as your new ML workflow:

STEP 1
Carbon Catalogue
       │
       ▼
Fixed 80/20 split
       │
       ├─────────────────────┐
       │                     │
       ▼                     ▼
Structured branch       Text branch
       │                     │
       │              Sentence-BERT
       │                     │
       │                  384-D
       │                     │
       │                Optional PCA
       │                     │
       └──────────┬──────────┘
                  ▼
            Feature Fusion
                  │
                  ▼
          Hybrid Feature Set
                  │
                  ▼
            ML Regression
                  │
                  ▼
             Validation
                  │
                  ▼
          Select best setup
                  │
                  ▼
                Freeze
                  │
                  ▼
          Final 20% test
                  │
                  ▼
          Final performance

12. My recommendation about PCA

I would not decide now that 384 is too high.

Let the experiment answer that.

Run:

Hybrid-384
Hybrid-PCA-200
Hybrid-PCA-100
Hybrid-PCA-50

and compare validation performance.

If 384 performs best:

Keep 384.

If 100 performs similarly or better:

Use 100 and explain that dimensionality reduction achieved comparable/better predictive performance with a lower-dimensional representation.

If 50 performs best:

Use 50.

So PCA becomes an empirical dimensionality-reduction experiment, rather than an assumption.

13. One thing I would change from the previous notebook

I would make the notebook extremely explicit with four sections:

01. DATA SPLIT
02. STRUCTURED FEATURES
03. SENTENCE-BERT TEXT EMBEDDINGS
04. HYBRID FEATURE FUSION
05. PCA EXPERIMENT
06. MODEL COMPARISON
07. MODEL SELECTION
08. FINAL TEST

Then anybody opening the notebook can understand exactly what is happening.

The core idea in one sentence

Sentence-BERT converts textual PCF metadata into a 384-dimensional semantic representation; PCA is optionally used to reduce this representation; the resulting text features are concatenated with structured PCF features to form a hybrid feature set, which is then used by a regression model to predict PCF.

That is the model I recommend you build from the beginning.

And yes, we can restart the notebook from Step 1 and build it cleanly rather than trying to repair the current complicated version.