Yes. Let's restart the ML/hybrid notebook from Step 01, but keep the decisions that are already sound. We will not redesign BEACON; this is only the ML prediction stage.
I recommend we build it one step at a time and run/check each step before moving to the next.
New notebook structure
01. Load Carbon Catalogue
02. Define target and input features
03. Fixed 80/20 train-test split
04. Create inner training / validation split
05. Structured-feature preprocessing
06. Text preparation
07. Sentence-BERT embeddings
08. Optional PCA experiment
09. Feature fusion
10. Baseline ML models
11. Hybrid ML models
12. Validation comparison
13. Select best configuration
14. Freeze final model
15. Final 20% test evaluation
16. Baseline vs Hybrid comparison
Step 01 — Load and inspect the dataset
Do not do feature engineering, PCA, Sentence-BERT, or modelling yet.
The first notebook cell should simply establish the dataset.
# ============================================================
# STEP 01 — LOAD AND INSPECT CARBON CATALOGUE
# ============================================================
import pandas as pd
import numpy as np
# Load dataset
df = pd.read_csv("YOUR_CARBON_CATALOGUE_FILE.csv")
print("Dataset shape:", df.shape)
print("\nColumns:")
print(df.columns.tolist())
print("\nFirst 5 rows:")
display(df.head())
print("\nData types:")
display(df.dtypes)
print("\nMissing values:")
display(df.isnull().sum())
print("\nDuplicate rows:", df.duplicated().sum())
What we want to establish here
After running Step 01, we should know:
- Number of rows
- Number of columns
- Exact column names
- Data types
- Missing values
- Duplicate rows
Most importantly, we need to confirm the exact PCF target column and the exact text columns you intend to use.
Then Step 02 will define the modelling inputs
Conceptually:
Target
│
└── Product Carbon Footprint (PCF)
Structured branch
│
├── Year
├── Weight
├── Industry information
├── Protocol information
└── other selected structured features
Text branch
│
├── Product Name
├── Company
└── Country
We should not guess these columns. We will take them from your existing notebook/dataset so that the new implementation remains consistent with the work you have already done.
Step 03 — Split the data
We will retain the good part of the previous design:
ALL DATA
866 rows
│
┌──────┴──────┐
│ │
80% 20%
Development Test
~692 ~174
│
┌────┴────┐
│ │
80% 20%
~553 ~139
Train Validation
The 174 test rows remain untouched until the very end.
That is important and I do not recommend changing it.
Step 04 onwards — the hybrid experiment
The central part will eventually become:
TEXT
│
▼
Sentence-BERT
│
▼
384-D
│
┌──────┴──────┐
│ │
No PCA PCA
│ │
│ 200 / 100 / 50
│ │
└──────┬──────┘
│
▼
TEXT FEATURES
│
│
STRUCTURED ──────────┘
FEATURES
│
▼
FEATURE FUSION
│
▼
HYBRID FEATURES
│
▼
ML REGRESSOR
│
▼
PCF
The 384-dimensional version is our natural starting point. PCA is then an experiment to determine whether reducing those dimensions improves or maintains performance.
So we are not assuming that 384 is too high.
One important rule for the restart
Let's not write the entire notebook in one go.
We should work like this:
Step 01 → run → inspect result → confirm → Step 02 → run → inspect → ...
That will prevent the previous problem where the notebook became technically complicated before we had established exactly what each part was doing.
Start now with Step 01.
Run the Step 01 cell using your actual Carbon Catalogue file.
Send me the output of:
Dataset shape
Columns
Data types
Missing values
Duplicate rows
Then I will give you Step 02 only, using the actual structure of your dataset.
Top comments (0)