DEV Community: Sajjad Rahman

ML_Log_Target_PCF

Sajjad Rahman — Sat, 15 Aug 2026 05:14:42 +0000

# 03_ML_Log_Target_PCF.ipynb

This experiment tests log-transformed PCF as the target.

The important difference from Notebook 02 is:

Component	Experiment 01	Experiment 02	Experiment 03
PCF target	Raw	Raw	`log1p(PCF)`
Product Weight	Raw	Log	Raw
Train/test split	Same	Same	Same
Models	7	7	7
Tuning	3	3	3
Evaluation	MAE/RMSE/R²	MAE/RMSE/R²	MAE/RMSE/R² on original PCF scale

There is one crucial technical difference: the model will train on log1p(PCF), but we must convert predictions back to the original PCF scale before calculating MAE, RMSE and R².

Otherwise, Experiment 03 would not be directly comparable with Experiments 01 and 02.

Notebook 03 — `03_ML_Log_Target_PCF.ipynb`

Cell 1 — Markdown

# BEACON Machine Learning — Log Target Experiment

## Experiment 03: Log-Transformed PCF Target

### Purpose

This experiment investigates whether logarithmic transformation of the
PCF target improves machine-learning prediction performance.

The PCF target is highly right-skewed. Therefore, `log1p(PCF)` is used
during model training to reduce the influence of extreme target values.

Product Weight remains in its original representation.

The experiment follows the same dataset, train-test split,
preprocessing strategy, regression models and evaluation procedure used
in Experiments 01 and 02.

### Experimental comparison

Experiment 01:
- Raw PCF target
- Raw Product Weight

Experiment 02:
- Raw PCF target
- Log-transformed Product Weight

Experiment 03:
- Log-transformed PCF target
- Raw Product Weight

### Evaluation

Models are trained using the logarithmic PCF target. Predictions are
converted back to the original PCF scale using `expm1()` before
calculating MAE, RMSE and R².

This allows the results to remain directly comparable with Experiments
01 and 02.

Cell 2 — Imports

# ============================================================
# 1. IMPORT LIBRARIES
# ============================================================

import numpy as np
import pandas as pd

from sklearn.model_selection import (
    train_test_split,
    GridSearchCV
)

from sklearn.preprocessing import OneHotEncoder

from sklearn.linear_model import (
    LinearRegression,
    ElasticNet,
    BayesianRidge
)

from sklearn.ensemble import (
    RandomForestRegressor,
    ExtraTreesRegressor,
    HistGradientBoostingRegressor
)

from sklearn.metrics import (
    mean_absolute_error,
    mean_squared_error,
    r2_score
)

from xgboost import XGBRegressor

from category_encoders import TargetEncoder

import warnings

warnings.filterwarnings("ignore")

RANDOM_STATE = 42

Cell 3 — Load dataset

# ============================================================
# 2. LOAD ORIGINAL CARBON CATALOGUE DATA
# ============================================================

DATA_PATH = (
    "/kaggle/input/datasets/"
    "sajjadrahman56/product-level-of-pcf/"
    "ProductLevel.csv"
)

df = pd.read_csv(
    DATA_PATH,
    encoding="latin1"
)

print("Original dataset shape:", df.shape)

Cell 4 — Create ML dataset

# ============================================================
# 3. CREATE ML DATASET
# ============================================================

df_ml = pd.DataFrame()

df_ml["Year"] = df["Year of reporting"]

df_ml["Company"] = df["Company"]

df_ml["Stage_Level_CO2e_Available"] = (
    df["*Stage-level CO2e available"]
)

df_ml["Country"] = (
    df["Country (where company is incorporated)"]
)

df_ml["Industry"] = (
    df["Company's GICS Industry"]
)

df_ml["Product_Weight"] = (
    df["Product weight (kg)"]
)

df_ml["PCF_Protocol"] = (
    df["Protocol used for PCF"]
)

df_ml["PCF"] = (
    df["Product's carbon footprint (PCF, kg CO2e)"]
)

print("ML dataset shape:", df_ml.shape)

display(df_ml.head())

Cell 5 — Clean PCF

# ============================================================
# 4. CLEAN PCF TARGET
# ============================================================

df_ml["PCF"] = pd.to_numeric(
    df_ml["PCF"],
    errors="coerce"
)

df_ml = df_ml[
    df_ml["PCF"].notna()
].copy()

df_ml = df_ml[
    df_ml["PCF"] >= 0
].copy()

print("Usable rows:", len(df_ml))

print("\nOriginal PCF summary:")
display(
    df_ml["PCF"].describe()
)

print(
    "\nOriginal PCF skewness:",
    df_ml["PCF"].skew()
)

You should again have:

Cell 6 — Define X and raw y

# ============================================================
# 5. DEFINE FEATURES AND RAW TARGET
# ============================================================

FEATURES = [
    "Year",
    "Company",
    "Stage_Level_CO2e_Available",
    "Country",
    "Industry",
    "Product_Weight",
    "PCF_Protocol"
]

TARGET = "PCF"

X = df_ml[
    FEATURES
].copy()

y = df_ml[
    TARGET
].copy()

print("X shape:", X.shape)
print("y shape:", y.shape)

Cell 7 — Train/test split

Exactly the same as the first two notebooks.

# ============================================================
# 6. TRAIN / TEST SPLIT
# ============================================================

X_train, X_test, y_train_raw, y_test_raw = train_test_split(
    X,
    y,
    test_size=0.20,
    random_state=RANDOM_STATE
)

print("Training rows:", len(X_train))
print("Testing rows :", len(X_test))

Expected:

Training rows: 692
Testing rows : 174

Cell 8 — Create safe copies

# ============================================================
# 7. CREATE SAFE COPIES
# ============================================================

X_train = X_train.copy()
X_test = X_test.copy()

y_train_raw = y_train_raw.copy()
y_test_raw = y_test_raw.copy()

Cell 9 — Create log target

This is the major change in Notebook 03.

# ============================================================
# 8. LOG-TRANSFORM PCF TARGET
#
# log1p(x) = log(1 + x)
# ============================================================

y_train = np.log1p(
    y_train_raw
)

y_test = np.log1p(
    y_test_raw
)

print("Raw PCF skewness:")
print(
    y_train_raw.skew()
)

print(
    "\nLog-transformed PCF skewness:"
)

print(
    y_train.skew()
)

You should see something similar to:

Raw PCF skewness: ~15+
Log-transformed PCF skewness: much lower

Cell 10 — Keep original y for evaluation

This is very important.

# ============================================================
# 9. STORE ORIGINAL TARGET FOR FINAL EVALUATION
# ============================================================

y_test_original = y_test_raw.copy()

print(
    "Training target used by models: log1p(PCF)"
)

print(
    "Evaluation target: original PCF"
)

Cell 11 — Country → Region

Same as the previous notebooks.

# ============================================================
# 10. COUNTRY → REGION
# ============================================================

country_to_region = {

    # North America
    "USA": "North America",
    "Canada": "North America",

    # Europe
    "Germany": "Europe",
    "Netherlands": "Europe",
    "United Kingdom": "Europe",
    "Switzerland": "Europe",
    "Sweden": "Europe",
    "Finland": "Europe",
    "Italy": "Europe",
    "France": "Europe",
    "Spain": "Europe",
    "Belgium": "Europe",
    "Ireland": "Europe",
    "Luxembourg": "Europe",
    "Lithuania": "Europe",
    "Greece": "Europe",

    # East Asia
    "Japan": "East Asia",
    "Taiwan": "East Asia",
    "South Korea": "East Asia",
    "China": "East Asia",

    # South Asia
    "India": "South Asia",

    # Southeast Asia
    "Malaysia": "Southeast Asia",
    "Indonesia": "Southeast Asia",

    # South America
    "Brazil": "South America",
    "Chile": "South America",
    "Colombia": "South America",

    # Africa
    "South Africa": "Africa",

    # Oceania
    "Australia": "Oceania"
}

X_train["Region"] = X_train[
    "Country"
].map(country_to_region)

X_test["Region"] = X_test[
    "Country"
].map(country_to_region)

X_train["Region"] = (
    X_train["Region"].fillna("Other")
)

X_test["Region"] = (
    X_test["Region"].fillna("Other")
)

X_train.drop(
    columns="Country",
    inplace=True
)

X_test.drop(
    columns="Country",
    inplace=True
)

Cell 12 — Rare Industry

# ============================================================
# 11. RARE INDUSTRY → OTHER
# ============================================================

industry_counts = X_train[
    "Industry"
].value_counts()

rare_industries = industry_counts[
    industry_counts < 10
].index

X_train["Industry"] = X_train[
    "Industry"
].replace(
    rare_industries,
    "Other"
)

X_test["Industry"] = X_test[
    "Industry"
].replace(
    rare_industries,
    "Other"
)

Cell 13 — PCF Protocol

# ============================================================
# 12. PCF PROTOCOL → TOP 5 + OTHER
# ============================================================

top_protocols = [
    "ISO",
    "Not reported",
    "GHGP",
    "PAS2050",
    "TRACI 2.1"
]

X_train["PCF_Protocol"] = X_train[
    "PCF_Protocol"
].where(
    X_train["PCF_Protocol"].isin(
        top_protocols
    ),
    "Other"
)

X_test["PCF_Protocol"] = X_test[
    "PCF_Protocol"
].where(
    X_test["PCF_Protocol"].isin(
        top_protocols
    ),
    "Other"
)

Cell 14 — Stage-level CO2e

# ============================================================
# 13. STAGE-LEVEL CO2e → BINARY
# ============================================================

binary_map = {
    "No": 0,
    "Yes": 1
}

X_train[
    "Stage_Level_CO2e_Available"
] = (
    X_train[
        "Stage_Level_CO2e_Available"
    ]
    .map(binary_map)
    .fillna(0)
)

X_test[
    "Stage_Level_CO2e_Available"
] = (
    X_test[
        "Stage_Level_CO2e_Available"
    ]
    .map(binary_map)
    .fillna(0)
)

Cell 15 — Product Weight winsorisation

Here we return to the raw Product Weight representation.

# ============================================================
# 14. PRODUCT WEIGHT — WINSORISATION
#
# No logarithmic transformation is applied to Product Weight
# in Experiment 03.
# ============================================================

Q1 = X_train[
    "Product_Weight"
].quantile(0.25)

Q3 = X_train[
    "Product_Weight"
].quantile(0.75)

IQR = Q3 - Q1

lower = Q1 - 1.5 * IQR
upper = Q3 + 1.5 * IQR

print("Winsorisation bounds")
print("Lower:", lower)
print("Upper:", upper)

X_train[
    "Product_Weight"
] = X_train[
    "Product_Weight"
].clip(
    lower=lower,
    upper=upper
)

X_test[
    "Product_Weight"
] = X_test[
    "Product_Weight"
].clip(
    lower=lower,
    upper=upper
)

Cell 16 — Company target encoding

# ============================================================
# 15. COMPANY → TARGET ENCODING
#
# IMPORTANT:
# The encoder is fitted using y_train, which is the log PCF
# target in this experiment.
# ============================================================

te = TargetEncoder(
    cols=["Company"],
    min_samples_leaf=20,
    smoothing=10
)

X_train = te.fit_transform(
    X_train,
    y_train
)

X_test = te.transform(
    X_test
)

Cell 17 — One-hot encoding

# ============================================================
# 16. ONE-HOT ENCODING
# ============================================================

categorical_features = [
    "Industry",
    "PCF_Protocol",
    "Region"
]

encoder = OneHotEncoder(
    drop="first",
    handle_unknown="ignore",
    sparse_output=False
)

encoded_train = encoder.fit_transform(
    X_train[
        categorical_features
    ]
)

encoded_test = encoder.transform(
    X_test[
        categorical_features
    ]
)

encoded_train_df = pd.DataFrame(
    encoded_train,
    columns=encoder.get_feature_names_out(
        categorical_features
    ),
    index=X_train.index
)

encoded_test_df = pd.DataFrame(
    encoded_test,
    columns=encoder.get_feature_names_out(
        categorical_features
    ),
    index=X_test.index
)

X_train = pd.concat(
    [
        X_train.drop(
            columns=categorical_features
        ),
        encoded_train_df
    ],
    axis=1
)

X_test = pd.concat(
    [
        X_test.drop(
            columns=categorical_features
        ),
        encoded_test_df
    ],
    axis=1
)

Cell 18 — Final feature check

# ============================================================
# 17. FINAL FEATURE CHECK
# ============================================================

print(
    "Final X_train shape:",
    X_train.shape
)

print(
    "Final X_test shape:",
    X_test.shape
)

print(
    "\nProduct Weight present:",
    "Product_Weight" in X_train.columns
)

print(
    "Log Product Weight present:",
    "Log_Product_Weight" in X_train.columns
)

print(
    "Target transformation: log1p(PCF)"
)

print(
    "\nAll features numerical:",
    X_train.select_dtypes(
        exclude="number"
    ).empty
)

Expected:

Product Weight present: True
Log Product Weight present: False
Target transformation: log1p(PCF)
All features numerical: True

Cell 19 — Seven baseline models

# ============================================================
# 18. DEFINE BASELINE ML MODELS
# ============================================================

models = {

    "Linear Regression":
        LinearRegression(),

    "ElasticNet":
        ElasticNet(
            random_state=RANDOM_STATE
        ),

    "Bayesian Ridge":
        BayesianRidge(),

    "Random Forest":
        RandomForestRegressor(
            random_state=RANDOM_STATE
        ),

    "Extra Trees":
        ExtraTreesRegressor(
            random_state=RANDOM_STATE
        ),

    "HistGradientBoosting":
        HistGradientBoostingRegressor(
            random_state=RANDOM_STATE
        ),

    "XGBoost":
        XGBRegressor(
            random_state=RANDOM_STATE
        )
}

Cell 20 — Train and evaluate correctly

This cell is different from Notebooks 01 and 02.

The model predicts log-PCF.

We convert it back:

log-PCF prediction → expm1() → PCF prediction

Then evaluate against original PCF.

# ============================================================
# 19. BASELINE MODEL EVALUATION
#
# Models are trained on log(PCF).
# Predictions are transformed back to the original PCF scale
# before calculating MAE, RMSE and R².
# ============================================================

baseline_results = []

for name, model in models.items():

    print(
        f"Training: {name}"
    )

    # Train using log-transformed target
    model.fit(
        X_train,
        y_train
    )

    # Predict log-PCF
    y_pred_log = model.predict(
        X_test
    )

    # Convert predictions back to original PCF scale
    y_pred = np.expm1(
        y_pred_log
    )

    # Prevent tiny numerical negative values
    y_pred = np.maximum(
        y_pred,
        0
    )

    # Evaluate on original PCF scale
    mae = mean_absolute_error(
        y_test_original,
        y_pred
    )

    rmse = np.sqrt(
        mean_squared_error(
            y_test_original,
            y_pred
        )
    )

    r2 = r2_score(
        y_test_original,
        y_pred
    )

    baseline_results.append({

        "Model": name,

        "MAE": mae,

        "RMSE": rmse,

        "R²": r2
    })

Cell 21 — Baseline results

# ============================================================
# 20. BASELINE RESULTS
# ============================================================

baseline_results_df = pd.DataFrame(
    baseline_results
)

baseline_results_df = (
    baseline_results_df
    .sort_values(
        "R²",
        ascending=False
    )
    .reset_index(drop=True)
)

display(
    baseline_results_df.style.format({
        "MAE": "{:,.2f}",
        "RMSE": "{:,.2f}",
        "R²": "{:.4f}"
    })
)

Cell 22 — Models for tuning

Same three as the other experiments.

# ============================================================
# 21. SELECT MODELS FOR HYPERPARAMETER TUNING
# ============================================================

tuning_models = {

    "Random Forest":
        RandomForestRegressor(
            random_state=RANDOM_STATE
        ),

    "Extra Trees":
        ExtraTreesRegressor(
            random_state=RANDOM_STATE
        ),

    "XGBoost":
        XGBRegressor(
            random_state=RANDOM_STATE
        )
}

print(
    "Models selected for tuning:"
)

for name in tuning_models:
    print("-", name)

Cell 23 — RF GridSearch

# ============================================================
# 22. RANDOM FOREST — GRID SEARCH
# ============================================================

rf_param_grid = {

    "n_estimators": [
        100,
        200,
        300
    ],

    "max_depth": [
        None,
        10,
        20
    ],

    "min_samples_split": [
        2,
        5
    ],

    "min_samples_leaf": [
        1,
        2
    ]
}

rf_grid = GridSearchCV(
    estimator=tuning_models[
        "Random Forest"
    ],

    param_grid=rf_param_grid,

    scoring="neg_root_mean_squared_error",

    cv=5,

    n_jobs=-1,

    verbose=1
)

rf_grid.fit(
    X_train,
    y_train
)

print(
    "\nBest Random Forest parameters:"
)

print(
    rf_grid.best_params_
)

Cell 24 — Tuned RF evaluation

Again, convert back to original PCF.

# ============================================================
# 23. EVALUATE TUNED RANDOM FOREST
# ============================================================

best_rf = rf_grid.best_estimator_

rf_pred_log = best_rf.predict(
    X_test
)

rf_pred = np.expm1(
    rf_pred_log
)

rf_pred = np.maximum(
    rf_pred,
    0
)

rf_mae = mean_absolute_error(
    y_test_original,
    rf_pred
)

rf_rmse = np.sqrt(
    mean_squared_error(
        y_test_original,
        rf_pred
    )
)

rf_r2 = r2_score(
    y_test_original,
    rf_pred
)

print("Random Forest")
print("MAE :", rf_mae)
print("RMSE:", rf_rmse)
print("R²  :", rf_r2)

Cell 25 — Extra Trees GridSearch

# ============================================================
# 24. EXTRA TREES — GRID SEARCH
# ============================================================

et_param_grid = {

    "n_estimators": [
        100,
        200,
        300
    ],

    "max_depth": [
        None,
        10,
        20
    ],

    "min_samples_split": [
        2,
        5
    ],

    "min_samples_leaf": [
        1,
        2
    ]
}

et_grid = GridSearchCV(
    estimator=tuning_models[
        "Extra Trees"
    ],

    param_grid=et_param_grid,

    scoring="neg_root_mean_squared_error",

    cv=5,

    n_jobs=-1,

    verbose=1
)

et_grid.fit(
    X_train,
    y_train
)

print(
    "\nBest Extra Trees parameters:"
)

print(
    et_grid.best_params_
)

Cell 26 — Tuned Extra Trees evaluation

# ============================================================
# 25. EVALUATE TUNED EXTRA TREES
# ============================================================

best_et = et_grid.best_estimator_

et_pred_log = best_et.predict(
    X_test
)

et_pred = np.expm1(
    et_pred_log
)

et_pred = np.maximum(
    et_pred,
    0
)

et_mae = mean_absolute_error(
    y_test_original,
    et_pred
)

et_rmse = np.sqrt(
    mean_squared_error(
        y_test_original,
        et_pred
    )
)

et_r2 = r2_score(
    y_test_original,
    et_pred
)

print("Extra Trees")
print("MAE :", et_mae)
print("RMSE:", et_rmse)
print("R²  :", et_r2)

Cell 27 — XGBoost GridSearch

# ============================================================
# 26. XGBOOST — GRID SEARCH
# ============================================================

xgb_param_grid = {

    "n_estimators": [
        100,
        200,
        300
    ],

    "max_depth": [
        3,
        5,
        7
    ],

    "learning_rate": [
        0.01,
        0.05,
        0.1
    ],

    "subsample": [
        0.8,
        1.0
    ],

    "colsample_bytree": [
        0.8,
        1.0
    ]
}

xgb_grid = GridSearchCV(
    estimator=tuning_models[
        "XGBoost"
    ],

    param_grid=xgb_param_grid,

    scoring="neg_root_mean_squared_error",

    cv=5,

    n_jobs=-1,

    verbose=1
)

xgb_grid.fit(
    X_train,
    y_train
)

print(
    "\nBest XGBoost parameters:"
)

print(
    xgb_grid.best_params_
)

Cell 28 — Tuned XGBoost evaluation

# ============================================================
# 27. EVALUATE TUNED XGBOOST
# ============================================================

best_xgb = xgb_grid.best_estimator_

xgb_pred_log = best_xgb.predict(
    X_test
)

xgb_pred = np.expm1(
    xgb_pred_log
)

xgb_pred = np.maximum(
    xgb_pred,
    0
)

xgb_mae = mean_absolute_error(
    y_test_original,
    xgb_pred
)

xgb_rmse = np.sqrt(
    mean_squared_error(
        y_test_original,
        xgb_pred
    )
)

xgb_r2 = r2_score(
    y_test_original,
    xgb_pred
)

print("XGBoost")
print("MAE :", xgb_mae)
print("RMSE:", xgb_rmse)
print("R²  :", xgb_r2)

Cell 29 — Tuned comparison

# ============================================================
# 28. TUNED MODEL COMPARISON
# ============================================================

tuned_results_df = pd.DataFrame([

    {
        "Model": "Random Forest",
        "MAE": rf_mae,
        "RMSE": rf_rmse,
        "R²": rf_r2
    },

    {
        "Model": "Extra Trees",
        "MAE": et_mae,
        "RMSE": et_rmse,
        "R²": et_r2
    },

    {
        "Model": "XGBoost",
        "MAE": xgb_mae,
        "RMSE": xgb_rmse,
        "R²": xgb_r2
    }

])

tuned_results_df = (
    tuned_results_df
    .sort_values(
        "R²",
        ascending=False
    )
    .reset_index(drop=True)
)

display(
    tuned_results_df.style.format({
        "MAE": "{:,.2f}",
        "RMSE": "{:,.2f}",
        "R²": "{:.4f}"
    })
)

Cell 30 — Baseline vs tuned

# ============================================================
# 29. BASELINE VS TUNED
# ============================================================

baseline_selected = (
    baseline_results_df[
        baseline_results_df["Model"].isin([
            "Random Forest",
            "Extra Trees",
            "XGBoost"
        ])
    ][
        ["Model", "MAE", "RMSE", "R²"]
    ]
    .copy()
)

baseline_selected["Stage"] = "Baseline"

tuned_comparison = tuned_results_df.copy()

tuned_comparison["Stage"] = "Tuned"

baseline_vs_tuned = pd.concat(
    [
        baseline_selected,
        tuned_comparison
    ],
    ignore_index=True
)

display(
    baseline_vs_tuned.sort_values(
        ["Model", "Stage"]
    )
)

Cell 31 — Save baseline results

# ============================================================
# 30. SAVE BASELINE RESULTS
# ============================================================

baseline_results_df.to_csv(
    "baseline_log_target_pcf_results.csv",
    index=False
)

print(
    "Saved baseline results."
)

Cell 32 — Save tuned results

# ============================================================
# 31. SAVE TUNED RESULTS
# ============================================================

tuned_results_df.to_csv(
    "tuned_log_target_pcf_results.csv",
    index=False
)

print(
    "Saved tuned results."
)

Cell 33 — Save hyperparameters

# ============================================================
# 32. SAVE BEST HYPERPARAMETERS
# ============================================================

best_parameters_df = pd.DataFrame({

    "Model": [
        "Random Forest",
        "Extra Trees",
        "XGBoost"
    ],

    "Best Parameters": [
        rf_grid.best_params_,
        et_grid.best_params_,
        xgb_grid.best_params_
    ]
})

display(
    best_parameters_df
)

best_parameters_df.to_csv(
    "tuned_log_target_pcf_best_parameters.csv",
    index=False
)

Cell 34 — Experiment summary

# ============================================================
# 33. EXPERIMENT SUMMARY
# ============================================================

print("=" * 80)
print("EXPERIMENT 03 — LOG TARGET PCF")
print("=" * 80)

print(
    "\nDataset rows:",
    len(df_ml)
)

print(
    "Training rows:",
    len(X_train)
)

print(
    "Testing rows:",
    len(X_test)
)

print(
    "\nTarget transformation: log1p(PCF)"
)

print(
    "Product Weight transformation: None"
)

best_baseline = baseline_results_df.iloc[0]

print(
    "\nBest baseline model by R²:"
)

print(
    best_baseline["Model"],
    "→ R² =",
    round(
        best_baseline["R²"],
        4
    )
)

best_tuned = tuned_results_df.iloc[0]

print(
    "\nBest tuned model by R²:"
)

print(
    best_tuned["Model"],
    "→ R² =",
    round(
        best_tuned["R²"],
        4
    )
)

print("\n" + "=" * 80)

Cell 35 — Important experiment note

Don't fill this with guesses. Run the notebook first, then insert the actual values.

## Experiment Summary

This experiment investigated whether logarithmic transformation of the
PCF target improved prediction performance. The transformation was
motivated by the substantial positive skewness of the PCF distribution.

The same 866 observations, 80/20 train-test split, random state,
feature preprocessing, regression models and evaluation metrics used in
Experiments 01 and 02 were retained to support a controlled comparison.

The PCF target was transformed using `log1p(PCF)` before model training.
Product Weight remained in its original representation, subject only to
the established winsorisation procedure. Model predictions were
converted back to the original PCF scale using `expm1()` before
calculating MAE, RMSE and R².

The strongest baseline model achieved an R² of [XX], while the best
tuned model achieved an R² of [XX].

Compared with the raw-target experiment, the log-target transformation
produced [an improvement / a reduction / a mixed effect] in predictive
performance. These results are used to determine whether target
transformation should be retained in the final machine-learning
configuration.

⚠️ One critical point about Notebook 03

There is a subtle issue we need to handle correctly.

In Notebook 03:

y_train = np.log1p(y_train_raw)

means Company target encoding is also being fitted against log-PCF, because:

te.fit_transform(X_train, y_train)

uses the transformed target.

That is logically consistent with a log-target experiment, but it means the experiment changes both the modelling target and the target encoding scale.

I do not want you to silently change this now, because your previous notebooks already use target encoding and we want methodological consistency.

However, when we write the dissertation methodology, we should explicitly document that the target encoding is fitted using the training target representation used by each experiment.

Your three notebooks are now cleanly defined

01_ML_Baseline_Raw_PCF.ipynb
    Raw PCF
    Raw Product Weight
             ↓
02_ML_Log_Product_Weight.ipynb
    Raw PCF
    Log Product Weight
             ↓
03_ML_Log_Target_PCF.ipynb
    Log PCF
    Raw Product Weight

Run Notebook 03 through Cell 21 first and send me the baseline_results_df. We can then check the baseline before you spend time running all three GridSearchCV experiments.

BEACON-01

Sajjad Rahman — Fri, 14 Aug 2026 09:01:06 +0000

A simple story of BEACON FRAMEWORK

What is data quality → why it matters → why checking missing values is not enough → why BEACON uses dimensions → where the eight dimensions came from → why the mapping table is necessary → why eight rather than another number → how the 15 rules operationalise the dimensions → how this ultimately connects to ML.

The important methodological basis already established in your project is that the eight dimensions were synthesised from established data-quality literature/standards and PCF-specific requirements, then filtered using three BEACON criteria: measurability, rule definition and computability. The Feature–Dimension Mapping is explicitly intended to be the conceptual core connecting dataset attributes to rules and automated assessment.

I would not claim that "8 is the Goldilocks zone" or that eight is objectively the universally correct number for all domain datasets. Eight is the selected BEACON set because it provides the required coverage for this PCF dataset while satisfying the project's objective assessment criteria and avoiding dimensions that could not be operationalised reliably

Why Data Quality Matters in Product Carbon Footprint Data:

Understanding the BEACON Framework

An accessible introduction to why carbon data needs more than a simple missing-value check

Imagine making an important decision with unreliable information

Imagine that you are comparing two products because you want to understand their environmental impact.

Product A appears to have a carbon footprint of:

10 kg CO₂e

Product B appears to have:

10,000 kg CO₂e

It would be tempting to conclude that Product B is 1,000 times worse.

But before making that conclusion, we need to ask:

Are the two products measured using the same basis?
Are their weights comparable?
Are the reporting boundaries the same?
Was the carbon footprint calculated using a recognised method?
Is the information complete?
Are the values physically plausible?
Can the reported figures be traced back to a source?
Are some countries or industries over-represented?
Are the descriptions clear enough to understand what the numbers actually represent?

This is the problem that data quality addresses.

For Product Carbon Footprint (PCF) data, this is particularly important because poor-quality information can affect not only reporting and sustainability decisions, but also any machine-learning model trained using that information.

The BEACON framework was developed to provide a systematic way of assessing these issues before PCF data is used for machine learning.

1. What does "data quality" actually mean?

Data quality does not simply mean:

"There are no empty cells."

A dataset can contain no missing values and still be unsuitable for analysis.

Consider four simple examples.

Example 1 — Missing information

Product weight = [missing]

The information is incomplete.

Example 2 — Impossible information

Product weight = -500 kg

The value exists, but it does not make physical sense.

Example 3 — Inconsistent information

One record says:

Country = USA

while another uses:

United States

The two may refer to the same country, but inconsistent representations can affect analysis.

Example 4 — Extreme but legitimate information

A 600-tonne wind turbine can have a very large total PCF.

That does not automatically mean the value is wrong.

This last example is particularly important.

A large PCF should be interpreted alongside the characteristics of the product, because:

[
PCF = Weight \times Carbon\ Intensity
]

Therefore:

A large number is not necessarily a poor-quality number.

This is why data quality requires several perspectives rather than one simple test.

2. Why do we need "dimensions"?

Think about a medical health check.

A doctor does not determine someone's health using only their weight.

Instead, they might consider:

blood pressure,
heart rate,
temperature,
blood tests,
medical history,
symptoms.

Each measure examines a different aspect of health.

Data quality works in a similar way.

A data-quality dimension represents a particular aspect of data health.

For example:

Completeness asks whether important information is present.

while:

Validity asks whether the information follows expected rules.

and:

Plausibility asks whether the values make sense in the real world.

These are different questions.

A dataset could therefore be:

100% complete

but still contain:

invalid values

or:

physically implausible values

or:

poorly documented information

This is why BEACON does not reduce data quality to a single missing-value check.

3. Where did BEACON's eight dimensions come from?

A natural question is:

"Did you just invent these eight dimensions?"

No.

But there is an important distinction.

The eight dimensions are not claimed as a new universal data-quality standard.

Instead, BEACON synthesises and adapts established data-quality concepts to the specific requirements of Product Carbon Footprint data.

The current BEACON methodology draws on established data-quality frameworks and standards including:

Wang and Strong's data-quality framework;
ISO/IEC 25012;
ISO 8000;
DAMA data-quality guidance;
ISO 14067 for Product Carbon Footprints;
the GHG Protocol Product Standard;
and the Carbon Catalogue documentation.

This is important because different frameworks use different terminology and organise data quality differently.

Therefore, BEACON does not simply copy one existing framework.

It asks:

Which quality aspects are necessary for this particular PCF dataset, and which can actually be assessed objectively and automatically?

4. The three BEACON selection criteria

This is one of the strongest parts of your methodology.

A potential quality dimension was included in BEACON only when it satisfied three requirements.

1. Measurability

Can we actually measure the quality aspect?

For example:

How many values are missing?

can be measured.

2. Rule Definition

Can we define a clear rule?

For example:

Product weight must be greater than zero.

is a rule that can be tested.

3. Computability

Can the assessment be performed consistently using the BEACON software?

For example:

weight > 0

can be evaluated automatically.

These criteria make the framework more reproducible and reduce subjective judgement. The same three criteria are already established in your Feature–Dimension Mapping methodology.

5. So what are BEACON's eight dimensions?

The resulting BEACON dimensions are:

Dimension	In simple terms
Completeness	Is the required information there?
Validity	Does the information follow the expected rules?
Consistency	Is information represented consistently?
Plausibility	Does it make sense in the real world?
Traceability	Can we understand where the information came from?
Timeliness	Is the information temporally appropriate?
Interpretability	Can a person understand what the information means?
Representativeness	Does the dataset adequately reflect the products, industries and regions it claims to represent?

These definitions are consistent with the current BEACON methodology.

6. Why does each dimension matter?

1. Completeness

Imagine:

Product Weight = missing

You may not be able to properly interpret the PCF.

Completeness therefore asks:

Do we have the information needed to use the data?

2. Validity

Suppose:

Year = "banana"

or:

Weight = -50 kg

The value exists, but it does not satisfy the expected rules.

Validity asks:

Is the value structurally and logically acceptable?

3. Consistency

Suppose the same concept appears as:

USA
United States
US

or different records use incompatible representations.

Consistency asks:

Are similar pieces of information represented in a compatible way?

4. Plausibility

This is particularly important for PCF data.

Suppose we see:

PCF = 3,700,000 kg CO₂e

The number looks enormous.

But then we discover:

Product = large wind turbine
Weight = 600,000 kg
Carbon intensity ≈ 6.2 kg CO₂e/kg

The large PCF can be explained by the product's physical scale.

Therefore, plausibility is not simply:

"Is this number large?"

It is:

"Does this number make sense given the context and relationships between relevant variables?"

This insight became particularly important during the BEACON development and synthetic-data investigation.

5. Traceability

Imagine a company reports:

PCF = 2,500 kg CO₂e

but provides no information about how the number was obtained.

Can an analyst verify it?

Can another researcher reproduce it?

Traceability therefore asks:

Can the origin or methodology of the information be understood and checked?

For PCF reporting, this is particularly important because methodological information such as the reporting protocol and data source provides context for interpreting the footprint.

6. Timeliness

Carbon information can change over time.

Emission factors, technologies, manufacturing processes and reporting practices may change.

Timeliness therefore asks:

Is the temporal information appropriate for the intended use?

An important BEACON design decision is that the Carbon Catalogue is a historical/static dataset, so the freshness rule is retained in the rule library but is not meaningfully executed against the historical benchmark.

This is a good example of why not every possible quality check should automatically be applied to every dataset.

7. Interpretability

A number without context can be difficult to understand.

For example:

PCF = 500

What does 500 mean?

What product?

What functional unit?

What reporting boundary?

What methodology?

Interpretability therefore asks:

Can a human understand what the data represents?

8. Representativeness

Imagine a dataset contains products almost entirely from one country and one industry.

The dataset may be complete and internally consistent.

But can we confidently use it to represent a much broader population?

Representativeness asks:

Does the dataset provide adequate coverage of the products, industries and geographical contexts relevant to the analysis?

This is especially important for machine learning because a model can learn the characteristics of the data it receives.

7. Why not use 15 dimensions?

Another reasonable question is:

"If ISO has many characteristics, why did BEACON only use eight?"

The answer is not that eight is mathematically optimal.

Instead, BEACON uses eight because the framework is intended to provide sufficient coverage without introducing dimensions that cannot be meaningfully and reproducibly assessed for the Carbon Catalogue.

Some concepts from larger data-quality models overlap when operationalised for this dataset.

For example, several detailed technical characteristics can be represented through broader operational categories such as:

Validity
Consistency
Plausibility

rather than creating a separate BEACON dimension for every possible characteristic.

At the other extreme, using only a few broad dimensions could hide important PCF-specific concerns such as:

Traceability
Interpretability
Representativeness

Therefore, the selection was guided by:

Established research
       +
PCF-specific requirements
       +
Carbon Catalogue characteristics
       +
Measurability
       +
Rule definition
       +
Computability

The resulting eight dimensions are therefore a purpose-built operational set, not a claim that all data-quality research should use exactly eight dimensions.

That distinction is important for your dissertation.

8. Why can't we just apply all eight dimensions to every column?

This leads to one of the most important ideas in BEACON:

Feature–Dimension Mapping

Consider:

Product Weight

It makes sense to ask:

Is it complete?
Is it valid?
Is it consistent?
Is it plausible?

But it makes much less sense to apply exactly the same questions to:

Product Name

For example, you can assess whether a product name is present and interpretable.

But asking whether a product name is "physically plausible" is meaningless.

Therefore:

Not every dimension applies to every feature.

BEACON explicitly maps each dataset feature to the dimensions that are relevant to it.

9. Why is the mapping table necessary?

The mapping table is not just documentation.

It is the bridge between the concept and the software.

Without the mapping, we have:

8 dimensions

and:

25 dataset features

but no systematic explanation of:

Which quality check applies to which feature?

The mapping solves this.

Conceptually:

Product Weight
      ↓
Completeness
Validity
Consistency
Plausibility
      ↓
Relevant validation rules
      ↓
Automated checks

while:

Product Name
      ↓
Completeness
Interpretability
      ↓
Relevant validation rules
      ↓
Automated checks

This makes BEACON's decisions transparent.

The current project documentation explicitly describes the mapping as the conceptual core connecting features, dimensions, validation rules, metrics and automated assessment.

10. From the mapping table to 15 rules

The mapping tells us:

What should be assessed?

The rules tell us:

How should it be assessed?

For example:

Feature
   ↓
Product Weight
   ↓
Plausibility
   ↓
Range Validation
   ↓
Is weight > 0?

Another example:

Feature
   ↓
PCF + Weight + Carbon Intensity
   ↓
Plausibility
   ↓
Cross-Field Validation
   ↓
Is PCF consistent with Weight × CI?

This is why BEACON does not contain 25 × 8 = 200 independent checks.

Instead, it uses reusable generic rule patterns.

The current methodology defines 15 rule patterns, of which 14 are executed for the historical Carbon Catalogue because R12 Freshness is not applicable in that context.

11. What are the 15 rules actually doing?

At a high level:

Rule	What it asks
R01 Missing Value Check	Is required information present?
R02 Data Type Validation	Is the data stored in the expected form?
R03 Domain Validation	Are values within allowed domains?
R04 Controlled Vocabulary	Are categories represented consistently?
R05 Measurement Unit	Are units handled consistently?
R06 Range Validation	Are numerical values within reasonable ranges?
R07 Cross-Field Validation	Do related fields make sense together?
R08 Identifier Uniqueness	Are identifiers unique where required?
R09 Provenance Verification	Is supporting source/method information available?
R10 Metadata Availability	Is relevant metadata present?
R11 Date Validation	Are reporting dates valid?
R12 Freshness Assessment	Is the information sufficiently current?
R13 Documentation Check	Is descriptive information adequately documented?
R14 Category Coverage	Are relevant categories represented?
R15 Distribution Assessment	Is the dataset distribution sufficiently representative?

These are generic rule patterns, not 15 completely independent algorithms. Each reusable validation function operates over the features identified by the Feature–Dimension Mapping.

12. Why does BEACON use rules instead of simply asking an analyst to inspect the data?

Because manual inspection is difficult to reproduce.

Imagine two researchers inspecting the same 866 records.

Researcher A might say:

"This looks suspicious."

Researcher B might say:

"This looks acceptable."

A rule-based framework attempts to make the decision process explicit.

For example:

IF Product Weight <= 0
THEN fail plausibility/validity check

or:

IF required field is missing
THEN fail completeness check

or:

IF PCF is inconsistent with related physical/carbon values
THEN flag for investigation

This makes the assessment:

repeatable,
auditable,
explainable,
automatable.

13. Does BEACON automatically "fix" everything?

No — and this is an important design principle.

Finding a problem and knowing the correct replacement value are two different things.

Suppose BEACON finds:

Country = "U.S.A."

and the controlled vocabulary uses:

United States

Standardisation may be straightforward.

But suppose BEACON finds:

PCF = 87,589 kg CO₂e

and determines that it is extreme.

It does not mean that BEACON should invent a new PCF value.

The correct action may be:

Investigate

or:

Flag for review

rather than:

Automatically change the number

This distinction protects the integrity of the original evidence.

14. This is particularly important for PCF data

During the BEACON development, an important issue emerged.

A large PCF value may initially appear suspicious.

For example:

600,000 kg product
+
6.2 kg CO₂e/kg
=
approximately 3.7 million kg CO₂e

The absolute PCF is very large.

But the value is explainable through the product's physical scale.

Therefore, BEACON should not simply say:

"Large PCF = bad."

Instead:

"Does the PCF make sense in relation to the relevant product and carbon characteristics?"

This is the role of domain-aware plausibility and cross-field validation.

15. What happens after all the rules run?

BEACON produces evidence at several levels.

Individual validation
        ↓
Rule score
        ↓
Dimension score
        ↓
Overall BEACON Quality Score

For example:

R06 Range Validation
        ↓
62.63%

R07 Cross-Field Validation
        ↓
84.99%

These are then aggregated into dimension-level scores and ultimately the overall score.

Your current implementation produces a Rule Evaluation Matrix, dimension scores, and an Issue Register.

16. Why have an overall score if the individual rules are more informative?

The overall score provides a summary.

Imagine a manager wants a quick answer:

"How strong is this dataset overall?"

A single score is useful.

But an overall score alone is insufficient for diagnosis.

Therefore BEACON deliberately provides both:

Overall score
      +
Dimension scores
      +
Rule scores
      +
Issue-level evidence

Think of it like a medical check-up again.

You might receive:

Overall health assessment: Good

but the doctor still tells you:

Blood pressure needs attention.

The summary and the detailed diagnosis serve different purposes.

17. Why is the mapping table so important to the whole architecture?

Because it creates a chain of accountability:

Research literature
       ↓
Quality dimension
       ↓
BEACON feature
       ↓
Feature–Dimension Mapping
       ↓
Validation rule
       ↓
Metric
       ↓
Python implementation
       ↓
Evidence
       ↓
Quality score

Every important decision can therefore be traced backwards.

For example, an examiner can ask:

"Why did you assess Product Weight for plausibility?"

You can answer:

Product Weight
↓
Physical characteristic
↓
Plausibility applicable
↓
Range rule
↓
Physical/empirical threshold
↓
Automated validation

That is far stronger than saying:

"We thought it would be useful."

18. Why eight dimensions rather than five?

With only five broad dimensions, some PCF-specific concerns could become hidden.

For example:

Traceability

could disappear inside a broad "quality" category.

Likewise:

Representativeness

is important when a dataset is used for machine learning across industries and countries.

BEACON therefore keeps these concepts visible because they have different practical implications.

19. Why not twelve?

The opposite problem is fragmentation.

If every subtle characteristic became its own dimension, the framework could become:

harder to implement
harder to explain
harder to score
harder to maintain

and potentially contain overlapping concepts.

BEACON therefore groups related quality concerns where they can be assessed through a common operational concept.

Again, the important claim is not:

"Eight is universally optimal."

The defensible claim is:

"Eight dimensions provided the selected operational coverage required for the Carbon Catalogue while meeting the predefined measurability, rule-definition and computability criteria."

That is the wording I would use in your dissertation.

20. What does this mean for machine learning?

This is ultimately why BEACON exists in your project.

Machine learning models learn patterns from the data they receive.

If the input contains:

missing values
inconsistent categories
invalid values
poorly documented observations
unrepresentative distributions

then the model may learn patterns that do not generalise well.

BEACON therefore sits before the ML stage:

PCF Data
   ↓
BEACON
   ↓
Quality assessment
   ↓
Quality treatment
   ↓
Reassessment
   ↓
ML-ready dataset
   ↓
Machine Learning

Your research then asks whether this quality-assurance process actually changes predictive performance.

That is where your RQ3 becomes important.

21. The final BEACON idea in one picture

                  PRODUCT CARBON DATA
                          │
                          ▼
                 "Can we trust this?"
                          │
          ┌───────────────┼────────────────┐
          │               │                │
          ▼               ▼                ▼
     Completeness      Validity       Consistency
          │               │                │
          ├───────────────┼────────────────┤
          │               │                │
          ▼               ▼                ▼
    Plausibility     Traceability     Timeliness
          │               │                │
          └───────────────┼────────────────┘
                          │
                 Interpretability
                          │
                 Representativeness
                          │
                          ▼
                FEATURE–DIMENSION
                     MAPPING
                          │
                          ▼
                    15 RULES
                          │
                          ▼
                 AUTOMATED CHECKS
                          │
                          ▼
                 QUALITY EVIDENCE
                          │
              ┌───────────┴───────────┐
              ▼                       ▼
          Score / Diagnose        Issue Register
                                      │
                                      ▼
                              Justified Treatment
                                      │
                                      ▼
                                  Reassess
                                      │
                                      ▼
                              ML EVALUATION

22. The key message for a non-technical reader

If someone remembers only one thing from this article, it should be this:

Good data is not simply data with no empty cells.

For Product Carbon Footprints, good-quality data should be:

complete enough to use, valid enough to trust, consistent enough to compare, plausible enough to make sense, traceable enough to verify, timely enough for its purpose, understandable enough to interpret, and representative enough for the population being studied.

BEACON brings these perspectives together into a structured assessment process.

The eight dimensions are not claimed as a universal replacement for existing data-quality standards. They are a PCF-focused synthesis, selected from established research and standards and filtered according to whether they can be measured, expressed as rules and implemented computationally.

And the Feature–Dimension Mapping is the key bridge: it prevents BEACON from applying arbitrary checks to every column and instead establishes which quality questions are meaningful for each type of PCF information.

Understanding SQL Server Platform Choices

Sajjad Rahman — Sun, 02 Aug 2026 02:01:30 +0000

Understand the difference between IaaS and PaaS and know when each SQL platform is used.

Why Should I Learn This?

Imagine you have designed a database.

The next question is:

"Where will I run this database?"

Microsoft offers several SQL platforms, and each platform provides a different type of control and responsibility.

This blog is not about writing SQL queries. It is about understanding where SQL Server runs and who manages it.

First, Understand Cloud Computing

Suppose you want to open a restaurant. There are two choices.

Option 1 — Build Everything Yourself

You buy:

Land
Building
Kitchen
Electricity
Security
Furniture

You maintain everything yourself. This is similar to:

Infrastructure as a Service (IaaS)

You have maximum control. You also have maximum responsibility.

Option 2 — Rent a Fully Managed Restaurant

The owner provides:

Building
Electricity
Security
Maintenance

You only cook food. This is similar to:

Platform as a Service (PaaS)

You focus on your application instead of infrastructure.

Understanding the Diagram

The diagram has two axes.

Horizontal Axis

On-Premises ------------------------> Cloud
(More Control)                   (Less Administration)

As you move to the right:

Microsoft manages more.
You manage less.

Vertical Axis

Higher Cost
↑
|
|
Lower Cost

Generally:

More dedicated resources = Higher cost
More shared cloud resources = Lower cost

Platform 1 — SQL Server (On-Premises)

Your Company
↓
Own Physical Server
↓
Install Windows
↓
Install SQL Server
↓
Create Database

Who manages everything?

You.

Responsibilities:

Buy hardware
Install operating system
Install SQL Server
Configure networking
Configure security
Backup database
Patch SQL Server
Replace failed hardware

Advantages:

Maximum control
Full customization

Disadvantages:

Expensive
Time-consuming

- Requires database administrators (DBAs)

Platform 2 — SQL Server on Azure Virtual Machine (IaaS)

Microsoft gives you:

A virtual machine

You install:

Windows or Linux
SQL Server

Responsibilities:

✔ Configure SQL Server
✔ Patch SQL Server
✔ Configure backups
✔ Manage security
✔ Performance tuning

Microsoft manages

Physical servers
Azure networking
Storage
Data center

Think of this as:

Renting a computer instead of buying one.

Platform 3 — Azure SQL Database (PaaS)

Microsoft already provides:

SQL Server
Operating System
Updates
Automatic Backups
High Availability
Hardware
Networking

You only create:

Databases
Tables
Indexes
Views
Stored Procedures
Constraints
Data

Your focus becomes:

Database Design

instead of

Server Administration

Platform 4 — Azure SQL Managed Instance

Imagine your company already has a large SQL Server application. It uses advanced SQL Server features. Moving directly to Azure SQL Database may require application changes.

Azure SQL Managed Instance solves this problem.

It provides:

Almost complete SQL Server compatibility

while Microsoft still manages:

Infrastructure
Updates
Backups
High Availability

Think of it as:

SQL Server in the cloud with minimal application changes.

Comparing IaaS and PaaS

Feature	IaaS	PaaS
Physical Server	Microsoft	Microsoft
Virtual Machine	You	Microsoft
Operating System	You	Microsoft
SQL Server Installation	You	Microsoft
SQL Updates	You	Microsoft
Backups	You	Microsoft
Database Design	You	You
Tables	You	You
Views	You	You
Stored Procedures	You	You
Data	You	You

Easy Way to Remember

IaaS

"I manage the server."

PaaS

"I manage the database."

Real-Life Example

Imagine you work as a database developer.

Your manager says:

"Don't waste time installing Windows or patching SQL Server.
Just design the database."

Which platform should you choose?

Answer: Azure SQL Database (PaaS)

DP-800 Exam Tips

"Which platform reduces infrastructure management?"
Answer: Azure SQL Database (PaaS)

"Which platform gives maximum control over SQL Server?"
Answer: SQL Server on Azure Virtual Machine (IaaS)

Summary

SQL Server (On-Premises) → You manage everything.
SQL Server on Azure VM → Microsoft manages hardware, you manage the VM and SQL Server.
Azure SQL Database → Microsoft manages the infrastructure; you focus on database development.
Azure SQL Managed Instance → Similar to Azure SQL Database, but designed for easier migration of existing SQL Server applications.

Key Takeaways

✔ IaaS = More control, more administration.
✔ PaaS = Less administration, more focus on database development.
✔ Microsoft manages the infrastructure in PaaS.
✔ Developers mainly work with tables, indexes, views, stored procedures, and data.

Now moving forward.........................

"The Hyperscale service tier eliminates many of the practical limitations traditionally associated with cloud databases..."

A beginner like me immediately asks:

What is a service tier?
What is Hyperscale?
What is a node?
What is storage architecture?
What is a replica?
What is a read-intensive workload?

Microsoft explains Hyperscale, but assumes you already know the other five concepts.

Before We Begin. Remember this:

A database has two parts.

Part 1 : Infrastructure

Hardware
Network
Operating System
SQL Server installation
Updates
Backups

-Part 2 : Database

Tables
Views
Indexes
Stored Procedures
Data

The difference between Microsoft's SQL platforms is simply:

Who manages Part 1?

Azure SQL Database

What is Azure SQL Database?

Azure SQL Database is Microsoft's fully managed cloud database
You do NOT install SQL Server
You do NOT patch Windows.
You do NOT replace failed hardware.

Microsoft does all of that. You simply create databases and write SQL.

Think of it as: Google Docs instead of Installing Microsoft Word.

What does Fully Managed mean?

Microsoft manages:
✔ Hardware
✔ Network
✔ Operating System
✔ SQL Server Updates
✔ Automatic Backup
✔ High Availability

You manage
✔ Tables
✔ Data
✔ Views
✔ Stored Procedures
✔ Indexes

What is a Service Tier?

Imagine buying a mobile internet package.

Basic Package
↓
Professional Package
↓
Unlimited Package

Azure SQL Database also has different plans. These are called

Service Tiers

Each tier gives different performance.

Hyperscale

Microsoft writes

Hyperscale removes storage limits.

Let's understand. Suppose your database grows like this.

10 GB
↓
100 GB
↓
2 TB
↓
20 TB

Many traditional databases eventually reach storage or performance limits. Hyperscale is designed to keep growing without needing to redesign your database.

Benefits

✔ Storage automatically grows

✔ No predefined maximum size

✔ Pay only for what you use

✔ Better performance for very large databases

What is a Replica?

Imagine 1,000 people want to read the same book. One copy of the book becomes crowded. Instead, the library creates five copies. Now everyone can read simultaneously.

Those extra copies are called Replicas They mainly help with Read Operations

Read Workload

SELECT

Reading data.

Write Workload

INSERT
UPDATE
DELETE

Changing data.

Read-intensive workload

Most users are reading data rather than changing it.

Example News website: Thousands of people reading Very few people editing.

Serverless

Suppose nobody connects to your database.

Traditional Server

Running

↓
Consumes CPU
↓
Costs Money

Serverless

Nobody connected?
↓
Automatically Pause
↓
No Compute Charges
↓
Someone connects
↓
Automatically Resume

This reduces costs.

Why does Microsoft mention Retry Logic?

When a paused database wakes up, it needs a few seconds. Your application should automatically retry the connection instead of immediately showing an error.

Automatic Tuning

Normally, a DBA analyzes slow queries. Creates indexes. Improves performance.

Azure SQL Database can do much of this automatically.

Think of it as

Database
↓
Detect Slow Query
↓
Recommend Index
↓
Sometimes Create It Automatically

High Availability

Suppose one server fails. Microsoft automatically switches your database to another server. Users usually don't notice.Microsoft guarantees 99.99% availability.

Azure SQL Managed Instance

Instead of saying

Near 100% compatibility...

Imagine your company already has a SQL Server application.

It uses

SQL Server Agent
Linked Servers
Database Mail
Service Broker

Moving directly to Azure SQL Database may require code changes. Azure SQL Managed Instance supports most SQL Server features. It lets companies move to Azure with minimal application changes.

SQL Server on Azure VM

Think of Azure VM as renting a computer.

Microsoft gives you:

✔ Virtual Machine

You install:

✔ Windows

✔ SQL Server

✔ Configure Everything

Maximum Control

Maximum Responsibility

SQL Database in Microsoft Fabric

This is where most tutorials fail.

I'd explain it like this:

Traditional Company

Application
↓
SQL Database
↓
ETL
↓
Data Warehouse
↓
Power BI

Many separate systems.

Fabric

Application
↓
SQL Database
↓
Automatic Mirroring
↓
OneLake
↓
Analytics
↓
Power BI

No manual ETL. Everything stays synchronized automatically.

OneLake

Think of OneLake as Google Drive for enterprise data. Every Fabric service shares the same storage.

Delta Parquet

This is NOT another database. It is a highly optimized file format used for analytics. Fabric automatically creates these files from your SQL tables.

Why?

Analytics queries can be extremely heavy. Instead of querying the live production database, Fabric queries the Delta Parquet copy. Your application remains fast.

Summary Table

Platform	Best Choice When
SQL Server	Full control on your own servers
SQL Server on Azure VM	Need OS/SQL customization
Azure SQL Database	Building modern cloud applications
Azure SQL Managed Instance	Migrating existing SQL Server apps
SQL Database in Fabric	Need OLTP + analytics + AI together

Reference : understanding-platform-choices

Build your skills in Microsoft Fabric, SQL, Power BI, or AI

Sajjad Rahman — Sat, 25 Jul 2026 14:03:17 +0000

Microsoft Fabric Data Days 2026 is a great opportunity to access free learning modules, live sessions, study groups, and certification preparation. Eligible participants can also work toward a free Microsoft certification exam voucher by completing the event requirements.

If you're interested in data, analytics, AI, or cloud technologies, don't miss this opportunity. Start learning today!

Join

DataDays #MicrosoftFabric #SQL #AI #MicrosoftLearn #DP600 #DP700 #DP800 #Learning

A. They always produce nonlinear decision boundaries
B. They use a hyperplane as a decision surface
C. They maximise classification accuracy directly
D. They ignore input features

✅ Answer: B
👉 From PDF: “The decision surface is a hyperplane.”

Q2. Sum of Squared Errors (SSE)

What is the goal when training a linear model?

A. Maximise likelihood
B. Minimise SSE
C. Maximise distance between points
D. Minimise number of features

✅ Answer: B
👉 Linear models minimise SSE

Q3. Effect of Outliers

What is true about samples far from the hyperplane?

A. They have no effect
B. They are ignored
C. They have a stronger effect on the model
D. They reduce variance

✅ Answer: C
👉 “Samples far away from hyperplane have a stronger effect.”

Q4. Linear Separability

When are data points typically linearly separable?

A. When ( n \gg d )
B. When ( n \le d + 1 )
C. Always
D. Never

✅ Answer: B
👉 From PDF: separable when ( n \le d + 1 )

Q5. KNN Prediction

What does KNN use to classify a new data point?

A. Mean of all points
B. Nearest neighbours
C. Gradient descent
D. Covariance matrix

✅ Answer: B
👉 Based on nearest neighbours

Q6. Role of K in KNN

What is the effect of choosing a small K?

A. Smooth decision boundary
B. Linear boundary
C. Highly flexible / irregular boundary
D. No effect

✅ Answer: C
👉 Small K → complex, irregular decision surfaces

Q7. Distance Metrics

Which is NOT mentioned as a valid distance in KNN?

A. Euclidean distance
B. Manhattan distance
C. Hamming distance
D. Fourier distance

✅ Answer: D
👉 Others are listed in PDF

Q8. Curse of Dimensionality

What happens as dimensionality increases?

A. Less data is needed
B. Nearest neighbours become more meaningful
C. Required training data increases exponentially
D. Models become simpler

✅ Answer: C
👉 “Required amount of training examples increases exponentially”

Q9. Cross-Validation

What is the purpose of cross-validation?

A. Increase training error
B. Evaluate model performance on unseen data
C. Remove noise
D. Reduce dimensionality

✅ Answer: B
👉 Used for validation/testing

Q10. Accuracy Definition

Accuracy is defined as:

A. TP / FP
B. (TP + TN) / Total
C. FN / Total
D. TP / (TP + FN)

✅ Answer: B
👉 Standard accuracy formula

📘 Master Note: The Hidden Mechanics of PCA & ICA

Sajjad Rahman — Sat, 04 Apr 2026 09:17:13 +0000

Understanding Whitening, SVD, and the Math that Powers Dimensionality Reduction.

Part 1: The Big Picture (Intuition)

Before diving into complex algorithms like ICA, we need to understand the "behind-the-scenes" heroes that prepare and decompose our data.

1. Whitening: The Essential Pre-step for ICA

Whitening prepares your data so that all variables are uncorrelated and have equal variance.

The Goal: Transform the data into a "decorrelated, equal-variance" form.
💡 The Intuition: Imagine your data looks like a stretched, tilted "egg" (oval cloud).
- After PCA: The egg is rotated straight.
- After Whitening: The egg becomes a perfect sphere.
Why bother? ICA looks for independent signals. If data is already "spherical," ICA doesn't get distracted by the width or tilt of the data; it focuses entirely on finding non-Gaussian independence.

2. SVD: The Practical Engine of PCA

SVD is a mathematical powerhouse that decomposes any matrix $X$ into three parts:

X = U Σ V^{T}

$U$: Directions in data space.
$\Sigma$: The strengths (importance) of each direction.
$V$: The directions of the features (The Principal Components).

Part 2: The Mathematical Engine

How do we actually move from raw data to a "white" sphere or a PCA result?

1. The Whitening Transformation

If $X$ is your original centered data, we use the Eigenvalues ($D$) and Eigenvectors ($E$) to transform it:

X_{w hi t e} = E D^{- 1/2} E^{T} X

The logic behind the math:

$E^T$: Rotates the data (PCA).
$D^{-1/2}$: The "Magic Step." It scales every axis by its inverse standard deviation. It shrinks long axes and stretches short ones until they are equal.

2. The SVD ↔ PCA Connection

You can reach Principal Components via two paths, but they lead to the same destination:

Path A (Classical PCA): Find Eigenvectors of the Covariance Matrix: $C = V Λ V^{T}$
Path B (Modern SVD): Decompose $X$ directly: $X = U Σ V^{T}$

The "Aha!" Moment:
The $V$ in SVD is identical to the $V$ (Eigenvectors) in PCA. The Singular Values ($\sigma$) are the square roots of the Eigenvalues ($\lambda$):

λ_{i} = \frac{σ _{i 2}}{n}

Part 3: Performance & Comparisons

Why use SVD instead of Classical PCA?

In real-world data science, SVD is the industry standard for computing PCA.

Feature	Covariance (Classical)	SVD (Modern)
Memory	Requires $XX^T$ (can be massive)	Works directly on $X$
Precision	Squaring numbers loses small details	Keeps high numerical precision
Stability	Prone to rounding errors	Highly stable and robust

🧠 Final Logic Map (Summary)

Step 1: Use SVD to find the "skeleton" (Principal Components) of your data efficiently.
Step 2: Apply Whitening to turn your data cloud into a perfect sphere.
Step 3: Run ICA on that sphere to find hidden, independent signals.

🔥 Quick Memory:

Whitening: "Make it a sphere before ICA."

SVD: "The efficient engine that makes PCA work in the real world."

📘 The Science of Un-Mixing Data (PCA & ICA)

Sajjad Rahman — Sat, 04 Apr 2026 09:09:50 +0000

Part 1: The Math Toolbox (Prerequisites)

Before we can understand PCA and ICA, we need to understand the tools they use. Think of these as the "rules of the game" for handling data.

🔹 Basic Concepts

The Matrix (The Table): Data is organized into a matrix, which is just a giant grid of numbers. The columns usually represent different types of measurements (like different microphones or cameras), and the rows represent each specific moment in time we recorded.
The Vector (The Arrow): A single row or column from that table is called a vector. Mathematically, a vector is like an arrow pointing to a specific spot in a multi-dimensional space.
Inner Product (The Shadow): This is a way to multiply two vectors together. It tells us how much one vector "overlaps" with another. We use this to project our data onto new axes to see it from a better angle.
Basis Vectors (The Directions): These are the "original" directions we use to measure things, like the X and Y axes on a graph.

🔹 Statistical Concepts

Covariance (Redundancy): This measures how much two measurements "change together". If Measurement A always goes up when Measurement B goes up, they are redundant (highly correlated), meaning we don't really need both.

👉 Covariance Equation (Added, not replacing anything):

cov (A, B) = \frac{1}{n} \sum a_{i} b_{i}

Gaussian Distribution (The Bell Curve): This is a smooth, bell-shaped curve that represents "randomness" or "noise". The Central Limit Theorem says that if you mix many different signals together, the result will always look like a Gaussian bell curve.
Kurtosis (The Peakedness): This is a math score that measures how "sharp" or "peaked" a distribution of numbers is. A high kurtosis means the data has a sharp point, while a Gaussian curve has a kurtosis of zero.

Part 2: PCA (Principal Component Analysis)

The Goal

The Goal: To simplify a giant pile of data by finding the "best angle" to look at it.

How it Works

How it works: PCA looks at the Covariance Matrix of the data to find where the measurements are repeating each other.
Eigenvectors and Eigenvalues: PCA calculates special directions called eigenvectors. The "largest" eigenvector points in the direction where the most "action" (variance) is happening.

Core Equation (Added)

P = D \cdot E

(D): data matrix
(E): eigenvectors of covariance
(P): principal components

👉 Also:

Covariance becomes diagonal after PCA

Key Properties

Dimensionality Reduction: By ignoring the tiny eigenvectors (which usually represent noise) and keeping only the big ones, we can make a huge data set much smaller without losing the important stuff.
The Rule of Orthogonality: In PCA, the new axes we find must always be orthogonal—which is a fancy way of saying they must be at 90-degree right angles to each other.

Algorithm Steps (Added Section)

PCA Steps

Center data (mean = 0)
Compute covariance matrix
Find eigenvectors
Project data

Why PCA Can Fail (Added Section)

👉 PCA fails when:

Data is non-linear
Or variance ≠ true structure

Example:

Ferris wheel → PCA cannot find circular motion

💡 Add this line:

PCA only captures linear structure, ICA can handle more complex separation.

Part 3: ICA (Independent Component Analysis)

The Goal

The Goal: To solve the "Cocktail Party Problem"—taking a messy mixture of signals and separating them into their original, clear sources.

How it Works

Blind Source Separation: ICA is used when we have mixtures (like two microphones recording two people) but we don't know exactly how they were mixed.
The Weight Matrix ($W$): ICA tries to find a mathematical "unmixing" tool called a weight matrix. When we multiply our messy data by this matrix, the original signals should pop out.

Core Equation (Added)

Y = X \cdot W

(X): mixed signals
(W): unmixing matrix
(Y): independent sources

👉 This is THE most important ICA equation

Key Concepts

The Search for Independence: ICA is stricter than PCA. It doesn't just want the data to be "not repeating"; it wants the signals to be statistically independent, meaning what happens in one signal tells you absolutely nothing about the other.

Non-Gaussianity (The Secret Trick): Because mixed-up signals look like smooth bell curves, ICA rotates the data until it finds the directions with maximum kurtosis (the most peaked shapes). A sharp peak usually means you've found a pure, unmixed source.

👉 Improved understanding:

Mixtures → Gaussian (Central Limit Theorem)
Sources → non-Gaussian (peaked)

💡 Key idea:

ICA works because mixing makes data Gaussian, so unmixing looks for non-Gaussian signals

Flexibility: Unlike PCA, the axes in ICA do not have to be at right angles. They can point in any direction needed to find the sources.

Algorithm Steps (Added Section)

ICA Steps

Center + whiten data
Initialize weights
Maximize non-Gaussianity
Iterate until convergence

Part 4: PCA vs ICA (Added Comparison Table)

Feature	PCA	ICA
Goal	Max variance	Independence
Output	Uncorrelated	Independent
Axes	Orthogonal	Not required
Uses	Compression	Signal separation
Assumption	Gaussian OK	Non-Gaussian needed

👉 Lecture explicitly says:

PCA → decorrelation
ICA → independence (stronger)

Part 5: Why do we use these?

These tools are used in many cool ways:

Fetal Heart Monitoring: Separating a baby's tiny heartbeat from the mother's much louder heartbeat.
EEG (Brain Waves): Removing "trash" signals like eye blinks or heartbeats from recordings of brain activity.
fMRI (Brain Imaging): Finding which specific parts of the brain are working together during a task.
Computer Vision: Understanding how our eyes and brain recognize edges and shapes in the world around us.

Stanford is Teaching the World to Code for Free!

Sajjad Rahman — Wed, 25 Mar 2026 12:57:35 +0000

Learn Python from Stanford ** 🚀

This isn't just a video course; it is a structured journey where you learn to build logic step-by-step with Chris and Mehran. If you want to stop watching tutorials and start actually coding, here is why you should join Stanford’s Code in Place:

What Makes This Course Unique?

The Karel Phase: You don't start with scary syntax. Instead, you command "Karel," a tiny robot, to master logic, loops, and problem-solving foundations.
Python Mastery: The curriculum moves smoothly from basic Print and Input to advanced topics like control flow and graphics.
Creative Freedom: You get to build your own projects, like a "Console" or the "Game of Nimm," giving you the confidence to create from scratch.
Data & Beyond: You will master real-world techniques involving Lists, Dictionaries, and Graphics.

The Best Part? The Community!

You aren't learning alone. You are supported by a massive core team and amazing section leaders who help you at every step of the way.

Join the Movement!

If you’re ready to take your first real step into programming, apply today. Use the link below to start your application:

👉 Apply Now: https://codeinplace.stanford.edu/public/join/cip6?r=ngqms6

Let’s learn and build together!

Understanding cmd /c and cmd /k

Sajjad Rahman — Wed, 07 Jan 2026 17:33:48 +0000

When I started learning Command Prompt, I was very confused about cmd /c and cmd /k. Even after running the commands, everything looked the same, so I thought nothing was executed. If you feel the same, this post is for you.

First, understand this important point

You are already inside a Command Prompt when you see something like:

I:\ethical-test>

This is your parent CMD.

What actually happens when you run `cmd /k dir`

Command:

cmd /k dir

What it does step by step:

Your current CMD starts a new CMD (child CMD)
The new CMD executes the dir command
Because of /k, the new CMD stays open
Since the path and prompt look the same, it appears unchanged
In reality, you are now inside the child CMD

You can prove this by typing:

exit

This will take you back to the parent CMD.

What happens when you run `cmd /c echo hello`

Command:

cmd /c echo hello

Steps:

A new CMD opens
It runs echo hello
Prints hello
Immediately closes the new CMD
Control returns to the parent CMD

That’s why you only see the output and nothing else.

Why this feels confusing

You are running CMD inside CMD
Windows does not show any visual difference
Same path, same prompt, same window
Execution happens very fast

So it looks like nothing changed, but it actually did.

Simple rule to remember

cmd → creates a new Command Prompt
/k → run command and keep CMD open
/c → run command and close CMD immediately

Final takeaway

cmd /c and cmd /k do work correctly.
The confusion happens because beginners usually run them inside an already opened CMD, so the difference is not visually obvious.

If you understand this, most CMD-related confusion disappears.

Windows CMD – File & Directory Management

Sajjad Rahman — Wed, 07 Jan 2026 16:42:37 +0000

Windows CMD – File & Directory Management

1️⃣ Directory Basics (`dir`)

Show files and folders

dir

Important entries

. → current directory
.. → parent directory
<DIR> → folder
File size shown in bytes

2️⃣ Creating Files with `echo`

Create a file with text

echo "sajjad" > samina.txt

Read file content

type samina.txt

Output

"sajjad"

⚠ Why quotes appear?

Because quotes are literal characters inside echo.

✔ To avoid quotes:

echo sajjad > samina.txt

3️⃣ Creating an Empty (Blank) File – IMPORTANT CONCEPT

Command used

echo 2>err.txt

❓ Why is `2` used here?

This is error stream redirection.

Stream	Meaning
`0`	Standard Input
`1`	Standard Output
`2`	Standard Error

What actually happens?

echo produces no error
Error stream (2) is empty
Empty stream redirected → file created with 0 bytes

✔ This is a CMD trick to create an empty file.

Better & common methods

type nul > file.txt

echo. > file.txt

4️⃣ Deleting Files (`del`)

del na.txt

Check after deletion

type na.txt

Output

The system cannot find the file specified.

✔ File deleted successfully

5️⃣ Renaming Files (`rename`)

rename err.txt error.txt

✔ File name changes only, content remains same

6️⃣ Creating & Moving Directories

Create directory

mkdir samsaj

Move file into directory

move samina.txt samsaj

Change directory

cd samsaj

Go back

cd ..

7️⃣ Path Symbols Explained

Symbol	Meaning
`.`	Current directory
`..`	Parent directory
`\`	Windows path separator

Example:

move samina.txt .\samsaj

8️⃣ Copying Files (`copy`)

copy samina.txt sajjad.txt

✔ Creates duplicate file with same content

9️⃣ Removing Directories (`rmdir`) – VERY IMPORTANT

Delete empty directory

rmdir sam

❌ If directory is NOT empty

The directory is not empty.

✔ Correct way to delete directory with files

rmdir /S sam

/S → deletes all files + subdirectories
Confirmation required

❌ Wrong commands (do not work):

rmdir -S sam
rmdir \S sam

🔟 Comparing Files (`fc`)

fc sajjad.txt samina.txt

Output meaning

Shows line-by-line differences
Useful for file integrity checking

1️⃣1️⃣ Symbolic Links (Soft Links) – `mklink`

⚠ CMD must be Run as Administrator

Create symbolic link

mklink linkfile file3.txt

Verify

dir

Output

<SYMLINK> linkfile [file3.txt]

✔ Both files point to same data
✔ Reading/writing link affects original file

1️⃣2️⃣ Searching Files with `dir`

Search specific file recursively

dir /s torjan.txt

Search all `.txt` files

dir /s *.txt

1️⃣3️⃣ Finding Files using `forfiles`

Command

forfiles /P I:\ /S /M torjan.txt /C "cmd /c echo @PATH"

Flags explained

Flag	Meaning
`/P`	Starting directory
`/S`	Recursive search
`/M`	Filename or pattern
`/C`	Command to run
`@PATH`	Full file path

❗ Access denied errors?

Normal behavior — system protected folders ($RECYCLE.BIN, System Volume Information)

✔ File still found successfully.

1️⃣4️⃣ Finding Text Inside Files – `find`

❌ Wrong usage

find "pass" I:

✔ Correct usage

find "pass" filename.txt

Example:

find "pass" I:\ethical-test\samsaj\s299\s299.txt

Output

pass : s299
pass : s265

⚠ find is case-sensitive and basic

1️⃣5️⃣ Advanced Text Search – `findstr` (Recommended)

Search word

findstr "pass" s299.txt

Output

pass : s299
pass : s265
sam pass samina password
sajjad password pass

Why `findstr` is better?

✔ Case-insensitive (by default)
✔ Supports patterns
✔ Searches multiple lines cleanly

✅ Key Takeaways (Revision Summary)

echo > file → creates file with content
echo 2> file → creates empty file via error redirection
type → read file
del → delete file
rename → rename file
mkdir / rmdir → directory management
/S is mandatory to delete non-empty folders
fc → compare files
mklink → symbolic link (admin required)
dir /s → recursive file search
forfiles → powerful file finder + executor
findstr → best for searching text inside files

DEV Community: Sajjad Rahman

ML_Log_Target_PCF

Notebook 03 — 03_ML_Log_Target_PCF.ipynb

Cell 1 — Markdown

Cell 2 — Imports

Cell 3 — Load dataset

Cell 4 — Create ML dataset

Cell 5 — Clean PCF

Cell 6 — Define X and raw y

Cell 7 — Train/test split

Cell 8 — Create safe copies

Cell 9 — Create log target

Cell 10 — Keep original y for evaluation

Cell 11 — Country → Region

Cell 12 — Rare Industry

Cell 13 — PCF Protocol

Cell 14 — Stage-level CO2e

Cell 15 — Product Weight winsorisation

Cell 16 — Company target encoding

Cell 17 — One-hot encoding

Cell 18 — Final feature check

Cell 19 — Seven baseline models

Cell 20 — Train and evaluate correctly

Cell 21 — Baseline results

Cell 22 — Models for tuning

Cell 23 — RF GridSearch

Cell 24 — Tuned RF evaluation

Cell 25 — Extra Trees GridSearch

Cell 26 — Tuned Extra Trees evaluation

Cell 27 — XGBoost GridSearch

Cell 28 — Tuned XGBoost evaluation

Cell 29 — Tuned comparison

Cell 30 — Baseline vs tuned

Cell 31 — Save baseline results

Cell 32 — Save tuned results

Cell 33 — Save hyperparameters

Cell 34 — Experiment summary

Cell 35 — Important experiment note

⚠️ One critical point about Notebook 03

Your three notebooks are now cleanly defined

BEACON-01

Why Data Quality Matters in Product Carbon Footprint Data:

Imagine making an important decision with unreliable information

1. What does "data quality" actually mean?

Example 1 — Missing information

Example 2 — Impossible information

Example 3 — Inconsistent information

Example 4 — Extreme but legitimate information

2. Why do we need "dimensions"?

3. Where did BEACON's eight dimensions come from?

4. The three BEACON selection criteria

1. Measurability

2. Rule Definition

3. Computability

5. So what are BEACON's eight dimensions?

6. Why does each dimension matter?

1. Completeness

2. Validity

3. Consistency

4. Plausibility

5. Traceability

6. Timeliness

7. Interpretability

8. Representativeness

7. Why not use 15 dimensions?

8. Why can't we just apply all eight dimensions to every column?

Feature–Dimension Mapping

9. Why is the mapping table necessary?

10. From the mapping table to 15 rules

11. What are the 15 rules actually doing?

12. Why does BEACON use rules instead of simply asking an analyst to inspect the data?

13. Does BEACON automatically "fix" everything?

14. This is particularly important for PCF data

15. What happens after all the rules run?

16. Why have an overall score if the individual rules are more informative?

17. Why is the mapping table so important to the whole architecture?

18. Why eight dimensions rather than five?

19. Why not twelve?

20. What does this mean for machine learning?

21. The final BEACON idea in one picture

Notebook 03 — `03_ML_Log_Target_PCF.ipynb`