# ============================================================
# MODEL IMPORTS
# ============================================================
from sklearn.linear_model import LinearRegression, Ridge
from sklearn.svm import SVR
from sklearn.ensemble import RandomForestRegressor
# Optional boosting models
try:
from xgboost import XGBRegressor
XGB_AVAILABLE = True
except ImportError:
XGB_AVAILABLE = False
try:
from catboost import CatBoostRegressor
CATBOOST_AVAILABLE = True
except ImportError:
CATBOOST_AVAILABLE = False
# ============================================================
# MODEL CONFIGURATION
# ============================================================
models = {
"Linear Regression": LinearRegression(),
"Ridge Regression": Ridge(
alpha=1.0
),
"SVR": SVR(
kernel="rbf",
C=10.0,
epsilon=0.1
),
"Random Forest": RandomForestRegressor(
n_estimators=300,
random_state=42,
n_jobs=-1
)
}
# XGBoost
if XGB_AVAILABLE:
models["XGBoost"] = XGBRegressor(
n_estimators=300,
learning_rate=0.05,
max_depth=6,
subsample=0.8,
colsample_bytree=0.8,
objective="reg:squarederror",
random_state=42,
n_jobs=-1
)
# CatBoost
if CATBOOST_AVAILABLE:
models["CatBoost"] = CatBoostRegressor(
iterations=300,
learning_rate=0.05,
depth=6,
loss_function="RMSE",
random_seed=42,
verbose=False
)
print("Models selected:")
for name in models:
print(" -", name)
Model Imports
This cell defines the candidate regression models that will be compared for PCF prediction.
The key idea is that you are not relying on only one algorithm. You are testing different model families:
Linear model
↓
Linear Regression
Regularised linear model
↓
Ridge Regression
Kernel-based model
↓
SVR
Tree ensemble
↓
Random Forest
Boosting ensembles
↓
XGBoost / CatBoost
This gives you a broader comparison of modelling approaches.
1. Import Linear Regression and Ridge
from sklearn.linear_model import LinearRegression, Ridge
This imports two linear regression algorithms.
LinearRegression
Ordinary linear regression assumes the target can be represented as a linear combination of the input features.
In simplified form:
Prediction = intercept + weighted features
It is useful as a simple baseline model.
Viva question: Why include Linear Regression?
“I included Linear Regression as a simple baseline. It provides a reference point against which the more complex nonlinear models can be compared.”
Ridge
Ridge Regression is a regularised version of linear regression.
It adds a penalty to large coefficients.
This can be useful when you have many correlated features, which is particularly relevant when using high-dimensional embedding features.
Viva question: Why Ridge?
“Ridge provides a regularised linear baseline. Since the hybrid representation can contain many correlated features, Ridge can reduce the influence of excessively large coefficients and provide a useful comparison with ordinary linear regression.”
2. Import SVR
from sklearn.svm import SVR
This imports Support Vector Regression.
Your configuration later uses:
kernel="rbf"
The RBF kernel allows SVR to model nonlinear relationships between the features and PCF.
So your model comparison includes a nonlinear kernel-based approach rather than only linear models.
3. Import Random Forest
from sklearn.ensemble import RandomForestRegressor
Random Forest is an ensemble of decision trees.
It can model nonlinear relationships and interactions between features.
This is particularly useful for your hybrid feature representation because relationships between:
SBERT features
+
structured features
may not be purely linear.
Optional XGBoost Import
try:
from xgboost import XGBRegressor
XGB_AVAILABLE = True
except ImportError:
XGB_AVAILABLE = False
This is a safe optional import.
try
Python first attempts:
from xgboost import XGBRegressor
If XGBoost is installed, the import succeeds.
Then:
XGB_AVAILABLE = True
means the model is available.
except ImportError
If XGBoost is not installed, Python catches the import error.
Instead of stopping the entire notebook, you set:
XGB_AVAILABLE = False
This means the rest of the notebook can continue without XGBoost.
Viva question: Why use try/except?
“XGBoost and CatBoost are optional dependencies. I used
try/exceptso that the notebook remains executable even if one of these external libraries is unavailable.”
That's a good software-engineering decision.
Optional CatBoost Import
try:
from catboost import CatBoostRegressor
CATBOOST_AVAILABLE = True
except ImportError:
CATBOOST_AVAILABLE = False
This does the same thing for CatBoost.
If installed:
CATBOOST_AVAILABLE = True
If not:
CATBOOST_AVAILABLE = False
Therefore, your core scikit-learn models can still run.
Model Configuration
models = {
You create a Python dictionary called models.
The dictionary stores:
Model name → Model object
For example:
"Random Forest" → RandomForestRegressor(...)
This is convenient because later you can iterate through all models using something like:
for name, model in models.items():
instead of writing separate training code for every model.
1. Linear Regression
"Linear Regression": LinearRegression(),
The key:
"Linear Regression"
is simply the human-readable model name.
The value:
LinearRegression()
creates the regression model.
This serves as a baseline.
2. Ridge Regression
"Ridge Regression": Ridge(
alpha=1.0
),
Creates a Ridge Regression model.
alpha=1.0
alpha controls the strength of the regularisation.
Conceptually:
Higher alpha
→ stronger coefficient penalty
Lower alpha
→ weaker coefficient penalty
Here you use:
alpha = 1.0
as the selected configuration.
Viva question: Is alpha=1.0 automatically optimal?
Answer:
“No. It is a chosen hyperparameter configuration. A more extensive experiment could tune alpha using cross-validation.”
This is safer than claiming that 1.0 is universally optimal.
3. SVR
"SVR": SVR(
kernel="rbf",
C=10.0,
epsilon=0.1
),
Creates the Support Vector Regression model.
kernel="rbf"
RBF stands for Radial Basis Function.
It allows SVR to model nonlinear relationships.
This is useful because PCF relationships may not be linear.
C=10.0
C controls the trade-off between:
- fitting the training data closely
- allowing some prediction error
A larger C generally puts more emphasis on reducing training errors.
Again, don't claim 10 is universally optimal.
epsilon=0.1
Defines an epsilon-insensitive region around the regression prediction.
Conceptually:
Prediction
↓
small errors inside epsilon
↓
not penalised in the same way
Viva question: Why RBF SVR?
“I used the RBF kernel because it allows SVR to capture nonlinear relationships between the hybrid features and the continuous PCF target.”
4. Random Forest
"Random Forest": RandomForestRegressor(
n_estimators=300,
random_state=42,
n_jobs=-1
)
Creates a Random Forest regression model.
n_estimators=300
Creates an ensemble of 300 trees.
The final prediction is based on the combined predictions of these trees.
random_state=42
Makes the stochastic parts reproducible.
n_jobs=-1
Uses all available CPU cores for parallel processing.
Viva question: Why Random Forest?
“Random Forest can capture nonlinear relationships and interactions without requiring the feature-target relationship to be linear. It is therefore a useful tree-based model for evaluating the hybrid representation.”
XGBoost
if XGB_AVAILABLE:
This checks whether XGBoost was successfully imported.
If:
XGB_AVAILABLE = True
then the following model is added.
models["XGBoost"] = XGBRegressor(
This adds XGBoost to the dictionary.
n_estimators=300
n_estimators=300
Uses 300 boosting iterations/trees.
learning_rate=0.05
learning_rate=0.05
Controls how strongly each new tree contributes to the overall model.
A smaller learning rate generally means the model learns more gradually.
max_depth=6
max_depth=6
Controls the maximum depth of individual trees.
Greater depth can capture more complex relationships but can also increase overfitting risk.
subsample=0.8
subsample=0.8
Uses approximately 80% of the training observations for each boosting stage.
This introduces randomness and can help reduce overfitting.
colsample_bytree=0.8
colsample_bytree=0.8
Uses approximately 80% of the features for each tree.
This introduces feature-level randomness.
objective="reg:squarederror"
objective="reg:squarederror"
Specifies that XGBoost is solving a regression problem using squared error.
This is appropriate because PCF is a continuous target.
random_state=42
Provides reproducibility.
n_jobs=-1
Uses available CPU resources for parallel processing.
CatBoost
if CATBOOST_AVAILABLE:
Checks whether CatBoost is available.
If it is:
models["CatBoost"] = CatBoostRegressor(
adds a CatBoost regression model.
iterations=300
iterations=300
Uses 300 boosting iterations.
learning_rate=0.05
Controls how strongly each boosting iteration contributes.
depth=6
Controls tree depth.
loss_function="RMSE"
loss_function="RMSE"
Specifies RMSE as the optimisation loss.
random_seed=42
Provides reproducibility.
verbose=False
Suppresses CatBoost's detailed training output.
This keeps the notebook output cleaner.
Print Selected Models
print("Models selected:")
Prints a heading.
Then:
for name in models:
loops through the dictionary keys.
Each key is a model name.
For example:
Linear Regression
Ridge Regression
SVR
Random Forest
XGBoost
CatBoost
Finally:
print(" -", name)
prints each model name.
So the output might look like:
Models selected:
- Linear Regression
- Ridge Regression
- SVR
- Random Forest
- XGBoost
- CatBoost
The exact final two depend on whether those libraries are installed.
Why Compare These Models?
This is a very likely viva question.
You are deliberately comparing different model families.
Linear Regression
→ simple linear baseline
Ridge
→ regularised linear model
SVR
→ nonlinear kernel model
Random Forest
→ bagging/tree ensemble
XGBoost
→ gradient boosting
CatBoost
→ gradient boosting
This gives you a broader experimental comparison.
Strong viva answer
“I selected models from different algorithmic families so that I could evaluate whether the hybrid PCF features work better with linear, regularised, kernel-based, bagging, or boosting approaches. Linear Regression provides a simple baseline, Ridge addresses regularisation, SVR captures nonlinear relationships through an RBF kernel, and Random Forest, XGBoost and CatBoost provide tree-based ensemble alternatives.”
Important Question: Why Not Just Use Random Forest?
If your examiner asks:
“If Random Forest works well, why did you test all these models?”
Answer:
“Because selecting a model based only on prior expectation would introduce unnecessary assumptions. Comparing multiple model families allows me to empirically determine which algorithm works best with the engineered hybrid representation.”
Important Question: Are XGBoost and CatBoost Necessary?
Answer:
“They are optional comparison models rather than mandatory components of the pipeline. I included them to broaden the model comparison. If the libraries are unavailable, the core scikit-learn models can still be evaluated.”
Important Question: Are these hyperparameters tuned?
Based only on this code, these are predefined configurations, not a comprehensive hyperparameter search.
So say:
“These are selected model configurations used for comparison. They are not the result of an exhaustive hyperparameter optimisation procedure.”
This is a very defensible answer.
Important Question: Which model is best?
You cannot answer that from this cell alone.
This cell only defines the models.
The actual answer must come from your later evaluation results.
The process is:
Models defined
↓
Train models
↓
Generate validation/test predictions
↓
Calculate MAE, RMSE, R²
↓
Compare models
↓
Select best-performing model
So don't say:
“Random Forest is the best model.”
unless your later results actually demonstrate that.
One Critical Point for Your Viva
Your modelling target is:
log-transformed PCF
but your evaluation metrics are calculated after converting predictions back to:
raw PCF scale
So your complete modelling process is:
Raw PCF
↓
log1p transformation
↓
Model training
↓
Prediction in log space
↓
expm1
↓
Raw PCF prediction
↓
MAE / RMSE / R²
This is worth remembering because an examiner may ask:
“Why are you training on log PCF but evaluating on raw PCF?”
Answer:
“The log transformation helps manage the skewed target distribution during modelling, while converting predictions back to the original PCF scale makes the evaluation metrics directly interpretable in the original target units.”
Presentation wording
“In this cell, I define the regression models used for comparative evaluation. I include Linear Regression as a simple baseline, Ridge Regression as a regularised linear model, SVR with an RBF kernel to capture nonlinear relationships, and Random Forest as a tree-based ensemble. I also include XGBoost and CatBoost as optional boosting models when their libraries are available. I store all models in a dictionary so that the same evaluation pipeline can be applied consistently across algorithms. The hyperparameters shown here are predefined configurations rather than the result of exhaustive hyperparameter optimisation. The final model selection is therefore based on the comparative validation or test performance obtained in the subsequent evaluation stage.”
For your model comparison
- Explain why raw-scale metrics can differ
- Check whether scaling changes SVR and Ridge
Top comments (0)