DEV Community

James Ononiwu
James Ononiwu

Posted on

3 1

Custom model Evaluation Metric

In building machine learning models, There comes a time you might want to evaluate your model to know it's a performance using a metric you created. while scikit-learn with all it's beauty provides the evaluation metrics we need, there comes a time we might need to create our own metrics. scikit-learn makes this easy by providing a function make_scorer.
to create a custom metric, all you need to do is create a function containing the metric definition and convert it to a scorer function using scikit-learn make_scorer. we will use the make_regression dataset to show an example of creating a custom metric below.

let's begin by importing the neccessary libraries

# Load libraries
from sklearn.metrics import make_scorer, r2_score
from sklearn.model_selection import train_test_split
from sklearn.linear_model import Ridge
from sklearn.datasets import make_regression
Enter fullscreen mode Exit fullscreen mode

Next we generate features and the target matrix.

features, target = make_regression(n_samples = 100,n_features = 4,
random_state = 1)
Enter fullscreen mode Exit fullscreen mode

random_state parameter ensures we get the same distribution when we use the same random_state value.

let's continue by creating the training and testing set using train_test_split

X_train, X_test, y_train, y_test = train_test_split(features, target, test_size=0.10, random_state=1)
Enter fullscreen mode Exit fullscreen mode

then create our custom metric function.

def custom_metric(X_test, target_predicted):
    # Calculate r-squared score
    r_squared = r2_score(X_test, target_predicted)
    # Return r-squared score
    return r_squared
Enter fullscreen mode Exit fullscreen mode

next convert the custom metric into a scorer metric

# Make scorer and define that higher scores are better
score = make_scorer(custom_metric, greater_is_better=True)
Enter fullscreen mode Exit fullscreen mode

greater_is_better signifies that higher score is better

Create the regression and train.

classifier = Ridge()
# Train ridge regression model
model = classifier.fit(X_train, y_train)
Enter fullscreen mode Exit fullscreen mode

Finally apply the custom metric to check model performance.

score(model, X_test, y_test)
Enter fullscreen mode Exit fullscreen mode

Image of Timescale

🚀 pgai Vectorizer: SQLAlchemy and LiteLLM Make Vector Search Simple

We built pgai Vectorizer to simplify embedding management for AI applications—without needing a separate database or complex infrastructure. Since launch, developers have created over 3,000 vectorizers on Timescale Cloud, with many more self-hosted.

Read more

Top comments (0)

Sentry image

See why 4M developers consider Sentry, “not bad.”

Fixing code doesn’t have to be the worst part of your day. Learn how Sentry can help.

Learn more