DEV Community

James Ononiwu
James Ononiwu

Posted on

3 1

Custom model Evaluation Metric

In building machine learning models, There comes a time you might want to evaluate your model to know it's a performance using a metric you created. while scikit-learn with all it's beauty provides the evaluation metrics we need, there comes a time we might need to create our own metrics. scikit-learn makes this easy by providing a function make_scorer.
to create a custom metric, all you need to do is create a function containing the metric definition and convert it to a scorer function using scikit-learn make_scorer. we will use the make_regression dataset to show an example of creating a custom metric below.

let's begin by importing the neccessary libraries

# Load libraries
from sklearn.metrics import make_scorer, r2_score
from sklearn.model_selection import train_test_split
from sklearn.linear_model import Ridge
from sklearn.datasets import make_regression
Enter fullscreen mode Exit fullscreen mode

Next we generate features and the target matrix.

features, target = make_regression(n_samples = 100,n_features = 4,
random_state = 1)
Enter fullscreen mode Exit fullscreen mode

random_state parameter ensures we get the same distribution when we use the same random_state value.

let's continue by creating the training and testing set using train_test_split

X_train, X_test, y_train, y_test = train_test_split(features, target, test_size=0.10, random_state=1)
Enter fullscreen mode Exit fullscreen mode

then create our custom metric function.

def custom_metric(X_test, target_predicted):
    # Calculate r-squared score
    r_squared = r2_score(X_test, target_predicted)
    # Return r-squared score
    return r_squared
Enter fullscreen mode Exit fullscreen mode

next convert the custom metric into a scorer metric

# Make scorer and define that higher scores are better
score = make_scorer(custom_metric, greater_is_better=True)
Enter fullscreen mode Exit fullscreen mode

greater_is_better signifies that higher score is better

Create the regression and train.

classifier = Ridge()
# Train ridge regression model
model = classifier.fit(X_train, y_train)
Enter fullscreen mode Exit fullscreen mode

Finally apply the custom metric to check model performance.

score(model, X_test, y_test)
Enter fullscreen mode Exit fullscreen mode

AWS Q Developer image

Your AI Code Assistant

Automate your code reviews. Catch bugs before your coworkers. Fix security issues in your code. Built to handle large projects, Amazon Q Developer works alongside you from idea to production code.

Get started free in your IDE

Top comments (0)

Postmark Image

Speedy emails, satisfied customers

Are delayed transactional emails costing you user satisfaction? Postmark delivers your emails almost instantly, keeping your customers happy and connected.

Sign up

👋 Kindness is contagious

Please leave a ❤️ or a friendly comment on this post if you found it helpful!

Okay