How to Judge your Recommendation System Model ?

There are many metrics to judge a classical machine learning model like Accuracy, Precision, Recall, F1 etc, but they cant be directly applied to a recommendation system model.
So in this post I'm writing about the metrics we can use to judge a recommendation system model.

Recommended Vs Relevant Items

• Recommended items: Items which our model predicted for a given user. We can get these items from the model predictions for a given user.
• Relevant items: Items in which the user is actually interested in. These can be obtained from several methods like:
  1. Items on which the user has clicked.
  2. Items which the user has searched for.

The goal of a Recommendation model is simply to maximize the intersection between recommended and relevant items for all users.

Here are some metrics we can use for checking the performance of our model:

Precision @K:

How many relevant items are present in top k recommendations?

For eg. P@5 would be calculated by taking top 5 recommendations from the system for each user and checking how many are relevant ones.
If 3 items are relevant then P@5 = 3/5 = 0.6.

Average Precision @K:

The mean of P@i for i = 1..K

For eg, AP@3 = SUM(P@1 + P@2 + P@3)/3. This will be calculated for each user.

Mean Average Precision @K:

Mean of AP@K for all users.

For eg, MAP@3 = SUM(AP@1 + AP@2 + AP@3)/3

These metrics are straightforward to implement, also can be obtained from here.
Happy Learning !