There are many metrics to judge a classical machine learning model like Accuracy, Precision, Recall, F1 etc, but they cant be directly applied to a recommendation system model.
So in this post I'm writing about the metrics we can use to judge a recommendation system model.
- Recommended items: Items which our model predicted for a given user. We can get these items from the model predictions for a given user.
Relevant items: Items in which the user is actually interested in. These can be obtained from several methods like:
- Items on which the user has clicked.
- Items which the user has searched for.
The goal of a Recommendation model is simply to maximize the intersection between recommended and relevant items for all users.
Here are some metrics we can use for checking the performance of our model:
How many relevant items are present in top k recommendations?
For eg. P@5 would be calculated by taking top 5 recommendations from the system for each user and checking how many are relevant ones.
If 3 items are relevant then P@5 = 3/5 = 0.6.
The mean of P@i for i = 1..K
For eg, AP@3 = SUM(P@1 + P@2 + P@3)/3. This will be calculated for each user.
Mean of AP@K for all users.
For eg, MAP@3 = SUM(AP@1 + AP@2 + AP@3)/3
These metrics are straightforward to implement, also can be obtained from here.
Happy Learning !