“I love it when my search engine retrieves irrelevant search results,” said nobody ever.
As developers, we search for things on a daily basis. Whether we are searching for a solution on how to fix a bug or how to build a new feature, it is incredibly frustrating when you are not quite getting what you are looking for.
That is what
relevance is all about!
When you search for something on your app, you want to get results that are directly related to what you are looking for.
Elasticsearch is a powerful search and analytics engine designed to do just that! By integrating Elasticsearch into your app, you can create a great search experience for your users.
This blog will help you understand the relevance of your search with Elasticsearch and Kibana!
By the end of this blog, you will be able to:
1) learn how precision and recall are used to measure how well Elasticsearch is searching for user's query
2) understand how scoring is used to rank relevancy of search results in Elasticsearch
3) use resources provided at the end of the blog to master Elasticsearch queries that fine-tunes the precision or recall of your search results
Interested in beginner friendly workshops on Elasticsearch and Kibana? Check out my Beginner's Crash Course to Elastic Stack series!
Here is the table of content of all the workshops I have created for the series thus far. This repo contains all resources shared during the workshops including the video recording, presentation, related blogs, Elasticsearch queries and more!
There are multiple factors that come into play. This blog will focus on
Before we get to that, let's do a quick recap!
Elasticsearch is a search engine that allows us to store, search, and analyze data. It stores data as documents. Documents that share similar traits are grouped into an index.
When you search for something, Elasticsearch retrieves relevant documents(green boxes). Then, it presents these documents as search results(orange box).
The two diagrams in the image above depict the same thing.
The diagram on the left shows documents grouped into an index. The same thing is shown on the diagram on the right.
The yellow rectangle represents your sample index. The grey dots are the documents contained in that index.
This blog will use the diagram on the right to explain
recall. But before we do that, let’s go over some terms!
When you send a search query to Elasticsearch, it retrieves documents that it considers relevant to the query.
These are the dots inside the white circle. These are the documents that Elasticsearch sends back as a response.
Some of the these retrieved documents are what you expect to see in your response. These are known as
You probably had an experience where you searched for something and some of the results were not relevant to what you were looking for.
These are known as
false positives. These are irrelevant search results that were retrieved by the Search engine for some reason.
Let’s focus on the dots in the yellow region of this diagram. These are the documents that were not returned by the search engine.
Some of them are truly irrelevant to the search query and were correctly rejected by the search engine. These are known as
Among the rejected documents, there may be relevant documents that should have been returned in the response. These are known as
Now that we have mastered these terms, let's get back to relevance! Earlier, I have mentioned that
recall are used to measure the relevance of a search engine.
precision has to do with the dots inside the white circle. These are documents that are returned as search results.
Precision is calculated by true positives divided by the sum of true positives and false positives.
Precision tells you what portion of the retrieved data is actually relevant to the search query.
Recall on the other hand is calculated by true positives divided by the sum of true positives and false negatives. It tells you what portion of relevant data is being returned as search results.
recall are inversely related.
Precision wants all the retrieved results to a perfect match to the query, even if it means returning less or no documents.
recall focuses more on quantity. It wants to retrieve more results even if the documents may not be a perfect match to the query. The dilemma is that we want to present the really relevant items but we also want to retrieve as many results as possible.
As you can see, these two factors are at odds with each other because if you want to improve
precision, it might cause a decline in
recall and vice versa.
Let’s recap real quick!
recall determine which documents are included in the search results. However,
recall do not determine which of these returned documents are more relevant than the other.
This is determined by
When you look at your search results, you will see that the most relevant results are at the top and the least relevant are at the bottom.
ranking or order is determined by a scoring algorithm.
Each result is given a
score. Ones with the highest scores are displayed at the top whereas ones with the lowest scores are displayed at the bottom.
Score is a value that represents how relevant a document is to that specific query. A score is computed for each document that is a hit, and hits are search results that are sent to the user.
The higher the score a document has, more relevant the document is to the query, and it’s going to end up higher in the order!
There are multiple factors that are used to compute a document's score. This blog will only focus on
term frequency(TF) and
inverse document frequency(IDF).
When you search for something, you type in a search query in the search box. Elasticsearch looks at the query and pulls up relevant documents or hits.Then, it calculates a score for each document and ranks them by relevance.
So how does this happen?
Let’s talk about how
term frequency plays a role in calculating a score.
In the image below, we have a search query(red bracket) “ how to form good habits”. This query is made of multiple search terms(blue arrows).
When the relevant documents are retrieved, Elasticsearch looks through these documents and calculates how many times each search term appears in a document.
This is known as
If a document mentions search terms more frequently, Elasticsearch assumes that this document is more relevant to the search query and it assigns a higher score to the document.
Let’s say we were looking at the frequency for the term habits(orange box).
In the field description of the first document(red box), the term habits appears 4 times(TF=4). In the field description of the the second document(blue box), the term habits appears 1 time(TF=1).
Therefore, in this example, the first document will be given a higher score and end up higher on the search results.
When we calculate a score based on
term frequency alone, this will not give us the most relevant documents. This happens because
term frequency considers all search terms to be equally important when assessing the relevance of a document.
We have search terms, "how" and "to" and "form" and "good" and "habits". Not all of these search terms will help you determine the relevance of a document.
For example, the first four search terms are commonly seen in many, if not all documents. Take a look at the hits(blue box), then at the documents highlighted with an orange box.
The documents like "how to form a meetup group" or "good chicken recipes" do contain some of the search terms. But these documents are completely irrelevant to what we are looking for!
But because of
term frequency, if these commonly found search terms were found in high frequency in any of these documents, these documents will end up with high scores, even though these are irrelevant to the query.
So Elasticsearch offsets this with
inverse document frequency(IDF). With Elasticsearch, if certain search terms are found in many documents in the result set, it knows that these terms are not useful at determining relevance.
When Elasticsearch goes through all the hits, it will reduce the score for documents with unimportant search terms. It will also increase the score for documents with important search term like habits.
Now that we have covered the basics of relevance, we are ready to fine-tune the
recall of our search results! The best way to show you how these queries work is by sending these queries from Kibana to Elasticsearch.
There are three things you must complete in order to follow along! This repo contains all of the directions and resources you will need to complete these steps.
When you open the repo and scroll down to the Resources section, you will see the following.
Step 1: Set up Elasticsearch and Kibana
You can access Elasticsearch and Kibana in two ways:
1) Access Elasticsearch and Kibana hosted on Elastic Cloud
2) Download Elasticsearch and Kibana on your local machine
Instructions for each option is included under Resources(green box).
Option 1: Accessing Elasticsearch and Kibana hosted on Elastic Cloud
Unlike the downloaded option, Elastic Cloud handles all the heavy lifting of managing the stack so you can focus on building your product instead.
You get a free 30 day trial. There is no credit card required and the trial expires on its own. I find that for beginners it is easier and faster to get started on Elastic Cloud.
Option 2: Downloading Elasticsearch and Kibana on your local machine
This option is free and there is no expiration date. However, you will need to manage Elasticsearch and Kibana on your own.
Both options will perfectly be able to handle all of the queries we are about to go over. Please choose whatever option that works best for you!
From the same repo, click on the video of the workshop(orange box). Go to time stamp 18:57.
This video will walk you through how you can add the dataset(yellow box) to Elasticsearch.
Step 3: Master the queries to fine-tune the precision or recall of your search results
The rest of the video will walk you through different queries that allows you to fine-tune the precision and recall of your search results!
All the search queries covered in the video are included in the same repo(below the Resources section). So make sure to keep the repo open while you follow along.
All right, you are all set to go. Master these queries and explore what you can do with Elasticsearch and Kibana!