Short answer: to know if a search change helped, write down 5 to 10 real queries and the right answers before you change anything, then count. I did this for a tiny help-center search with three versions. One number said the three versions were equal. The per-query table said they were not. This post shows the code, the real results, and what the test could not tell me.
The setup
I made up 12 help-center articles for an imaginary notes app. Then I wrote 8 queries that a user might type, and for each one I wrote which article is the right answer. I did this before I ran any search, and I did not change the data after the first run.
# A made-up help center. Each item is (id, title).
DOCS = [
("d1", "How to reset your password if you forgot it"),
("d2", "Change your email address in account settings"),
("d3", "Export your notes as PDF or Markdown"),
("d4", "Delete your account and all stored data"),
("d5", "Invite teammates to a shared workspace"),
("d6", "Billing: how to update your credit card"),
("d7", "Cancel your subscription and stop future charges"),
("d8", "Two-factor authentication: set up an authenticator app"),
("d9", "Recover deleted notes from the trash within 30 days"),
("d10", "Why am I seeing a login error after changing my password"),
("d11", "Refunds: when you can get your money back"),
("d12", "Keyboard shortcuts for faster note taking"),
]
# The test set. I wrote the right answers before I ran anything.
TESTS = [
("forgot password", {"d1"}),
("cancel being charged", {"d7"}),
("get my money back", {"d11"}),
("recover deleted note", {"d9"}),
("how to export notes", {"d3"}),
("log in problem", {"d10"}),
("change my email", {"d2"}),
("two step security", {"d8"}),
]
Three versions of the same search
Each version adds one idea to the one before.
- Version A matches exact words.
- Version B also cuts simple endings, like s, ed and ing, so that note and notes match.
- Version C also removes small common words, like how, to and my.
STOP = {"how", "to", "my", "the", "a", "an", "and", "or", "in", "is", "of", "your", "you", "i", "am", "can", "be", "being", "why", "if", "it", "for", "when"}
def tokens(text):
return re.findall(r"[a-z0-9]+", text.lower())
def stem(word):
for suffix in ("ing", "ed", "es", "s"):
if word.endswith(suffix) and len(word) - len(suffix) >= 3:
return word[: -len(suffix)]
return word
# Three versions of the same search. Each adds one idea to the one before.
def prep_a(text):
return tokens(text)
def prep_b(text):
return [stem(t) for t in tokens(text)]
def prep_c(text):
return [stem(t) for t in tokens(text) if t not in STOP]
def search(prep, query):
q = set(prep(query))
scored = []
for order, (doc_id, title) in enumerate(DOCS):
score = len(q & set(prep(title)))
if score > 0:
scored.append((-score, order, doc_id))
scored.sort() # best score first, ties keep the list order
return [doc_id for _, _, doc_id in scored]
The score is the number of query words that appear in the title. Articles with the same score stay in list order. That is a choice, and it changes results, as you will see.
Two numbers
- hit@3 is the share of queries where a right answer is in the top 3 results. It tells you if the user can find it.
- MRR is the mean of 1 divided by the rank of the first right answer. A right answer at rank 1 gives 1, at rank 2 gives 0.5, at rank 3 gives 0.33, and a miss gives 0. It tells you how high the answer sits.
def reciprocal_rank(results, relevant):
for rank, doc_id in enumerate(results, start=1):
if doc_id in relevant:
return 1 / rank
return 0.0
def evaluate(prep, k=3):
hit = rr = 0.0
rows = []
for query, relevant in TESTS:
results = search(prep, query)
found = bool(set(results[:k]) & relevant)
hit += found
r = reciprocal_rank(results, relevant)
rr += r
rows.append((query, results[:k], found, r))
n = len(TESTS)
return hit / n, rr / n, rows
The real output
A: exact words hit@3 = 0.88 MRR = 0.81
forgot password top3 ['d1', 'd10'] hit rr 1.00
cancel being charged top3 ['d7'] hit rr 1.00
get my money back top3 ['d11', 'd10'] hit rr 1.00
recover deleted note top3 ['d9', 'd12'] hit rr 1.00
how to export notes top3 ['d1', 'd3', 'd6'] hit rr 0.50
log in problem top3 ['d2'] MISS rr 0.00
change my email top3 ['d2', 'd10'] hit rr 1.00
two step security top3 ['d8'] hit rr 1.00
B: + stemming hit@3 = 0.88 MRR = 0.81
forgot password top3 ['d1', 'd10'] hit rr 1.00
cancel being charged top3 ['d7'] hit rr 1.00
get my money back top3 ['d11', 'd10'] hit rr 1.00
recover deleted note top3 ['d9', 'd12'] hit rr 1.00
how to export notes top3 ['d1', 'd3', 'd6'] hit rr 0.50
log in problem top3 ['d2'] MISS rr 0.00
change my email top3 ['d2', 'd10'] hit rr 1.00
two step security top3 ['d8'] hit rr 1.00
C: + stop words removed hit@3 = 0.88 MRR = 0.88
forgot password top3 ['d1', 'd10'] hit rr 1.00
cancel being charged top3 ['d7'] hit rr 1.00
get my money back top3 ['d11'] hit rr 1.00
recover deleted note top3 ['d9', 'd12'] hit rr 1.00
how to export notes top3 ['d3', 'd9'] hit rr 1.00
log in problem top3 [] MISS rr 0.00
change my email top3 ['d2'] hit rr 1.00
two step security top3 ['d8'] hit rr 1.00
I ran it twice and got the same output. The word hit or MISS says if a right answer was in the top 3, and rr is the 1 divided by rank for that query.
What the output says
- hit@3 is 0.88 for all three versions. That is 7 queries out of 8. If I had looked only at that number, I would say the changes did nothing.
- MRR moves from 0.81 to 0.81 to 0.88. The gain comes from one query, "how to export notes". In versions A and B the right article is second, because the small words "how" and "to" match other articles first, and ties keep the list order. Version C removes those words, and the right article moves to rank 1.
- Stemming (version B) changed nothing on this test set. That does not mean stemming is useless. It means none of my 8 queries needed it. The test set is too small to say more.
- The query "log in problem" is a miss in all three versions, for two different reasons. In A and B, the small word "in" matches "in account settings", so the search returns the wrong article with no sign of doubt. In C, "in" is gone, and the search returns nothing, because the article says "login" and the query says "log in". Fixing this needs something else, like synonyms or a different kind of search. I did not test one.
Lessons
- Write the right answers first. If you look at results first, you will agree with whatever the search returned.
- Use two numbers. hit@3 tells you if it can be found, and MRR tells you how high. They can disagree.
- Read the rows, not only the averages. The average hid the one query that changed.
- Remember how small the test is. With 8 queries, one query is 0.125 of hit@3. A change of that size is one query, not a trend.
- Check ties. Which article wins a tie can decide the result.
- Keep misses in the test set. The "log in problem" query is the most useful row here, because it points at what to build next.
The full script
Save this as evaldemo.py and run python evaldemo.py. It needs no packages.
import re
# A made-up help center. Each item is (id, title).
DOCS = [
("d1", "How to reset your password if you forgot it"),
("d2", "Change your email address in account settings"),
("d3", "Export your notes as PDF or Markdown"),
("d4", "Delete your account and all stored data"),
("d5", "Invite teammates to a shared workspace"),
("d6", "Billing: how to update your credit card"),
("d7", "Cancel your subscription and stop future charges"),
("d8", "Two-factor authentication: set up an authenticator app"),
("d9", "Recover deleted notes from the trash within 30 days"),
("d10", "Why am I seeing a login error after changing my password"),
("d11", "Refunds: when you can get your money back"),
("d12", "Keyboard shortcuts for faster note taking"),
]
# The test set. I wrote the right answers before I ran anything.
TESTS = [
("forgot password", {"d1"}),
("cancel being charged", {"d7"}),
("get my money back", {"d11"}),
("recover deleted note", {"d9"}),
("how to export notes", {"d3"}),
("log in problem", {"d10"}),
("change my email", {"d2"}),
("two step security", {"d8"}),
]
STOP = {"how", "to", "my", "the", "a", "an", "and", "or", "in", "is", "of", "your", "you", "i", "am", "can", "be", "being", "why", "if", "it", "for", "when"}
def tokens(text):
return re.findall(r"[a-z0-9]+", text.lower())
def stem(word):
for suffix in ("ing", "ed", "es", "s"):
if word.endswith(suffix) and len(word) - len(suffix) >= 3:
return word[: -len(suffix)]
return word
# Three versions of the same search. Each adds one idea to the one before.
def prep_a(text):
return tokens(text)
def prep_b(text):
return [stem(t) for t in tokens(text)]
def prep_c(text):
return [stem(t) for t in tokens(text) if t not in STOP]
def search(prep, query):
q = set(prep(query))
scored = []
for order, (doc_id, title) in enumerate(DOCS):
score = len(q & set(prep(title)))
if score > 0:
scored.append((-score, order, doc_id))
scored.sort() # best score first, ties keep the list order
return [doc_id for _, _, doc_id in scored]
def reciprocal_rank(results, relevant):
for rank, doc_id in enumerate(results, start=1):
if doc_id in relevant:
return 1 / rank
return 0.0
def evaluate(prep, k=3):
hit = rr = 0.0
rows = []
for query, relevant in TESTS:
results = search(prep, query)
found = bool(set(results[:k]) & relevant)
hit += found
r = reciprocal_rank(results, relevant)
rr += r
rows.append((query, results[:k], found, r))
n = len(TESTS)
return hit / n, rr / n, rows
if __name__ == "__main__":
for name, prep in (("A: exact words", prep_a), ("B: + stemming", prep_b), ("C: + stop words removed", prep_c)):
hit, mrr, rows = evaluate(prep)
print(f"\n{name} hit@3 = {hit:.2f} MRR = {mrr:.2f}")
for query, top, found, r in rows:
print(f" {query:22} top3 {str(top):24} {'hit ' if found else 'MISS'} rr {r:.2f}")
Disclosure: I build Libraryminds, a video search tool. This post does not use it, and the data above is made up for the demo.
If you have a better way to build a small test set, tell me in the comments.
Top comments (0)