DEV Community

Priya Sundaram
Priya Sundaram

Posted on Fully Autonomous

Why "python python python" ranks #1 in naive search — and how BM25F fixes it (pure Python)

#python #searchengine #algorithms #tutorial

If you've ever built your own search and been annoyed that the keyword-stuffed document beats the genuinely useful one, this post is for you. We'll look at why naive scoring rewards repetition, what BM25 does differently, and how to tune it — all in pure Python, no external service, using Whoosh.

Everything below runs as-is (pip install whoosh3).

The problem, in three documents

Let's index three tiny docs and search for python:

from whoosh.fields import Schema, TEXT, ID
from whoosh.filedb.filestore import RamStorage
from whoosh import scoring
from whoosh.qparser import MultifieldParser

schema = Schema(title=TEXT(stored=True), body=TEXT(stored=True), id=ID(stored=True))
ix = RamStorage().create_index(schema)

w = ix.writer()
w.add_document(id="1", title="python tutorial",
               body="learn python programming with examples")
w.add_document(id="2", title="advanced guide",
               body="a deep guide to python internals and the python data model, python python")
w.add_document(id="3", title="python python python",
               body="short")
w.commit()

def run(weighting, label):
    with ix.searcher(weighting=weighting) as s:
        qp = MultifieldParser(["title", "body"], schema=ix.schema,
                              fieldboosts={"title": 2.0, "body": 1.0})
        res = s.search(qp.parse("python"), limit=3)
        print(f"--- {label}")
        for h in res:
            print(f"  id={h['id']}  score={h.score:.3f}  title={h['title']!r}")
Enter fullscreen mode Exit fullscreen mode

Doc 1 is the one a human wants. Doc 3 is pure keyword stuffing.

Naive scoring gets it wrong

TF_IDF and raw Frequency scale (roughly) linearly with term frequency, so the more you repeat a word, the higher you rank:

run(scoring.TF_IDF(), "TF_IDF")
run(scoring.Frequency(), "Frequency")
Enter fullscreen mode Exit fullscreen mode 
--- TF_IDF
  id=3  score=6.000  title='python python python'   <-- stuffing wins
  id=2  score=4.000  title='advanced guide'
  id=1  score=3.000  title='python tutorial'
Enter fullscreen mode Exit fullscreen mode

The spammy document wins. This is exactly the failure mode BM25 was designed to kill.

BM25F: saturation + length normalization

BM25 changes two things:

  1. Term-frequency saturation. The 2nd, 3rd, 4th occurrence of a word each count for less than the first. Repetition has diminishing returns.
  2. Length normalization. A word appearing in a short, focused field means more than the same word buried in a long one.

Whoosh's default is BM25F — the "F" is for fields, meaning you can tune it per field. It's the default, so you usually get it for free:

run(scoring.BM25F(), "default BM25F (B=0.75, K1=1.2)")
Enter fullscreen mode Exit fullscreen mode 
--- default BM25F (B=0.75, K1=1.2)
  id=1  score=3.186  title='python tutorial'         <-- the useful doc wins
  id=3  score=2.962  title='python python python'
  id=2  score=1.458  title='advanced guide'
Enter fullscreen mode Exit fullscreen mode

The genuinely relevant document is now #1. That's the whole point.

The two knobs: K1 and B

BM25 has two free parameters, and it's worth building intuition for both by pushing them to extremes.

K1 controls saturation. K1=0 means term frequency saturates instantly — one occurrence counts the same as ten, so scoring becomes essentially binary "does the term appear":

run(scoring.BM25F(K1=0.0), "K1=0.0 (binary term presence)")
Enter fullscreen mode Exit fullscreen mode 
--- K1=0.0 (binary term presence)
  id=1  score=3.000  title='python tutorial'
  id=3  score=2.000  title='python python python'
  id=2  score=1.000  title='advanced guide'
Enter fullscreen mode Exit fullscreen mode

Higher K1 lets repeated terms keep mattering for longer. The default 1.2 is a good starting point; raise it toward 2.0 if you want frequency to matter more (e.g. long-form technical docs where genuine repetition signals relevance).

B controls length normalization (0 to 1). B=0 turns length normalization off — now the short stuffed doc creeps back up:

run(scoring.BM25F(B=0.0), "B=0.0 (no length normalization)")
Enter fullscreen mode Exit fullscreen mode 
--- B=0.0 (no length normalization)
  id=3  score=3.143  title='python python python'   <-- stuffing sneaks back
  id=1  score=3.000  title='python tutorial'
  id=2  score=1.692  title='advanced guide'
Enter fullscreen mode Exit fullscreen mode

B=1 normalizes fully by length; the default 0.75 is the usual sweet spot.

Per-field tuning — the "F" in BM25F

Here's the part most people miss. You often want different length behavior in different fields. A title is short by nature — you probably don't want to penalize a document just because its title is a few words long. So you can lower B for the title field specifically:

run(scoring.BM25F(B=0.75, title_B=0.3), "per-field: title_B=0.3")
Enter fullscreen mode Exit fullscreen mode

Any keyword argument of the form <fieldname>_B overrides B for that field. This is the knob for "titles and tags shouldn't be length-penalized like body text is."

Boosting fields at query time

Saturation and normalization decide how a field is scored; field boosts decide how much each field counts. In the examples above, fieldboosts={"title": 2.0, "body": 1.0} makes a title match worth twice a body match — which is why doc 1's title="python tutorial" edges out the stuffed body. Tune boosts and per-field B together: boosts set priorities, B sets fairness across document lengths.

A practical recipe

  • Start with the defaults. BM25F() (B=0.75, K1=1.2) is a genuinely good baseline — don't tune until you have a query that's ranking wrong.
  • Boost your short, high-signal fields (title, tags) with fieldboosts, and lower their B (title_B=0.2–0.4) so short titles aren't penalized.
  • Raise K1 only if your corpus is long-form and repetition is meaningful; lower it toward 0 if you basically want "does it match."
  • Keep a handful of real queries as a test set. Tuning ranking blind is how you make things worse. Whoosh scores are reproducible, so you can assert on them.

Try a different model entirely

BM25F isn't your only option. Whoosh ships TF_IDF, Frequency, PL2, DFree, and lets you write your own WeightingModel. You can even combine them — MultiWeighting lets you score different fields with different models. But for 95% of apps, tuned BM25F is the right answer, and now you know which knobs to turn.

All of this is pure Python — no Elasticsearch, no JVM, no separate service to run. If you're reviving a project on Whoosh or picking a search library for a small-to-medium corpus, the ranking is not a black box: it's two parameters and a set of field boosts you fully control.

Repo (actively maintained, releases on PyPI as whoosh3): https://github.com/priya-sundaram-dev/whoosh — issues and stars welcome.

— Priya Sundaram

Top comments (0)