BM25 Length Normalization: Why Long RAG Chunks Never Rank

A pricing table chunk in my index contained the exact answer to the query — the product name three times, the tier name twice, the number the user asked for. It never cracked the lexical top 10. What did rank first was a 60-token section heading that mentioned the product once and nothing else.

The embedding arm wasn't the problem. The reranker never saw the chunk because the candidate set was already wrong. The culprit was BM25 length normalization: the b=0.75 default that ships with Lucene, Elasticsearch, OpenSearch, and every rank_bm25 copy-paste. It was designed for TREC news articles, and your RAG chunk corpus violates the assumption it encodes.

TL;DR

• BM25's length penalty is multiplicative in the denominator and applies to every matched term: at b=0.75, a 900-token chunk with 3 occurrences of a term scores ~30% lower per term than a 60-token chunk with 1 occurrence.
• b interpolates between "no length correction" (b=0) and "full correction" (b=1). The default assumes long documents are long because they're verbose, not because they cover more ground. Chunked RAG corpora break that assumption.
• avgdl is a mean over your chunk length distribution. If you chunk tables, code, and prose into one index, that distribution is bimodal and the mean describes nothing.
• In Elasticsearch, avgdl and docCount are per shard, per field, and include deleted-but-unmerged docs — so the same chunk scores differently depending on which shard it landed on unless you use dfs_query_then_fetch.
• Lucene stores the length norm in a single byte with a 4-bit mantissa. Above a few dozen tokens, lengths are bucketed geometrically — 300 and 320 tokens are often literally the same number to the scorer.

What does b actually do in the BM25 formula?

b controls how much a document's length discounts its term frequency. The Lucene form:

$$\text{score} = \sum_{t \in q} \text{idf}(t) \cdot \frac{tf_t \cdot (k_1 + 1)}{tf_t + k_1 \cdot \left(1 - b + b \cdot \frac{dl}{avgdl}\right)}$$

The length term sits in the denominator next to tf. That placement matters: it isn't a small additive correction, it's a divisor that scales with how far the document is from average length. At b=0.75 and avgdl=250, a 900-token chunk gets a normalizer of 1.2 * (0.25 + 0.75 * 3.6) = 3.54, versus 0.516 for a 60-token chunk. Seven times the penalty, applied to every query term independently.

K1_DEFAULT, B_DEFAULT = 1.2, 0.75

def tf_component(tf, dl, avgdl, k1=K1_DEFAULT, b=B_DEFAULT):
    """The per-term saturation factor. idf is a separate multiplier."""
    norm = k1 * (1 - b + b * dl / avgdl)
    return tf * (k1 + 1) / (tf + norm)

AVGDL = 250
heading = dict(tf=1, dl=60)    # "Enterprise pricing" section header
table   = dict(tf=3, dl=900)   # the chunk that actually has the answer

for b in (0.75, 0.5, 0.4, 0.3, 0.0):
    h = tf_component(**heading, avgdl=AVGDL, b=b)
    t = tf_component(**table,   avgdl=AVGDL, b=b)
    print(f"b={b:<5} heading={h:.3f}  table={t:.3f}  winner={'table' if t > h else 'heading'}")

b=0.75  heading=1.451  table=1.009  winner=heading
b=0.5   heading=1.261  table=1.146  winner=heading
b=0.4   heading=1.199  table=1.212  winner=table
b=0.3   heading=1.142  table=1.285  winner=table
b=0.0   heading=1.000  table=1.571  winner=table

The crossover is around b≈0.4. Everything above it says a chunk that mentions the term three times in 900 tokens is less about that term than one that mentions it once in 60. For a news corpus that's a reasonable prior — a long article that says "Fed" once is probably not about the Fed. For a chunked technical corpus it's backwards: the long chunk is long because it's a table with rows, and the short one is short because it's a heading with no content.

Why does BM25 length normalization hurt RAG more than web search?

Because avgdl stops being meaningful the moment your chunk lengths are multimodal. BM25 was fit on collections whose lengths are roughly log-normal and unimodal — one hump, a mean that describes the typical document.

A real RAG index has at least three humps: headings and list fragments at 20–80 tokens, prose chunks pinned near your splitter's target of 400–600, and structural blobs (tables, code, API schemas) that blow past the target because the splitter refuses to cut mid-table. The mean lands in a valley between the humps. Every prose chunk gets a mild penalty, every table gets a savage one, and every heading fragment gets a bonus it did nothing to earn.

This also explains a symptom people misdiagnose as "our chunking is bad": lexical recall that gets worse after you enable structure-aware splitting. Keeping tables intact is the right call for generation quality, and it directly lowers those chunks' BM25 scores.

Don't expect hybrid fusion to rescue you either. Mean-pooled dense embeddings dilute the signal over long inputs for the same structural reason — one relevant sentence in 900 tokens moves the centroid less than one relevant sentence in 60. Both arms share the bias, so fusion averages two correlated errors instead of cancelling them.

What is avgdl actually computed over?

Not your corpus. In Lucene it's sumTotalTermFreq(field) / docCount(field) from the current reader — per field, per shard, and computed from segment-level term statistics that still include deleted documents until a merge rewrites them.

Three consequences that bite in production:

1. Shard skew. Elasticsearch defaults to query_then_fetch, which scores locally on each shard with local stats. Two identical chunks on shards with different length profiles get different scores. Use search_type=dfs_query_then_fetch to gather global statistics first, or keep small indices single-shard. The difference is largest exactly where it hurts — small indices, where per-shard sample sizes are tiny.
2. Delete drift. Bulk-reindex a subset and avgdl shifts under you until merges complete. Scores move without any document changing.
3. Field scoping. Norms are per field. If you copy the chunk body into a second analyzed field for a different analyzer, the two fields carry separate avgdl, and a multi_match blends two different length priors.

Why do two chunks of different length get the same score?

Because Lucene doesn't store the exact length. BM25Similarity.computeNorm compresses the field's token count into a single byte with a 4-bit mantissa and 4-bit exponent. Small values round-trip exactly; past a few dozen tokens the buckets widen geometrically. By the time you're at chunk-sized documents, neighboring representable lengths are tens of tokens apart.

Two practical implications. First, micro-tuning b to three decimals is theater — the length signal feeding it is quantized well below that resolution. Second, "trim 20 tokens of boilerplate off every chunk" usually changes nothing, because the trimmed length lands in the same bucket.

One more wrinkle: discount_overlaps defaults to true, so tokens with a zero position increment — synonyms, decompounded forms, word_delimiter_graph output — don't count toward length. If you added a synonym filter to improve recall, you also silently changed the length norm of every document containing a synonym, and therefore its score on all queries.

How should you set k1 and b for chunked corpora?

Start from the length distribution, not from a grid search. Compute the coefficient of variation of your chunk token counts. Under ~0.3, the default is fine. Above ~0.6, avgdl is fiction and you should either fix the chunks or lower b.

My default for a mixed technical corpus: b in 0.3–0.45, k1 around 0.9. Lower k1 saturates term frequency faster, which is what you want when a chunk is a bounded window — the third occurrence of a term inside 500 tokens is weak extra evidence, unlike the third occurrence in a 5,000-word article.

PUT /chunks
{
  "settings": {
    "index": {
      "number_of_shards": 1,
      "similarity": {
        "chunk_bm25": {
          "type": "BM25",
          "k1": 0.9,
          "b": 0.35,
          "discount_overlaps": true
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "body":       { "type": "text", "similarity": "chunk_bm25" },
      "token_count":{ "type": "integer" },
      "chunk_kind": { "type": "keyword" }
    }
  }
}

Index token_count and chunk_kind even if you don't score on them — you need them for the diagnostic below and for splitting length bands into separate fields later.

How do you tell if b is the problem?

Measure rank against length directly. For each query, record the length percentile of the chunks BM25 returns and compare it to the length percentile of the known-relevant chunk:

from scipy.stats import spearmanr

# hits: list of (chunk_id, token_count) in BM25 rank order, per query
def length_rank_bias(all_hits):
    ranks, lengths = [], []
    for hits in all_hits:
        for r, (_, tokens) in enumerate(hits):
            ranks.append(r)
            lengths.append(tokens)
    rho, p = spearmanr(ranks, lengths)
    return rho, p   # rho >> 0 means longer chunks systematically rank worse

A strongly positive rho across queries with varied gold-chunk lengths is your evidence. Then sweep b over {0.75, 0.5, 0.35, 0.2} on a fixed query set and watch recall@50 of the lexical arm alone — not end-to-end answer quality, which is too noisy to attribute at this granularity.

Watch the failure mode on the other side. As b → 0, a 5,000-token appendix that mentions your term once ranks with a title that is exactly your term. If your corpus has genuine long-tail junk documents, b=0 will surface them. The cleaner fix, when you can afford the reindex, is to make chunk lengths uniform enough that avgdl means something again — cap oversized structural chunks by splitting tables on row boundaries with a repeated header, and merge heading-only fragments into the section that follows them. Then the default b=0.75 is defensible again.

So why do long RAG chunks never rank under BM25?

Because BM25 length normalization treats length as evidence of verbosity, and b=0.75 applies that penalty multiplicatively to every matched term. In a chunked corpus, length is evidence of structure instead — a table, a code block, a dense spec section — so the penalty is inverted relative to reality. It compounds with a bimodal length distribution that makes avgdl meaningless, per-shard statistics that make scores non-comparable, and byte-quantized norms that make fine tuning pointless. Fix it by narrowing your chunk length distribution where you can, dropping b to roughly 0.3–0.45 and k1 to ~0.9 where you can't, forcing global statistics with dfs_query_then_fetch, and verifying with a Spearman correlation between chunk length and lexical rank before and after.