Where keyword search and vector search each fail

Sep 8, 2026 · engineering

We indexed 10,003 real questions that customers asked an online bank and ran the same queries through BM25 and vector search. The failures are instructive in both directions, and they are the whole case for hybrid search. This post shows the failures. The full walkthrough in the docs is runnable end to end: dataset, embeddings, index build, every query shape, and a read-back of the produced files with a plain Parquet reader.

The paraphrase that breaks keyword search

BM25 ranks rows by the exact tokens they share with the query. A user asking about a card that never arrived phrases it their own way, and BM25 anchors on "not shown up" and drifts into refund territory by the second hit:

keyword_miss.py

table.bm25_search("text", "my new card still has not shown up", 5)

17.39 Why has my new card still not come? 15.52 My refund has not shown up on my statement. 15.15 my check or cash deposit has not shown up 14.04 My refund has not shown up in my account. Where is it? 13.42 My statement has not shown my refund.

Vector search matches meaning, so the same paraphrase works even where a hit shares almost no vocabulary with the query:

semantic.py

table.vector_search("embedding", qvec, 5, projection=["text", "score"])

0.117 Why has my new card still not come? 0.165 why have i not got my new card? 0.172 I just got a new card how do I get it to start working? 0.180 where is my new card? 0.181 2 weeks ago I ordered my new card. It isn't here. What should I do?

The token that breaks vector search

Invert the query and the failure inverts with it. On a term of art, BM25 is precise:

keyword.py

table.bm25_search("text", "SWIFT transfer", 5, projection=["text", "score"])

9.46 Is a SWIFT transfer acceptable? 9.46 Can I transfer using SWIFT? 9.46 Can I Transfer from SWIFT? 9.08 Can I do a SWIFT transfer? 8.72 is there a fee for swift transfer

Vector search has no notion of an exact token. A rare identifier, a product name, or an error code can dissolve into its embedding neighborhood, outranked by rows that are merely on topic. A real query stream contains both shapes daily. Whichever single retriever you pick, some of your users lose.

Run both, fuse the rankings

Hybrid search runs both retrievers and merges the two ranked lists with reciprocal rank fusion, so a row both retrievers surface outranks a row only one of them found:

hybrid.py

q = "top up with Apple Pay"
table.hybrid_search("text", q, "embedding", model.encode(q).tolist(), 5,
                    projection=["text", "score"])

0.0323 Is top-up possible with Apple Pay? 0.0320 How can I top up with Apple Pay? 0.0308 Am I able to top up with Apple Pay? 0.0161 My Apple Pay top up with my American Express card is not working? 0.0161 is there a way to do top up with apple pay

The exact-token half of the query (Apple Pay) and the intent half (topping up) each pull their weight. Neither retriever alone ranks this list this way. If you want the concept in depth, what is hybrid search? covers the two retrievers and the fusion math.

All of this ran directly on Apache Parquet files. The index lives inside the files, they stay readable by any plain Parquet reader with Infino nowhere in the read path, and the same code targets S3 by changing one string. The docs guide is the complete walkthrough: install, dataset, embeddings, the index build, filtered search, SQL composition over search results, and the read-back. Inside a Parquet superfile covers how the format carries an index without breaking compatibility.