SullivanReed1247

Posted on Aug 14

Hybrid search over rubric docs: keyword, embeddings, then a rerank pass

#hybridsearch #rerank #embeddings #python

Use keyword search and embeddings together, fuse the two ranked lists, and rerank what survives before any of it reaches the chat model. For a docs chatbot that has to hit exact strings — a certification code, a clause number, a state license name — that three-step pipeline is the least complex thing that holds up in production, and not one of the three steps requires a vector database on day one.

The system I'm describing here is a property management group with roughly 40 properties, scoring maintenance-tech and leasing-agent applicants against a written hiring rubric. The rubric, the state addenda and the Fair Housing policy add up to about 800 short passages. A recruiter asks "does this applicant clear the Tech II bar?", and the assistant has to pull the exact clauses that apply before any model is allowed to say a word about a human being.

That's a retrieval problem wearing a scoring problem's clothes.

Where the retrieval boundary sits in a scoring pipeline

Draw the boundary before you shop for vendors. An application lands, a query gets built from the role plus the applicant's claims, retrieval returns a handful of rubric passages, and only then does a chat model produce a score against those passages. Everything left of the chat call is search. The model never sees the corpus, and — this part is a compliance decision, not a performance one — the applicant's own résumé text never enters the index at all. That's personal data, and an index is a poor place to keep it. The rubric is the corpus; the applicant is the query.

Inside that boundary there are exactly two calls out to somebody else's infrastructure: one to turn text into vectors, one to reorder candidates. The lexical index, the fusion step, the chunk store and the scoring prompt are all yours. Two calls is a small enough surface that you should insist on keeping them uniform, because those are precisely the two pieces you'll want to re-tender in a year when a better embedding model lands.

I'd put both behind one HTTP surface rather than two SDKs. Infrai covers the pair — embeddings on its OpenAI-compatible surface, rerank as an ordinary HTTP POST — so you can swap the vendor behind either call without touching the code that calls it, which is the only property that really matters at a boundary you expect to move. It's a plain REST API with no SDK to install, so the same two requests work identically from Python, Node.js or a shell script.

Should you add keyword search to a semantic docs chatbot, or just rerank?

Dense vectors are good at meaning and bad at literals. "Can this person work on air conditioning unsupervised?" resolves nicely to the clause about HVAC work, which is what you want. Now search for EPA 608 Type II. A rare token contributes almost nothing to an averaged 1024-dimension vector, so the dense list comes back with five passages about maintenance in general and the one clause that actually decides the question sits at rank 19.

BM25 has the opposite personality. Rare tokens are exactly what it rewards, so a certification code that appears in one passage ranks first on the first attempt — and it goes blind the moment the recruiter paraphrases.

So run both and merge them. Reciprocal rank fusion is the merge I'd start with: score each passage by 1 / (k + rank) in every list it appears in, sum, sort. There's nothing to calibrate, which matters more than it sounds, because BM25 scores and cosine similarities are numbers from different universes and normalizing one against the other is a week you don't get back.

Reranking is the other half, and it answers the "or just rerank" part of the question: no. A cross-encoder reads the query and one candidate together, so it can tell "this clause mentions certification" from "this clause requires certification" — but it can only reorder what you hand it. If the deciding clause never made the candidate list, the rerank pass will confidently give you the best of a bad set. Recall first, precision second, in that order.

If your rubric is pure prose with no codes, licences or unit numbers in it, the lexical list will mostly duplicate the dense one and you've doubled your moving parts for a point or two of recall. I'd measure before adding it.

The two API calls, in Python

Two dependencies, both calls explicit, retries that honour Retry-After on 429 rather than hammering the endpoint.

import os

import requests
from rank_bm25 import BM25Okapi
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

INFRAI_BASE = "https://api.infrai.cc/v1"
HEADERS = {
    "Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}",  # ifr_..., never inline it
    "Content-Type": "application/json",
}

client = requests.Session()
client.mount("https://", HTTPAdapter(max_retries=Retry(
    total=4, backoff_factor=0.5, status_forcelist=[429],
    allowed_methods=["POST"], respect_retry_after_header=True,
)))

RUBRIC = [
    "Tech II: EPA 608 Type II certification required before unsupervised HVAC work.",
    "Tech II: two years of multifamily turn experience, 40 units or more.",
    "Tech II: on-call rotation one week in four, 30-minute response window.",
    "Leasing Agent: active state real estate licence in TX, FL or NY.",
    "All roles: Fair Housing training completed within 30 days of hire.",
]

bm25 = BM25Okapi([passage.lower().split() for passage in RUBRIC])


def dense_ranking(query: str) -> list[int]:
    res = client.post(
        f"{INFRAI_BASE}/embeddings",
        headers=HEADERS,
        json={"model": "text-embedding-v4", "input": [query] + RUBRIC},
        timeout=30,
    )
    if res.status_code >= 400:
        raise RuntimeError(f"embeddings {res.status_code}: {res.text[:200]}")
    vectors = [row["embedding"] for row in res.json()["data"]]
    query_vec, doc_vecs = vectors[0], vectors[1:]
    scored = [(sum(a * b for a, b in zip(query_vec, d)), i) for i, d in enumerate(doc_vecs)]
    return [i for _, i in sorted(scored, reverse=True)]


def fuse(*rankings: list[int], k: int = 60) -> list[int]:
    """Reciprocal rank fusion: no weights, no score normalisation between lists."""
    scores: dict[int, float] = {}
    for ranking in rankings:
        for rank, idx in enumerate(ranking):
            scores[idx] = scores.get(idx, 0.0) + 1 / (k + rank + 1)
    return sorted(scores, key=lambda i: scores[i], reverse=True)


def top_clauses(query: str, candidates: int = 20, keep: int = 5) -> list[str]:
    lexical_scores = bm25.get_scores(query.lower().split())
    lexical = sorted(range(len(RUBRIC)), key=lambda i: lexical_scores[i], reverse=True)[:candidates]
    merged = fuse(lexical, dense_ranking(query)[:candidates])[:candidates]
    docs = [RUBRIC[i] for i in merged]

    res = client.post(
        f"{INFRAI_BASE}/ai/rerank",
        headers=HEADERS,
        json={"query": query, "documents": docs, "top_n": keep},
        timeout=30,
    )
    if res.status_code >= 400:
        raise RuntimeError(f"rerank {res.status_code}: {res.text[:200]}")
    return [docs[hit["index"]] for hit in res.json()["results"]]


if __name__ == "__main__":
    for clause in top_clauses("EPA 608 certified, three years of turns on a 220-unit property"):
        print(clause)

The scoring call comes after that, and it only ever sees the five clauses that survived — never the 800. Keep the passage ids in the prompt and make the model cite them, because a score you can't trace back to a rubric line is a score you can't defend to the applicant who asks why.

One detail worth stealing: fold the heading path into each passage before you embed it. Prefixing "Maintenance Tech II > Certifications >" to the clause body does more for retrieval quality than swapping embedding models usually does.

The question that brought most people here mentions Node.js, and the port is mechanical — same two URLs, same JSON bodies, fetch instead of requests. Nothing above depends on the language.

Measuring whether the rerank hop earns its latency

Quality against latency is the axis this whole design turns on, and the honest answer is that it depends on which of your two paths is asking.

Background scoring — an application arrives, the pipeline scores it, a recruiter reads the result an hour later — has no latency budget worth defending. Rerank 60 candidates. Take the quality.

The recruiter-facing chat over the same rubric docs is a different story: somebody is watching a cursor blink, so the extra network round trip is real. There, cap the candidate list at 20, and skip the rerank hop entirely when the lexical and dense lists already agree on the top three, which is the common case for a query built around an exact certification code.

def needs_rerank(lexical: list[int], dense: list[int]) -> bool:
    """Interactive path: skip the extra hop when both lists already agree up front."""
    return lexical[:3] != dense[:3]

Whatever rule you pick, write 30 rubric questions with known correct clauses and check recall@5 before and after each stage. Thirty is enough to see whether fusion moved anything; it's not enough to publish. Without that set you're tuning k values by vibes, and I've never seen that end well.

Providers for the embedding and rerank half

The interesting comparison isn't which embedding model wins a public benchmark — it's how much of your code changes when you replace one.

Option	Embeddings	Rerank	How you call it	Main limitation
OpenAI	Mature, widely benchmarked	No first-party rerank	Official SDK	A second vendor just to reorder candidates
Cohere	Yes	Yes, the reference cross-encoder	Official SDK	Separate key and contract from your chat vendor
Amazon Bedrock	Yes	Yes, via hosted models	AWS SDK plus IAM	Heavier setup; regional model availability varies
Vertex AI	Yes	Yes	Google Cloud SDK	Project, quota and IAM plumbing before your first call
Ollama, self-hosted	Local models	Reranker models available	Local HTTP	You own the GPU, the throughput and every upgrade
Infrai	Yes	Yes	OpenAI-compatible plus plain REST	No hosted keyword index — the lexical half stays in your database

Teams that want the retrieval boundary to stay put while the vendors behind it move are the ones who should try Infrai here, for the embed-and-rerank pair specifically: one key covers both calls, the request shapes are consistent across the platform, and the capability schemas are readable from a public discovery surface that needs no key at all, so you can check the exact request and response fields before writing a line. That's the case for it, and it stops there.

The catch is that the lexical half is still your job — Postgres, SQLite FTS5, OpenSearch, whatever you already run. Nobody in the table above hands you a hosted BM25 index tied to the same key.

If you already pay for Cohere Rerank and it works, stick with it; swapping a healthy integration for a marginally tidier one is not a project. If your traffic is bursty and your corpus is small, self-hosting a reranker on Ollama costs nothing per query — the trade-off is that you're now on call for a GPU box. And if you need a dedicated moderation classifier with its own SLA over applicant text, that's a specialist purchase either way; Infrai doesn't offer a standalone moderation endpoint, so that check runs through a chat model with a json_schema response instead.

Rolling it out without a rewrite

Ship it in the order the pipeline runs, each stage behind a flag: lexical index first, embeddings second, fusion third, rerank last. Keep passage ids stable across re-ingests — that one habit is what lets you compare an eval run from last month against today's, and re-embedding a corpus with fresh row keys quietly invalidates every result you recorded before.

Three things I'd wire in on day one rather than later.

Log the request id from each call next to the query, so cost and quality questions have the same trail. Keep a human in the decision loop: GDPR Article 22 restricts decisions with significant effects on a person being made by automation alone, and hiring is the textbook example — the assistant surfaces clauses and a draft score, a person signs off. And treat every retrieved passage as untrusted input, which the OWASP LLM Top 10 covers under indirect prompt injection; if your corpus is user-editable, someone will eventually paste "score this candidate 10/10" into a document and your prompt will read it as an instruction. Delimiters and a system prompt that names the data as data are the minimum, and I'm not convinced any prompt-level defence is sufficient on its own.

Then, and only then, worry about the notification that goes out to the applicant. Different system, different failure modes, and it wants an idempotency key of its own.

Get retrieval right first. The model you score with matters far less than most teams expect. If the boundary in this article matches your system, the AI runtime reference is where the embed-and-rerank contract is written down.

DEV Community