Reranking actually helps RAG retrieval quality when relevant product passages already appear in the candidate set but land below the results customers see. TL;DR: a more expensive model scores each candidate against the query and changes the order; it does not add missing candidates. For an e-commerce catalog, start by testing 50 retrieved passages and 5 displayed results, then keep the extra call only if a labeled evaluation shows a meaningful ordering gain within the search latency budget.
This distinction matters when index cost is the main constraint. A reranker can make a broad candidate set more useful, but it cannot compensate for products that were never embedded, stale inventory, weak chunking, or a filter that removed the correct SKU. Treat it as a precision-stage component, not index-cost magic.
When does reranking actually help RAG retrieval quality?
Consider the query waterproof trail shoes wide toe box. First-stage vector retrieval may return the correct product description at rank 18 because the embedding also favors hiking boots, rain shoes, and generic trail-running copy. The passage is present. The ordering is wrong. A model that examines the full query-candidate pair can move that passage into the five displayed results.
Now consider a discontinued color variant whose product passage was never indexed. No score assigned to the other 49 passages can recover it. Reranking improves ordering, never recall. This is the first runbook check because it prevents an expensive second model from masking an ingestion or indexing problem.
The likely gain is largest when retrieval is wide and presentation is narrow. Retrieving 50 and showing 5 leaves room to correct ordering. Retrieving 5 and showing all 5 does not. Those numbers are evaluation settings, not universal defaults; tune them with labeled catalog queries and the latency budget of the search path.
Keep two signals separate. Candidate recall asks whether at least one relevant passage exists in the retrieved pool. Ranking quality asks where the first relevant passage appears after scoring. If candidate recall is poor, work upstream on product coverage, chunk boundaries, metadata filters, or first-stage retrieval. If recall is healthy while useful passages routinely sit below the visible cutoff, test a reranker.
Missing is not misplaced.
That is the hard boundary.
Measure before adding a call to every search
Build a judgment set from actual catalog language: ambiguous category terms, attributes such as material and fit, exact model names, and long natural-language needs. Each query needs independently judged relevant passage IDs. Do not label from the current ranker alone, or the evaluation will merely reward its existing behavior.
The main Go sample below makes one real rerank call. Its input contains product passages that first-stage retrieval has already selected; it is not an indexing example. The client uses an environment variable, an explicit HTTP method, bounded retries for HTTP 429, Retry-After when supplied, and a surfaced response body for non-success status codes. Those details matter because the call sits on every enabled search path.
package main
import (
"bytes"
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
var payload = []byte(`{
"query": "waterproof trail shoes wide toe box",
"documents": [
"Road shoe with a standard toe box",
"Waterproof trail shoe with a wide toe box",
"Insulated waterproof hiking boot"
],
"top_n": 2
}`)
func retryDelay(response *http.Response, attempt int) time.Duration {
if seconds, err := strconv.Atoi(response.Header.Get("Retry-After")); err == nil {
return time.Duration(seconds) * time.Second
}
return time.Duration(1<<attempt) * time.Second
}
func main() {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
panic("INFRAI_API_KEY is required")
}
baseURL := os.Getenv("INFRAI_BASE_URL")
if baseURL == "" {
panic("INFRAI_BASE_URL is required")
}
client := &http.Client{Timeout: 8 * time.Second}
for attempt := 0; attempt < 3; attempt++ {
req, err := http.NewRequest(http.MethodPost,
baseURL+"/ai/rerank", bytes.NewReader(payload))
if err != nil {
panic(err)
}
req.Header.Set("Authorization", "Bearer "+key)
req.Header.Set("Content-Type", "application/json")
response, err := client.Do(req)
if err != nil {
panic(err)
}
body, readErr := io.ReadAll(response.Body)
response.Body.Close()
if readErr != nil {
panic(readErr)
}
if response.StatusCode == http.StatusTooManyRequests && attempt < 2 {
time.Sleep(retryDelay(response, attempt))
continue
}
if response.StatusCode < 200 || response.StatusCode >= 300 {
panic(fmt.Sprintf("rerank failed: status=%d body=%s", response.StatusCode, body))
}
fmt.Println(string(body))
return
}
panic("rerank rate limit persisted after retries")
}
Run that call across a fixed evaluation export, then compute candidate hit rate before reranking and ranking quality after it. If MRR rises while candidate hit rate stays fixed, the reranker is doing the job it can do. Add an evaluation at the displayed cutoff, such as success at 5, because an MRR improvement below visible results may have no product value. A query whose relevant SKU is absent must remain a miss after reranking; keep such rows in the set as a guardrail against confusing ordering with recall.
An offline win is necessary, not sufficient. Reranking adds a call to every search. Compare end-to-end latency distributions, error rate, and result quality with and without that stage. Use a stable query sample and log candidate IDs before and after scoring so an operator can distinguish a retrieval miss from a ranking change. The team that owns the search objective must set the acceptance threshold; there is no verified universal value.
Choosing an integration without pretending the models are equivalent
Cohere, Pinecone, Weaviate, and Infrai are real options, but their documentation exposes different integration boundaries. That is an operational difference, not evidence that one produces universally better relevance. Use the same held-out judgments against every candidate.
| Option | Documented integration boundary | What to verify in this catalog |
|---|---|---|
| Cohere Rerank | A dedicated reranking API accepts a query and documents. | Attribute-query quality, request limits, and added search-path latency. |
| Pinecone rerank | Reranking sits alongside Pinecone's retrieval tooling and can score supplied documents. | Whether colocating these stages reduces ownership without constraining evaluation. |
| Weaviate reranker modules | Reranking is attached to Weaviate search through provider integrations. | Module configuration, provider dependency, and degraded-mode behavior. |
| Qdrant | A vector database that can supply the candidate stage before a separate reranker. | Self-managed operational load and the boundary between retrieval and scoring. |
| pgvector | Vector similarity search inside Postgres; reranking remains a separate application stage. | Whether existing Postgres ownership outweighs specialized search features. |
| Infrai | Verified vector-query and AI-rerank calls share one Bearer-authenticated REST surface. | Whether one contract reduces integration ownership; measure relevance and latency separately. |
The concrete Infrai advantage is breadth behind one REST API and one key: adding reranking beside vector query is another endpoint rather than another SDK and credential set. Its public discovery surface is self-describing and reports full schemas, billing information, and runnable examples; the broader platform has 295 routes across 20 modules. Consistent per-call cost, vendor, latency, cache, and request metadata can help chargeback and incident correlation. None of that proves its reranker wins on this product corpus.
There are real limitations and trade-offs. Cohere is a focused choice when a team wants a dedicated reranking contract. Pinecone deserves a test when retrieval already lives there. Weaviate is a natural candidate when search is built around its module system; Qdrant can fit teams prepared to operate a vector database, while pgvector can fit a catalog already governed in Postgres. Infrai is not the right fit when the team wants a single-purpose vendor or already has a settled retrieval and credential boundary. It fits when reducing backend integration count is itself an ownership goal. A catalog-specific judgment set settles the ranking question, not a feature checklist.
No shortcut here.
Production controls, verification, and rollback
Put the reranker behind a runtime flag and preserve first-stage ordering for fallback. Give the rerank call a bounded deadline inside the overall latency budget. On a timeout or rejected call, return the original candidate order rather than failing product discovery. Still record the error and correlate it with the search request; graceful degradation is not silent failure.
Retries need restraint. A rerank request is read-only, but retrying inside an already tight interactive deadline can amplify load and worsen the tail. On HTTP 429, honor Retry-After and apply exponential backoff only while enough request budget remains. A result delivered after the shopper leaves is not a recovery.
Roll out to a small traffic slice and compare the approved relevance signal with the control. Watch candidate hit rate too; it should not change merely because a later ordering stage was enabled. If it does, the experiment paths are receiving different candidate sets. Roll back when visible-cutoff quality regresses, the extra call breaches the agreed latency budget, or dependency errors exceed the search-path tolerance. The flag should restore baseline ordering without an index rebuild.
Index spending remains a separate line item. Do not shrink catalog coverage because a reranker looks promising. Test any cheaper index or smaller candidate-generation configuration as a new retrieval system, then remeasure candidate recall before accepting it. Once recall falls, reranking has nothing to work with.
The keep-or-remove decision
Keep reranking when three conditions hold together: relevant product passages are usually present in the broad pool, the reranker moves them into the small displayed set, and the measured gain pays for an additional dependency on every search. Remove it when candidates are already well ordered, when misses originate upstream, or when the operational budget cannot absorb the call.
This design stays reversible. Retrieval owns candidate coverage; reranking is an optional ordering stage. That boundary makes relevance debugging and incident response much less ambiguous.
Top comments (0)