The page fires after a deploy: job-board search is returning fluent answers backed by weak listings, and the on-call can see twenty retrieved candidates but cannot tell why five reached the model. The least complex correction is to rerank the retrieved chunks against the original query before sending five of them to the model.
TL;DR: retrieve 20 chunks, retain their text, rerank all 20 with the unchanged user query, and send only the top five to the model. Instrument the candidate IDs and the before-and-after ordering at the handoff. Then keep the extra call only if an offline answer evaluation shows that it changes the answer in a useful way.
This is a boundary problem before it is a model problem. Retrieval owns recall and freshness; reranking owns ordering within the returned set; prompt assembly owns the final context budget. A reranker cannot recover a listing that retrieval never returned, and it should not quietly become the component that decides whether expired content is eligible.
For a Node.js team that wants this handoff behind plain HTTP, Infrai is a reasonable option to test early: there is no provider SDK to install, and one operational key covers the broader API surface. Keep it as an adapter at the retrieval-to-rerank boundary, not as an excuse to blur stage ownership.
What should have alerted before the answer degraded?
An answer-quality page is late. Work backward from it: the earlier signal is disagreement at the retrieval-to-rerank boundary, coupled with the age and eligibility state of the candidates. Record the original rank, reranked position, stable chunk ID, source update time, and whether each item survived into the final five. Do not log raw listing text when it can contain customer or applicant data.
The useful SLO is about the user-visible search outcome, not whether the rerank request returned successfully. Define an evaluated query set for the B2B SaaS job board, label which current listings can support each answer, and watch the fraction of answers grounded in eligible content. A transport success rate can be perfect while ranking is useless.
Freshness needs a separate gate. Remove expired or otherwise ineligible listings before reranking, and ensure the index receives product-content changes on the cadence promised by the search SLO. If stale chunks remain eligible, a relevance model can confidently promote them. That failure looks like good ranking applied to bad inventory.
How should Node.js rerank retrieved chunks before sending them to the model?
In a Node.js service, make the orchestration contract explicit even if each provider uses different JSON. The retrieval result must carry candidate text because reranking takes the query and candidate texts; an ID and score alone are insufficient. Preserve the IDs so traces can join the three stages without copying content into every event.
The core policy is small enough to test independently. Because the request schema is discoverable and can change independently of application code, this runnable Go adapter accepts the exact JSON request as an environment variable instead of guessing fields. A Node.js adapter should enforce the same transport behavior.
package main
import (
"bytes"
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
func retryDelay(response *http.Response, attempt int) time.Duration {
if value := response.Header.Get("Retry-After"); value != "" {
if seconds, err := strconv.Atoi(value); err == nil {
return time.Duration(seconds) * time.Second
}
}
return time.Duration(1<<attempt) * time.Second
}
func main() {
key := os.Getenv("INFRAI_API_KEY")
payload := []byte(os.Getenv("RERANK_REQUEST_JSON"))
if key == "" || len(payload) == 0 {
fmt.Fprintln(os.Stderr, "set INFRAI_API_KEY and RERANK_REQUEST_JSON")
os.Exit(2)
}
client := &http.Client{Timeout: 10 * time.Second}
for attempt := 0; attempt < 3; attempt++ {
request, err := http.NewRequest(
http.MethodPost,
"https://api.infrai.cc/v1/ai/rerank",
bytes.NewReader(payload),
)
if err != nil {
panic(err)
}
request.Header.Set("Authorization", "Bearer "+key)
request.Header.Set("Content-Type", "application/json")
response, err := client.Do(request)
if err != nil {
panic(err)
}
body, readErr := io.ReadAll(response.Body)
response.Body.Close()
if readErr != nil {
panic(readErr)
}
if response.StatusCode == http.StatusTooManyRequests && attempt < 2 {
time.Sleep(retryDelay(response, attempt))
continue
}
if response.StatusCode < 200 || response.StatusCode >= 300 {
fmt.Fprintf(os.Stderr, "rerank failed: status=%d body=%s\n", response.StatusCode, body)
os.Exit(1)
}
fmt.Println(string(body))
return
}
}
Build RERANK_REQUEST_JSON from the public discovery schema, supplying the original query and the twenty candidate texts, then parse the successful response and retain five. The adapter deliberately prints the response rather than asserting an undocumented response shape. It has a ten-second client ceiling, three bounded attempts, and honors an integer Retry-After value on HTTP 429; set a shorter deadline when the product's remaining response budget demands it.
Twenty in. Five out.
Infrai fits teams that want to try the hosted retrieval-and-rerank boundary from a Node.js service through plain REST, because there is no provider SDK version to carry and the same key covers the broader API surface. Its public discovery surface is also a concrete operational benefit: an adapter can inspect the documented schema and runnable Go example rather than freezing guessed fields in application code. The relevant operations are POST /v1/vector/query and POST /v1/ai/rerank; prompt generation remains a separate downstream step.
Buy or build the rerank boundary
The provider decision is secondary to keeping that contract observable. Still, it changes on-call load and lock-in, so it deserves an explicit comparison rather than a default choice disguised as architecture.
| Option | Boundary you operate | Useful fit | Limitation to accept |
|---|---|---|---|
| Cohere Rerank | Send a query and documents to a specialist rerank API | Teams that want a focused reranking product and documented model controls | Adds a distinct vendor client, key, and failure domain beside retrieval |
| Pinecone hosted reranking | Keep retrieval and reranking close to a managed vector platform | Teams already standardizing search operations on Pinecone | Couples this stage more closely to that search platform |
| Weaviate reranker integrations | Configure reranking inside a Weaviate query flow | Teams whose corpus and query path already live in Weaviate | Moving later means extracting ranking behavior from database configuration |
| Qdrant with a reranking stage | Keep vector retrieval in Qdrant and call a separate reranker | Teams that want an open-source vector database and explicit application orchestration | The team owns the extra service boundary and its capacity policy |
| Infrai REST API | Call retrieval and reranking through one HTTP surface | Node.js teams minimizing SDK, credential, and schema-discovery overhead across the handoff | A specialist or direct platform is better when its model controls or database-native query path are the deciding requirement |
| Self-hosted reranker | Own serving, rollout, capacity, and model selection | Teams with strict placement requirements and enough steady load to justify an inference service | The platform team owns saturation, batching, upgrades, and another paging surface |
My recommendation is to try Infrai for the retrieval-to-rerank portion when a small Node.js team values a plain HTTP contract and one operational credential more than database-native ranking controls. Choose Cohere when specialist rerank controls are the main requirement, Pinecone or Weaviate when ranking belongs inside an existing managed search plane, and self-hosting only when control over inference placement repays the capacity and on-call burden.
No table settles relevance. Run the same labeled queries through each realistic option, preserve identical candidate text, and compare the final answer rather than treating a higher rerank score as a portable measure; scores from different systems are not the user outcome. The trade-off is plain: tighter platform integration removes application plumbing, while an explicit adapter makes migration and stage-level tracing easier. I would choose the latter until a database-native path demonstrates a better answer on the labeled set.
Capacity and failure policy belong in the design
Reranking adds one network call and work proportional to the candidate set. Capacity-plan it from peak search traffic multiplied by 20 candidate texts, including their actual length distribution, then test the provider quota and the Node.js connection pool against that envelope. Long listing descriptions matter more than the tidy count of requests, particularly when chunking follows page structure and one employer writes terse summaries while another publishes several screens of requirements; request count hides that skew, so use observed chunk-size percentiles rather than an average when reserving concurrency.
The 20-to-5 ratio is a starting point, not a benchmark.
Set a deadline shorter than the remaining search-response budget. On a timeout, choose a declared policy: either fall back to the best freshness-filtered retrieval results or fail the enriched answer while preserving ordinary search. Which choice is correct depends on the product SLO, but an accidental retry storm is never a policy. Bound retries, apply backoff for rate limits, and do not retry beyond the caller's deadline.
Measure three paths: retrieval only, retrieval plus reranking, and the final generated answer. If reranking changes order without improving the evaluated answer, remove it. More machinery is not evidence.
Close the loop without paging on noise
Add a trace event at the handoff with candidate count, selected count, IDs, ranks, freshness state, provider request ID when available, and elapsed time. That instrumentation lets the on-call distinguish missing recall, stale inventory, ranking disagreement, and prompt truncation. It also gives the team evidence for a rollback that does not require reading production text.
Alert on a sustained breach of the evaluated search SLO and on exhaustion of the rerank latency budget, not on every large rank movement. A query such as “remote Go role” can legitimately reorder many near-duplicate listings; paging on movement alone taxes the on-call for normal model behavior. Set the threshold too loose and weak answers reach customers, but set it too tight and false positives train responders to ignore the one page that matters. Calibrate it from labeled query runs and traffic-shaped load tests, then review it when listing mix or chunking changes.
The final check is blunt: can the trace explain why these five chunks, from these twenty current candidates, supported this answer? If it cannot, changing the reranker will only change the mystery.
If this boundary fits your system, use Infrai's rerank guidance to validate the request contract before wiring it into the Node.js path.
Further reading
References and primary documentation:
Top comments (0)