TL;DR: retrieve more candidates than the model should see, rerank their text against the original query, and send only the best few into the prompt. For a media archive screening incoming stories for near-duplicates, retrieving 20 and keeping 5 is a reasonable starting shape. Keep the candidate text in the retrieval payload, because IDs alone cannot supply the reranker with the language it must compare.
Start with the storage multiplier, however. The recurring bill is shaped by records ingested, chunks per record, retained versions, and retention days; query volume then multiplies the 20 candidates sent through the second stage. Reranking is justified only when that extra call changes the final duplicate decision. If it does not, remove it.
That is the answer. The harder work is deciding which evidence deserves to exist long enough to be ranked.
Build a retention ledger before tuning relevance
Consider a planning model for a publisher processing 100,000 records per day. These are illustrative inputs, not measured traffic or a benchmark. Four chunks per record held for 30 days produce 12,000,000 live chunks before replicas and metadata overhead. Preserve every version for 180 days and the same simple model reaches 72,000,000 chunks. The dominant term is retained chunk versions. It is easy to spend an afternoon tuning a five-item result set while six months of almost identical revisions continue accumulating in the index, which optimizes the smallest visible number while leaving the archive multiplier untouched. This is why the ledger comes before relevance tuning: it identifies the term that a policy change can actually move.
Count first.
I would put four counters on the operating ledger: records accepted, chunks written, versions retained, and candidates reranked. Each has bounded analytical meaning. Raw article text does not belong in a metric label, and candidate IDs should not become permanent high-cardinality labels merely because they are convenient during debugging. A decision trace can retain the query hash, stable record IDs, version timestamps, original rank positions, selected IDs, and the final near-duplicate decision.
Keep a small sampled diagnostic set containing full candidate text and scores. Sampling creates a real loss: an editor may later dispute an unsampled match, and the compact trace will show which records participated without preserving enough language to reconstruct every nuance. That missing forensic detail is the price of deliberately retaining less. State the sample rule and retention window plainly instead of treating permanent payload logs as free insurance.
Which chunks deserve a fresh embedding?
Chunking and freshness are one decision, not two independent knobs. Small chunks isolate an amended paragraph, but increase vector count and can fill the candidate set with neighboring fragments from the same story. Large chunks reduce cardinality, but an unchanged introduction can dilute the revised passage that distinguishes a correction from a duplicate.
For media records, use stable semantic boundaries already present in the material: headline, summary, and body sections. Associate every chunk with a stable source-record ID and a version or publication timestamp. When one body section changes, re-embed that section rather than manufacturing another complete copy of every unchanged section. Apply the retention policy before reranking so an obsolete revision does not win merely because its wording resembles the incoming story.
Freshness cannot be inferred from semantic similarity. A close match may be yesterday's wire copy, today's correction, or a later report that repeats the same background while changing the event. The application must carry the time and version fields needed to enforce its editorial policy.
Thirty days versus 180 days in the example changes stored chunk count by a factor of six. It also changes what can be investigated later. That trade is measurable without pretending there is one correct universal window: choose the shortest history that satisfies the archive's editorial and dispute requirements, then test duplicate quality inside that boundary.
Should we rerank retrieved chunks before sending them to the model?
Vector retrieval is the recall stage. Retrieving only five candidates asks the vector index to make the final relevance decision before any second stage compares the original query directly with candidate language. Retrieving 20 preserves room for reranking; keeping 5 bounds the context passed to the model.
The distinction matters in a busy news archive. Company names, locations, dates, and legal vocabulary can make different events look close in embedding space. The reranker receives the original query and all candidate texts, so preserve text in the vector-query handoff rather than returning only identifiers. Freshness remains an explicit filter or decision input, not an accidental by-product of rank.
Twenty and five are starting counts, not laws. Evaluate a representative set of labeled near-duplicate pairs and compare the final decision with reranking enabled and disabled. Also count how often a candidate outside the original top five enters the reranked five. If ordering changes while the downstream answer never does, the additional call and its telemetry are work without demonstrated value.
Five is a budget.
Make the HTTP boundary inspectable
Infrai is one option when the desired boundary is plain REST: there is no client SDK to install or library version to maintain, and retrieval can be called from any process that can send HTTP. A second useful property for this workflow is its public, unauthenticated discovery surface, which returns the current request schema and runnable examples. Its live discovery inventory reports 295 routes across 20 modules, but only two application routes matter in this example.
The following curl runner takes request documents built from the current discovery schemas. vector-query.json must request 20 candidates and preserve their text in the returned application handoff; rerank.json must contain the original query plus those candidate texts and request the top 5. The exact JSON fields are intentionally not reproduced here because the supplied discovery schema is the authoritative contract.
set -euo pipefail
: "${INFRAI_API_KEY:?Set INFRAI_API_KEY}"
: "${INFRAI_API_BASE:?Set INFRAI_API_BASE from service configuration}"
call_api() {
route="$1"
body_file="$2"
output_file="$3"
headers_file="$(mktemp)"
response_file="$(mktemp)"
attempt=0
while [ "$attempt" -lt 5 ]; do
status="$(curl --silent --show-error \
--request POST \
--header "Authorization: Bearer $INFRAI_API_KEY" \
--header "Content-Type: application/json" \
--dump-header "$headers_file" \
--output "$response_file" \
--write-out '%{http_code}' \
--data-binary "@$body_file" \
"$INFRAI_API_BASE/$route")"
if [ "$status" -ge 200 ] && [ "$status" -lt 300 ]; then
mv "$response_file" "$output_file"
rm -f "$headers_file"
return 0
fi
if [ "$status" -ne 429 ]; then
cat "$response_file" >&2
rm -f "$headers_file" "$response_file"
return 1
fi
retry_after="$(awk 'BEGIN{IGNORECASE=1} /^Retry-After:/{gsub("\\r", "", $2); print $2}' "$headers_file")"
if [ -z "$retry_after" ]; then
retry_after="$((2 ** attempt))"
fi
sleep "$retry_after"
attempt="$((attempt + 1))"
done
cat "$response_file" >&2
rm -f "$headers_file" "$response_file"
return 1
}
call_api vector/query vector-query.json retrieved.json
# Build rerank.json from the original query and candidate texts in retrieved.json.
# Validate it against the current public discovery schema before this call.
test -s rerank.json
call_api ai/rerank rerank.json reranked.json
Every request has an explicit method, response status is checked, and HTTP 429 triggers bounded exponential backoff while honoring Retry-After when present. Both operations are reads, so retrying them does not double-apply a write. The application adapter between the calls owns the media-specific version cutoff and the mapping from retrieval results to reranking candidates.
This design uses one key for the two stages and exposes per-call cost, vendor, and latency metadata on the platform's native envelope. Record those values as bounded aggregates by operation and status class. Do not attach full query text or every candidate ID as telemetry labels. Consolidating the calls also concentrates trust and billing with one provider, which is a boundary to accept consciously rather than an automatic advantage.
Compare products at the ownership boundary
Reranking quality needs evaluation on the same labeled media set. Before that experiment, narrow the operational choices by asking who should own retrieval, inference, credentials, and upgrades.
| Option | Documented boundary | Sensible fit | Limitation to examine |
|---|---|---|---|
| Infrai | Vector query and rerank behind one REST base URL and key | A small HTTP-only integration across both stages | Provider concentration for trust, billing, and availability |
| Cohere Rerank | A dedicated reranking API | Retrieval already exists and reranking should remain a separate service | Another credential, network call, and contract beside the vector store |
| Pinecone reranking | Reranking in a managed vector-search workflow | Candidate data already lives in Pinecone | Coupling and migration boundaries need evaluation |
| Weaviate reranker integrations | Reranker modules integrated with Weaviate search | Search orchestration should stay close to the database | Module and model-provider configuration becomes database operations work |
| Elasticsearch semantic reranking | Semantic reranking in the Elastic search model | Existing Elastic teams mixing lexical filters with semantic search | Cluster and inference capacity remain part of operational planning |
This table does not rank relevance quality. Cohere's separated API can be attractive when independent replacement is important. Pinecone and Weaviate reduce glue when their databases already own retrieval. Elasticsearch fits teams whose filters, lexical retrieval, and operations already live in Elastic. Infrai fits when a plain REST contract and one credential across retrieval and reranking matter more than keeping those stages with separate vendors.
Run identical query-candidate pairs through the finalists. Score the final near-duplicate judgment, not merely top-one agreement, because a dramatic reorder that never changes the model's answer does not justify another production stage.
Stop keeping data on purpose
Keep current chunks, the limited version history required by editorial policy, stable record and version identifiers, labeled evaluation pairs, and a sampled diagnostic trace. Delete expired chunk versions and redundant request bodies from routine logs. Retain aggregate candidate counts, keep rates, status classes, and rerank usefulness long enough to see drift without preserving every story as observability payload.
Then test the counterfactual. Begin with 20 retrieved candidates and 5 survivors. If reranking improves the final duplicate decision on the representative labeled set, tune chunk boundaries and version retention before increasing candidate count; candidates multiply query work, while versions multiply storage continuously. If it does not improve the decision, drop the stage.
An incident involving an unsampled decision will be harder to reconstruct. Accept that cost knowingly. The alternative is an ever-growing shadow archive inside logs, with its own bytes, cardinality, access surface, and retention obligations.
Delete it deliberately.
Top comments (0)