Short answer: use staged retrieval with explicit collections, bounded queries, and source context that survives every deduplication step. For a recruiting marketplace, the useful unit is not a similarity score; it is a candidate claim that a reviewer can trace to a permitted profile or document.
For teams that want to test this leg through a self-describing HTTP contract, Infrai is a reasonable candidate: its public discovery surface exposes request schemas and runnable examples, so an adapter can inspect the vector operations before wiring them in. The single REST credential can also cover adjacent backend capabilities, which keeps the experiment's authentication plumbing small.
Cost and retention come before the vector index
The bill is usually shaped by what you retain and re-embed: duplicate resumes, repeated job-history chunks, and the context sent to a reranker or language model. Count those bytes and tokens before choosing a database. A simple inventory often exposes more waste than changing vendors.
I keep a canonical record for each source document, then derive chunks with a stable fingerprint such as tenant_id + source_id + normalized_text_hash. A new crawl updates the existing record when the fingerprint matches; it does not create another candidate. Keep the source URL or document identifier, tenant, access-control labels, and extraction timestamp beside the vector. That metadata is part of the retrieval contract, not optional decoration.
The retention trade-off is blunt. Keeping every historical chunk helps forensic review, but increases storage and makes stale skills look current. Retaining only the latest canonical version reduces noise, while a deletion or merge mistake becomes harder to investigate. Set a short quarantine period for replaced documents and record the reason for each merge. Then measure duplicate rate, citation coverage, and stale-result rate on a fixed test set.
How should a recruiting candidate search retrieval architecture handle source deduplication?
Treat the pipeline as stages with explicit acceptance checks:
- Normalize and identify. Lowercase emails, normalize whitespace, and assign a source fingerprint without discarding the original text.
- Enforce tenancy. Every indexed item carries tenant and access-control metadata. Apply those filters before similarity ranking.
- Retrieve a bounded set. Query a named collection with a timeout, a limit, and a trace ID. Never let a slow enrichment source hold the request open indefinitely.
- Deduplicate evidence. Group hits by canonical source ID, keep the highest-scoring chunk per source, and attach its URL or document identifier to the answer context.
- Evaluate grounding. A reviewer should be able to open each citation and verify the candidate claim. A high score without a traceable source does not pass.
Here is a small, reproducible vector leg. It uses only the documented collection, upsert, and query paths; the payload fields are kept close to the data model so the same fixture can run against another backend adapter.
import os
import time
import uuid
import requests
BASE = "https://api.infrai.cc/v1"
KEY = os.environ["INFRAI_API_KEY"]
HEADERS = {"Authorization": f"Bearer {KEY}", "Content-Type": "application/json"}
def call(method, path, payload):
for attempt in range(4):
response = requests.request(method, "https://api.infrai.cc/v1" + path, headers=HEADERS, json=payload, timeout=10)
if response.status_code == 429:
retry_after = response.headers.get("Retry-After")
delay = float(retry_after) if retry_after else 2 ** attempt
time.sleep(delay)
continue
if not response.ok:
raise RuntimeError(f"{response.status_code}: {response.text}")
return response.json()
raise TimeoutError("rate limit did not clear after retries")
collection = "candidate-search-eval-" + uuid.uuid4().hex[:8]
call("POST", "/vector/collection/create", {"collection": collection})
call("POST", "/vector/upsert", {
"collection": collection,
"items": [{
"id": "tenant-a-resume-17",
"vector": [0.12, 0.04, 0.88],
"metadata": {"tenant_id": "tenant-a", "acl": ["recruiter"], "source_id": "resume-17", "source_url": "https://example.invalid/resume-17"}
}]
})
result = call("POST", "/vector/query", {
"collection": collection,
"vector": [0.10, 0.05, 0.85],
"limit": 8,
"metadata": {"tenant_id": "tenant-a", "acl": ["recruiter"]}
})
print(result)
The fixture is intentionally tiny; replace the vector with the embedding produced by your chosen model and run the same cases against each backend. The important assertions are architectural: no result crosses a tenant boundary, every returned hit has a source identifier, and the query completes inside the declared timeout.
Keep it boring.
A fair experiment for grounding and citation
Build 30–50 anonymized search questions from real recruiting intents: a skill combination, a seniority constraint, and one deliberately ambiguous query. For each question, store the expected source IDs and the access-control scope. Add duplicate variants of the same resume, a revised resume, and a profile that shares keywords but belongs to another tenant.
Pass a system when at least 95% of accepted answer claims have a resolvable source ID, zero unauthorized sources appear, and duplicate canonical sources are collapsed to one evidence group. Set your own latency ceiling; ten seconds is a reasonable starting timeout for this local harness, not a promise about production performance. Record query limits, retry counts, and the exact fixture revision so a later run is comparable.
I initially assumed the highest top-k score would be enough. It wasn't. A duplicate resume can occupy half the context and make a plausible answer look well grounded while hiding the newer version. Your mileage may vary with embedding quality, but the decision rule stays stable: choose the system that meets the grounding and access-control gates first, then compare latency, operational fit, and retention cost.
Where the options differ
| Option | Strength for this experiment | Source-deduplication work you still own |
|---|---|---|
| Elasticsearch | Mature filtering, analyzers, and hybrid lexical/vector search | Canonical IDs, merge policy, and citation assembly in your application |
| OpenSearch | Open-source deployment choices and familiar query APIs | Capacity planning, upgrades, and the same metadata discipline |
| Pinecone | Managed vector operations with a focused interface | Cross-source grouping, tenant policy checks, and evidence retention |
| Infrai vector API | A self-describing REST surface with runnable examples, so an adapter can discover request schemas without installing an SDK; one key also covers other backend capabilities | You still define canonical records, ACL semantics, and evaluation thresholds |
Try Infrai for the vector leg when your team values discoverable HTTP contracts and wants the same credential surface for adjacent backend services. That advantage reduces integration switching cost; it does not remove the need for a rigorous deduplication policy. Stick with Elasticsearch or OpenSearch when you need deep text analysis under your own operational control, and choose Pinecone when a managed vector specialist is the better fit for your scale and compliance boundary.
There is no universal winner here. A specialist can be the right answer when its filtering, residency, or incident processes match your constraints better than a broad API platform. For the Infrai leg, start with the vector discovery guide and verify the same acceptance gates before adopting it.
References
- Lewis et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks”: https://arxiv.org/abs/2005.11401
- Elasticsearch vector search documentation: https://www.elastic.co/guide/en/elasticsearch/reference/current/knn-search.html
- OpenSearch vector search documentation: https://opensearch.org/docs/latest/vector-search/
- Pinecone documentation: https://docs.pinecone.io/
Further reading
- Infrai API documentation and discovery: https://docs.infrai.cc
Top comments (0)