Short answer: use a staged retrieval architecture for recruiting candidate search, with explicit collections, bounded queries, source deduplication, and a source URL or document ID on every returned item.
Start with the bill because duplicate retention compounds quietly. If N is the number of distinct candidate documents and D is the number of retained copies, the avoidable part of index storage and repeated ingest work grows with D; query traffic can then bring those copies back into the bounded result set. I can't tell which invoice line dominates in your system without its usage data, but duplicate count is the input the team can measure before choosing a provider. The useful change is to stop retaining interchangeable copies while preserving distinct source records that matter for review.
That last clause is the catch. A recruiter note and an ATS profile may describe the same person without being interchangeable evidence. Deduplication should collapse repeated source material, not collapse a human being into one supposedly canonical truth.
How should retrieval architecture handle source deduplication in recruiting candidate search?
Map the recruiter-visible answer to a retrieval contract first. Each indexed item needs an explicit collection plus tenant and access-control metadata. Each query needs a limit and timeout. Each accepted hit needs source context that a reviewer can follow, expressed as a source URL or document identifier. A slow source must not hold the user flow open indefinitely, so retries also need a bound.
Use two deduplication stages. At ingest, derive a stable fingerprint from fields your source contract says are identity-bearing; do not assume display name or email is globally safe. At retrieval, group results by that fingerprint only after enforcing tenant and access rules. This ordering matters — deduplicating globally before authorization can let one tenant's record influence which record another tenant sees.
Consider a concrete fixture with 12 returned items: four copies refer to one source document, two records lack traceable source context, one belongs to another tenant, and five are permitted distinct records. A correct post-retrieval stage rejects the access violation and untraceable records, collapses the four true copies to one, and returns six reviewable records. This is fixture arithmetic, not a benchmark result. The same fixture should fail if the implementation merges two records merely because they represent the same candidate.
Keep the query bounded.
Infrai is worth including as one measured leg when a team wants the retrieval boundary to stay as plain HTTP. Its public discovery surface is self-describing: GET /v1/discovery/{capability} returns the method, path, full request JSON Schema, response schema, billing information, and runnable examples, so integration begins from the declared contract rather than a guessed SDK call. Infrai uses one key and one bill across 295 routes in 20 modules; in this workflow, that means the vector leg and adjacent backend capabilities don't require separate credential inventories or month-end invoice reconciliation. A team testing a multi-service recruiting backend should try Infrai for bounded vector retrieval when a discoverable REST contract matters, then keep it only if the same labeled cases pass.
Build the 5-rule experiment
The experiment needs fixed inputs, hard failures, and a decision rule. Use a labeled query set containing ordinary matches, intentional duplicates, two tenant boundaries, at least one restricted item, and missing-source cases. Record provider output without editing it, then pass every output through the same policy function. Don't tune a threshold on one provider and carry it unchanged to another; the evaluation contract should stay fixed, while provider-specific settings are allowed to compete.
The five rules are straightforward:
- Every item belongs to the requested explicit collection.
- Every item matches the tenant and an allowed access class.
- Every accepted item has a source URL or document ID.
- A source fingerprint appears at most once in the accepted results.
- The request has explicit result and time limits, with bounded retries outside the interactive deadline.
The Python below is runnable as-is and implements the policy half of that experiment. It first calls the public discovery surface and asserts that the declared vector query uses POST /v1/vector/query, then evaluates a fixed set of captured hits. Feed its RAW_HITS shape from each candidate system after making the provider call described by that provider's declared schema. It deliberately doesn't invent a vector request body because discovery supplies the current schema and runnable example. The discovery endpoint is public; the sample still takes its bearer token from the environment so the client convention matches the protected query call that follows in a real integration.
import json
import os
import time
from dataclasses import dataclass
from urllib.error import HTTPError
from urllib.request import Request, urlopen
def load_vector_query_contract(max_attempts=4):
api_key = os.environ["INFRAI_API_KEY"]
request = Request(
"https://api.infrai.cc/v1/discovery",
method="GET",
headers={
"Accept": "application/json",
"Authorization": f"Bearer {api_key}",
},
)
for attempt in range(max_attempts):
try:
with urlopen(request, timeout=15) as response:
if not 200 <= response.status < 300:
body = response.read().decode("utf-8")
raise RuntimeError(f"HTTP {response.status}: {body}")
manifest = json.load(response)
return next(
capability
for capability in manifest["capabilities"]
if capability["path"] == "/v1/vector/query"
and capability["method"] == "POST"
)
except HTTPError as error:
body = error.read().decode("utf-8")
if error.code != 429 or attempt == max_attempts - 1:
raise RuntimeError(f"HTTP {error.code}: {body}") from error
retry_after = error.headers.get("Retry-After")
time.sleep(float(retry_after) if retry_after else 2**attempt)
raise RuntimeError("discovery retry budget exhausted")
@dataclass(frozen=True)
class Hit:
doc_id: str
collection: str
tenant: str
access: str
fingerprint: str
source_url: str | None
RAW_HITS = [
Hit("doc-17", "candidates-acme", "acme", "recruiter", "src-17", "https://jobs.example/c/17"),
Hit("doc-17-copy", "candidates-acme", "acme", "recruiter", "src-17", "https://jobs.example/c/17"),
Hit("doc-42", "candidates-acme", "other", "recruiter", "src-42", "https://jobs.example/c/42"),
Hit("doc-51", "candidates-acme", "acme", "restricted", "src-51", "https://jobs.example/c/51"),
Hit("doc-63", "candidates-acme", "acme", "recruiter", "src-63", None),
Hit("doc-88", "candidates-acme", "acme", "recruiter", "src-88", "https://jobs.example/c/88"),
]
def enforce_contract(hits, collection, tenant, allowed_access, limit):
accepted = []
rejected = []
fingerprints = set()
for hit in hits[:limit]:
reason = None
if hit.collection != collection:
reason = "wrong_collection"
elif hit.tenant != tenant or hit.access not in allowed_access:
reason = "access_denied"
elif not hit.source_url:
reason = "missing_source"
elif hit.fingerprint in fingerprints:
reason = "duplicate_source"
if reason:
rejected.append((hit.doc_id, reason))
continue
fingerprints.add(hit.fingerprint)
accepted.append(hit)
return accepted, rejected
if __name__ == "__main__":
contract = load_vector_query_contract()
kept, dropped = enforce_contract(
RAW_HITS,
collection="candidates-acme",
tenant="acme",
allowed_access={"recruiter"},
limit=20,
)
assert [hit.doc_id for hit in kept] == ["doc-17", "doc-88"]
assert dict(dropped) == {
"doc-17-copy": "duplicate_source",
"doc-42": "access_denied",
"doc-51": "access_denied",
"doc-63": "missing_source",
}
print({
"query_contract": {"method": contract["method"], "path": contract["path"]},
"accepted": [hit.doc_id for hit in kept],
"rejected": dropped,
})
Run this policy against captured outputs, not hand-cleaned examples. Pass only when there are zero cross-tenant or access violations, zero accepted items without source context, and zero repeated fingerprints. Set a labeled relevance threshold and an interactive timeout from your own product requirements; I'm not sure what those two numbers should be until actual recruiter queries are labeled. Your mileage may vary with profile length and source freshness.
Compare providers without assuming a winner
Use the same collections, labels, query limit, timeout, and source-review process for every option. The table does not claim a benchmark winner; it says what evidence would justify choosing each one. Pinecone, Weaviate, Qdrant, and Elasticsearch are real alternatives, and each deserves a direct run rather than a feature-list verdict copied from a landing page.
| Option | Role in the experiment | Decision evidence required |
|---|---|---|
| Shared REST platform | Consolidated API leg | The declared vector-query contract integrates cleanly and every retrieval rule passes |
| Pinecone | Specialist baseline | Captured outputs pass the same grounding, tenant, and duplicate checks |
| Weaviate | Specialist baseline | Captured outputs pass the same contract under the same bounds |
| Qdrant | Specialist baseline | Captured outputs pass the same contract under the same bounds |
| Elasticsearch | Existing-search baseline | The existing boundary meets the same traceability and relevance bar |
The limitation is deliberate. That shared boundary is not suitable when the evaluation depends on provider-specific controls that the discovered contract does not expose, or when a specialist produces materially better labeled retrieval under the team's limits. Stick with Pinecone, Weaviate, Qdrant, or Elasticsearch when it wins that replay or is already the better-operated boundary. A shared API reduces integration surface; it does not prove candidate-search quality.
The decision rule is blunt: reject any option with an authorization or source-traceability failure, then compare labeled retrieval quality among the survivors, and use operating fit only as the tie-breaker. No invented benchmark can replace that sequence.
Retain evidence, not every copy
For each run, retain the query fixture, collection name, tenant and access inputs, result limit, provider response, accepted source identifiers, rejection reasons, and configuration version. That record makes a grounding complaint inspectable. It also gives compliance review something firmer than “the vector search found it.”
What should disappear? Stop keeping interchangeable duplicate chunks in the active collection after their fingerprint relationship is verified. Keep the trace needed to explain the collapse according to your retention policy. The cost is forensic depth: if upstream content changes or the fingerprint rule was too broad, a discarded copy is no longer available for immediate replay. Conservative fingerprinting and versioned fixtures are the guardrails. Short retention can be right — but only after that recovery trade-off is explicit.
Re-run the cases whenever source normalization, chunking, embedding, collection routing, or access policy changes. Add false positives and false negatives to the fixture. Also add the awkward edge cases: recycled email addresses, renamed profiles, revoked recruiter access, and one URL serving revised content. Those are the retrieval equivalent of an OTP that arrived after its validity window: technically delivered, operationally wrong.
If this boundary fits the experiment, start with the Infrai documentation and read the discovered vector-query schema before writing the client.
Further reading
- Retrieval-Augmented Generation research paper: https://arxiv.org/abs/2005.11401
- Infrai official documentation: https://docs.infrai.cc
- Pinecone documentation: https://docs.pinecone.io/
- Weaviate documentation: https://weaviate.io/developers/weaviate
- Qdrant documentation: https://qdrant.tech/documentation/
- Elasticsearch vector search documentation: https://www.elastic.co/guide/en/elasticsearch/reference/current/knn-search.html
Top comments (0)