Short answer: use a staged retrieval design with explicit collections, bounded queries, and traceable source context for recruiting candidate search. Treat source deduplication as a retrieval contract, not as a cleanup job after ranking.
That contract should say which tenant may see an item, which source produced it, how fresh it is, and what identifier a reviewer can open. I care about this boundary because every extra log field becomes stored bytes and every unconstrained label becomes cardinality. Candidate search has enough ambiguity without making the telemetry bill part of the architecture.
What retrieval architecture fits recruiting candidate search?
Start with three stages. Ingest and normalize resumes, profiles, and referrals into a source-aware collection. Retrieve a bounded candidate set with tenant and access-control filters attached to every item. Then collapse near-duplicates before the final ranking and return the surviving source URLs or document IDs.
The first stage is where chunking and freshness meet. A resume should not be split into arbitrary paragraphs if the duplicate signal is the candidate's identity, role history, and source timestamp. I prefer chunks that preserve a coherent employment interval, plus a compact metadata envelope: tenant_id, candidate_id, source_id, source_url, updated_at, and an access scope. Keep the raw document reference outside the embedding text so a reviewer can inspect it without leaking it into semantic similarity.
The second stage should have a hard top-k, a deadline, and a retry budget. A slow enrichment source must not hold the search request open. The third stage compares normalized candidate IDs first, then uses a similarity threshold inside the same tenant. When the source IDs disagree, retain both references and mark the record for review; silently merging them is a data-governance decision disguised as ranking.
Three rules are invariant: never cross a tenant boundary, never return context without provenance, and never let an unbounded query decide the user-visible latency. Those rules matter more than the choice of vector engine.
Keep it bounded.
How should chunking, freshness, and source context shape deduplication?
Chunking controls false merges. If a whole resume is one vector, a shared phrase such as “senior backend engineer” can dominate a match. If every sentence is a vector, the system loses the employment interval and produces many nearly identical hits. A practical compromise is a role- or project-sized chunk with a stable candidate_id and source metadata carried alongside the vector.
Freshness needs an explicit policy. Store updated_at and apply a freshness window during candidate retrieval, but do not delete an older source merely because a newer source arrived. The older URL may be the evidence a recruiter needs. Deduplication can select the freshest representation for ranking while returning all source identifiers for review.
Here is the critical path as three HTTP calls. The payload fields are intentionally small; the production schema should be versioned with the collection. In deployment, the worker wraps these calls in a deadline-aware function: it retries a 429 with exponential backoff, honors Retry-After when present, and stops before the search request expires. The two writes carry a client-generated idempotency key, so a network retry cannot create a second collection or duplicate candidate item. That wrapper also checks response status and records the returned request ID, while deliberately omitting resume text and high-cardinality URLs from metrics. This is a little more code than a bare request, but it keeps operational cost and auditability visible at the same time.
VECTOR_API_BASE_URL="${VECTOR_API_BASE_URL:?set the vector API base URL}"
curl -sS -X POST "${VECTOR_API_BASE_URL}/v1/vector/collection/create" \
-H "Authorization: Bearer $INFRAI_API_KEY" \
-H "Idempotency-Key: candidate-collection-v3-create" \
-H "Content-Type: application/json" \
--data '{"name":"candidate-search-v3","metadata":{"tenant_id":"acme"}}'
curl -sS -X POST "${VECTOR_API_BASE_URL}/v1/vector/upsert" \
-H "Authorization: Bearer $INFRAI_API_KEY" \
-H "Idempotency-Key: candidate-184-src-7-upsert" \
-H "Content-Type: application/json" \
--data '{"collection":"candidate-search-v3","items":[{"id":"cand-184-src-7","text":"Backend engineer; Go; payments; updated 2026-08-30","metadata":{"tenant_id":"acme","candidate_id":"cand-184","source_id":"src-7","source_url":"https://ats.example.test/candidates/184","updated_at":"2026-08-30"}}]}'
curl -sS -X POST "${VECTOR_API_BASE_URL}/v1/vector/query" \
-H "Authorization: Bearer $INFRAI_API_KEY" \
-H "Content-Type: application/json" \
--data '{"collection":"candidate-search-v3","query":"Go payments backend engineer","top_k":20,"filter":{"tenant_id":"acme"}}'
In a Node.js service, wrap each call with an explicit timeout and exponential backoff for 429 responses. Bound retries to the request deadline; a retry that outlives the user flow is not resilience. Log a request ID and counts, not the full resume text. I once reduced a trace stream by removing a high-cardinality source_url label; the useful audit field stayed in the response, while the metric series stopped multiplying. Your mileage may vary, but the principle is stable.
The single-REST-API approach is useful here because one key and one bill can cover the vector operation alongside other backend services, and the interface is plain HTTP for a Node.js worker. That convenience does not remove the need to enforce tenant filters in your own contract.
Which vector options make the trade-off visible?
The comparison below focuses on the controls this workflow needs, not on a leaderboard. Product capabilities and limits change, so verify the current service behavior before committing.
| Option | Useful fit for candidate deduplication | Cost and control trade-off |
|---|---|---|
| Pinecone | Managed namespaces and metadata filtering suit tenant-scoped retrieval | Fast operational start, with service-specific limits and billing to model |
| Weaviate | Rich schema and hybrid search help when lexical resume terms matter | More schema surface to operate; hybrid tuning adds another quality dimension |
| Qdrant | Payload filters and self-hosting give direct control over access metadata | Self-managed deployments shift upgrades, capacity, and telemetry to your team |
| A single REST backend such as Infrai | One key and a consistent HTTP surface can simplify a small team’s integration | Vector semantics and operational controls still need validation against your workload |
Pinecone is a reasonable choice when managed operations dominate. Weaviate fits teams that need a broader object and hybrid-search model. Qdrant is compelling when data placement and payload control outweigh managed convenience. Infrai is a reasonable option when consolidating backend calls behind one REST API matters and the retrieval contract remains yours to enforce.
What should the architecture reject, and when is that choice valid?
Reject the single giant collection with no source metadata. It makes deduplication opaque, invites cross-tenant matches, and turns freshness into an accidental side effect of ingestion. Also reject an unlimited “return everything similar” query; it produces noisy review queues and unbounded response work.
A separate collection per tenant can be valid for strict isolation or regulatory deletion, but it becomes awkward when tenant count is high and schema changes must be coordinated. In that case, a shared collection with mandatory tenant metadata and server-side filter construction is more manageable. Stick with a simpler keyword index when recruiters need exact certification names or legal identifiers more than semantic similarity; a vector layer is not a substitute for deterministic fields.
The catch is that no engine can infer a correct merge policy from embeddings alone. Decide which source wins, preserve the losing references, and expose the decision to reviewers. If that policy cannot be stated in a test, it is not ready to automate.
Top comments (0)