Short answer: deduplicate candidate records before embedding, keep every source assertion in a provenance ledger, and let retrieval return evidence IDs alongside text. In a recruiting candidate search system, that three-part boundary matters more than choosing a particular vector index. A polished answer built from four copies of the same résumé is still one unsupported answer.
The practical constraint is identity. One person can appear in an ATS export, a recruiter note, a referral feed, and a public profile, each with different spelling and a different freshness timestamp. A source-deduplication job that only compares URLs will miss that cluster; a fuzzy matcher that merges everything with a similar name will erase two different people. I treat the searchable document as a claim about a candidate, not as the candidate itself.
The identity boundary comes before retrieval
Normalize fields at ingestion, but never throw away the raw payload. Unicode normalization, lower-casing email addresses, canonicalizing phone digits, and trimming whitespace are useful transformations; they are not proof of identity. Keep a source-specific key, an observed-at timestamp, and a content hash for each observation. The hash detects a repeated export, while the source key and timestamp preserve a meaningful update.
For each pair of observations, classify the relation instead of returning a binary duplicate flag:
| Relation | Search representation | What must remain visible |
|---|---|---|
| Byte-equivalent copy | One canonical chunk | Every contributing source ID |
| Same candidate, changed field | One entity, versioned claims | Which source asserted each value |
| Similar person, uncertain match | Separate entities | The unresolved match decision |
| Conflicting claim | One entity, competing claims | Confidence, timestamp, and source |
This distinction prevents the most expensive failure mode: collapsing a correction into a duplicate. Consider a candidate imported on Monday from an ATS export, copied on Tuesday into a recruiter note, and refreshed on Wednesday by a referral feed. The three payloads can be 92 percent textually similar while carrying different contact details and different evidence dates. If a résumé says “security clearance: active” and a later verified source says “expired,” the newer claim should influence ranking, but the older claim must remain auditable. A naïve nightly job often hashes only the rendered résumé, sees a match, and deletes the Wednesday observation; the ranking then looks stable while the citation points at stale text. The safer flow writes all observations first, links a possible copy in a relation table, and rebuilds the canonical projection from those links. That extra write is cheap compared with explaining to a recruiter why an answer cites a record that was already superseded. Durability and searchability are separate promises.
Duplicates lie.
I use an explicit identity score only to create a review queue. A deterministic key such as a verified email can auto-link; a name plus employer should not. Your mileage may vary on thresholds because the acceptable false-merge rate depends on whether a recruiter is browsing leads or making a compliance-sensitive decision. Measure both false merges and missed links against labeled pairs.
How can a Node.js retrieval pipeline preserve candidate source evidence?
The pipeline can be implemented with ordinary HTTP workers and a durable relational store. Node.js is a reasonable orchestration choice when feed adapters, queue consumers, and search calls already share one runtime; it doesn't solve entity resolution by itself. The important contract is that an indexed chunk carries entity_id, claim_id, source_id, observed_at, and a stable content_hash.
Here is a small Python reference for the ledger write and deduplication decision. It is deliberately storage-agnostic, so the same record can be persisted in Postgres and projected into a lexical or vector index.
from dataclasses import dataclass
from hashlib import sha256
from typing import Literal
Relation = Literal["copy", "update", "uncertain", "conflict"]
@dataclass(frozen=True)
class Observation:
entity_id: str
source_id: str
source_record_id: str
observed_at: str
text: str
verified_identity: bool = False
@property
def content_hash(self) -> str:
normalized = " ".join(self.text.split()).casefold()
return sha256(normalized.encode("utf-8")).hexdigest()
def classify(observation: Observation, known_hashes: set[str], same_entity: bool) -> Relation:
if observation.content_hash in known_hashes:
return "copy"
if same_entity and observation.verified_identity:
return "update"
if same_entity:
return "conflict"
return "uncertain"
incoming = Observation(
entity_id="candidate-1842",
source_id="ats-export",
source_record_id="ats-7781",
observed_at="2026-08-21T09:30:00Z",
text="Python engineer; security clearance active",
verified_identity=True,
)
relation = classify(incoming, {"previous-hash"}, same_entity=True)
print(incoming.content_hash, relation)
The example does not claim that a hash proves two people are the same. It only makes repeated content cheap to detect. Entity matching remains a separate, reviewable decision. Store the decision as data, with the matcher version and the reviewer or rule that produced it. When a model is re-run, you can rebuild the search projection without rewriting history.
At query time, retrieve a small set of canonical chunks, then attach citations from the ledger. Deduplication should reduce repeated evidence, not reduce the citation set to one favored provider. If three sources contributed the same sentence, expose the canonical claim plus three source IDs; if they disagree, retrieve the competing claims and make the answer-generation layer state the disagreement.
Failure modes that look like good search
The obvious metric, top-k relevance, is too forgiving. A duplicate cluster can occupy half of the top ten and make recall look healthy while diversity and provenance collapse. Track entity recall, unique-source coverage, conflict recall, and unsupported-answer rate separately. A candidate search result that cannot show why a skill was included is not grounded enough for a recruiter to trust.
Four production checks catch most surprises:
- Duplicate amplification: count unique entities and source records in every result page. A sudden ratio change usually means an ingestion or normalization change.
- Temporal leakage: run evaluation at a fixed observation time. A later résumé revision must not appear in an earlier test case.
- False merge: maintain a labeled set of hard pairs, including common names and shared employers. Treat a false merge as a higher-severity error than a missed link when the workflow has compliance implications.
- Citation drift: verify that each returned evidence ID still resolves to the stored payload and its source timestamp.
Keep the index snapshot ID, embedding configuration, matcher version, and ledger sequence with each evaluation run. Otherwise a score cannot explain which corpus it measured. I would rather have a smaller, reproducible index than a larger one whose contents cannot be reconstructed after a feed replay.
Choose a rollout that names the loss
Shadow the new projection beside the current search for at least one complete feed cycle. Compare the same queries, not two randomly sampled batches. Promote only when hard gates pass: no forbidden temporal leakage, a bounded false-merge rate, and citation resolution above the team’s declared threshold. Then canary one recruiter workflow and watch duplicate amplification, conflict frequency, and p95 retrieval latency.
The catch is that source deduplication is not suitable when every raw observation must be independently searchable for legal discovery or when identity labels are unavailable and cannot be reviewed. Keep separate records in that case, add a result-level diversity rule, and accept the index and review cost. Conversely, aggressive merging is appropriate for high-volume browsing where a repeated syndicated profile creates more harm than a slower audit path, provided the original claims remain durable.
Do not make price the decision rule. The durable ledger, deterministic replay, and evidence-bearing result contract are the real engineering costs and benefits. A different stack may be a better fit when your team needs managed identity resolution, strict residency controls, or a specialized graph query; the boundary and tests stay the same.
The architecture is therefore plain: preserve observations, classify relationships, index canonical claims, and cite the ledger on every answer. Retrieval quality is only credible when the system can show which source made a claim and which duplicates were intentionally collapsed.
References
- Lewis, Patrick et al. “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.” https://arxiv.org/abs/2005.11401
- RFC 3986, “Uniform Resource Identifier (URI): Generic Syntax.” https://www.rfc-editor.org/rfc/rfc3986
- Unicode Standard Annex #15, “Unicode Normalization Forms.” https://www.unicode.org/reports/tr15/
Top comments (1)
Keeping source observations separate from the canonical projection is the essential choice. I would make retrieval return both the current claim and any superseded conflicting claims when the query touches a compliance-sensitive field, with timestamps and source IDs. That lets a reviewer see that a ranking changed because of newer evidence rather than assuming the old record disappeared.