Short answer: build a two-stage retrieval path, then release it only when relevance, latency, and audit checks pass on a frozen candidate-search set. Use broad lexical and vector recall first; spend the latency budget on reranking only for the smaller shortlist.
This is an architecture decision record for a recruiting candidate search system that watches changing job pages and alerts a recruiter when the candidate pool materially changes. The invariant is simple: every alert must be traceable to source text, and a faster result cannot silently lower recall for required skills. I have fought delivery gaps in notification systems before, so I treat an unexplainable alert as a failed release, not a cosmetic issue.
The decision and its boundaries
Candidate profiles arrive as resumes, structured fields, and recruiter notes. Normalize them into a document with stable IDs, then keep the original spans beside the searchable representation. A hybrid first pass (BM25-style lexical matching plus embeddings) handles exact certification names and paraphrased experience. A cross-encoder or rules-based reranker can then inspect only the shortlist.
The boundary matters. Retrieval may return evidence; it must not invent a qualification. The alert worker should carry document IDs, source offsets, model version, and the query snapshot. If any of those are missing, the result is not ready for production.
| Option | Relevance behavior | Latency behavior | Use it when |
|---|---|---|---|
| Lexical only | Strong on exact skills, weak on paraphrases | Predictable and low | Compliance terms and small corpora dominate |
| Vector only | Good semantic recall, can miss rare tokens | Index and query cost vary | Synonyms matter more than exact credentials |
| Hybrid + rerank | Best chance to preserve both signals | Highest tail latency | Recruiting search needs explainable, high-quality alerts |
How should retrieval, candidate search, and release checks work together?
Start with a labeled evaluation set. Include positive matches, near misses, withdrawn candidates, and deliberately difficult queries such as a certification acronym, a misspelled employer, and a requirement hidden in a note. Store judgments separately from production indexes so an index rebuild cannot rewrite history.
The critical path can stay small and inspectable:
from dataclasses import dataclass
from typing import Iterable, Protocol
@dataclass(frozen=True)
class Hit:
candidate_id: str
score: float
evidence: tuple[str, ...]
class Retriever(Protocol):
def lexical(self, query: str, limit: int) -> Iterable[Hit]: ...
def semantic(self, query: str, limit: int) -> Iterable[Hit]: ...
def rerank(self, query: str, hits: list[Hit]) -> list[Hit]: ...
def retrieve(retriever: Retriever, query: str, recall_limit: int = 50) -> list[Hit]:
pool = {h.candidate_id: h for h in retriever.lexical(query, recall_limit)}
for hit in retriever.semantic(query, recall_limit):
prior = pool.get(hit.candidate_id)
if prior is None or hit.score > prior.score:
pool[hit.candidate_id] = hit
shortlist = list(pool.values())
return retriever.rerank(query, shortlist)
The merge is intentionally conservative: a candidate survives if either retriever finds it, and the reranker decides order. In production, cap the pool, enforce a timeout, and record which branch produced each hit. A timeout should produce a visible “no alert” decision with telemetry, never a fabricated match. I don't ship this path until that distinction is observable in a dashboard and asserted in a test.
What belongs in a recruiting retrieval release checklist?
Run the checks against the same query and candidate snapshot used for comparison. The checklist should include:
- Recall at the required cutoff for must-have skills, with separate slices for sparse and dense profiles.
- Precision for alerts, including duplicate profiles and candidates marked unavailable.
- p95 latency for lexical, semantic, reranking, and end-to-end notification stages.
- Evidence coverage: every returned candidate has a source span and a stable document version.
- Change detection: an unchanged page does not emit a second alert; a changed page records the diff that triggered it.
- Privacy and retention review for resume text, embeddings, and recruiter notes.
- A rollback test that restores the previous index and model version without changing candidate IDs.
I ran into this class of failure in notification work: the ranking score looked healthy while the downstream deduplication key had changed. The fix was not a new model. It was making the key, source version, and alert decision part of one test fixture. That fixture also replayed the page watcher across several edits, preserved the old candidate snapshot, checked suppression after each replay, and compared the evidence spans byte-for-byte; without that longer check, a clean ranking report can hide a duplicate alert that wakes a recruiter twice for the same change. Small detail, large consequence.
Keep thresholds explicit in version control. A release passes only when no required slice regresses beyond its agreed tolerance and the latency budget still holds. Your mileage may vary by corpus size and hardware; publish the measurement window with the result so a green number cannot be mistaken for a universal guarantee.
Log query hash, index version, candidate ID, component timings, and the reason an alert was suppressed. Do not log raw resume text by default. Sample the full evidence payload behind access control, and redact phone numbers and email addresses before sending traces to a shared system.
The rejected option is vector-only retrieval with a generous top-k. It is attractive when semantic similarity is the primary concern, but it can bury a rare license number or an exact security clearance term. It remains valid for exploratory talent discovery, where a recruiter can inspect a broad list manually and latency is less strict. It is a poor default for automated change alerts.
The catch is operational: hybrid retrieval plus reranking costs more compute and creates more moving parts. It is not suitable when the team cannot version indexes, labels, and model configuration together. In that case, stick with a lexical baseline until the release process is mature; a simpler system with evidence is safer than an opaque score.
Top comments (0)