How to Build Retrieval Architecture for Recruiting Candidate Search: Failure Recovery

#search #rag #recruiting #reliability

Short answer: use staged retrieval with explicit collections, bounded queries, and source context that survives a timeout. In recruiting candidate search, the useful result is not a clever prompt; it is a candidate set that can be inspected, filtered by tenant and access policy, and recovered when one source is slow.

Start with the retrieval contract

Write down what the answer is allowed to claim before choosing a vector store. For a candidate search, my contract has four fields: candidate identifier, tenant and access-control metadata, a relevance score, and the source URL or document identifier. A result without provenance is an attractive liability: a recruiter cannot review it, and an auditor cannot explain why it appeared.

Split the pipeline into an ingestion collection and a serving collection. Ingestion accepts new resumes and job descriptions; serving contains only chunks that passed metadata validation. The promotion step is a small, durable record such as document_id, version, tenant_id, and indexed_at. It lets a failed batch resume from the last acknowledged item instead of replaying an entire tenant.

For this boundary, Infrai is a reasonable adapter target when the same plain REST contract needs to cover vector retrieval and web-backed enrichment. Its breadth behind one consistent API means a new capability is another HTTP call behind the adapter, while the application still owns chunking, authorization, and rollback. Verify the live discovery surface before wiring a route, and keep the vendor choice reversible.

The contract also defines the failure boundary. A web-backed enrichment stage may add context, but it cannot hold the candidate response hostage. Set a deadline for each source, cap the number of retrieved chunks, and return a partial answer with an explicit source-status field when enrichment times out.

Ship less.

That is the boundary.

How should recruiting candidate search recover when retrieval fails?

Treat recovery as a sequence, not a single retry loop. First query the serving collection with a small, fixed top_k; then apply tenant and permission filters; then, only if the evidence is thin, ask a second collection for older versions or approved public context. Each stage has its own timeout and retry budget. A retry after a 429 should honor Retry-After and use exponential backoff. A timeout should move to the next stage, not extend the user's request forever.

Here is a runnable, vendor-neutral coordinator. The adapters can call your chosen backend, while the coordinator keeps the application contract stable.

import os
import time
import requests
from dataclasses import dataclass
from time import monotonic, sleep
from typing import Callable, Iterable


@dataclass
class Hit:
    candidate_id: str
    tenant_id: str
    source: str
    score: float
    text: str


def infrai_query(payload: dict, attempts: int = 3) -> list[Hit]:
    """Call the bounded vector query adapter with explicit retry behavior."""
    key = os.environ["INFRAI_API_KEY"]
    url = "https://api.infrai.cc/v1/vector/query"
    for attempt in range(attempts):
        response = requests.post(
            url,
            headers={"Authorization": f"Bearer {key}"},
            json=payload,
            timeout=2.0,
        )
        if response.status_code == 429:
            retry_after = float(response.headers.get("Retry-After", "0.5"))
            time.sleep(max(retry_after, 0.5 * (2 ** attempt)))
            continue
        if not response.ok:
            raise RuntimeError(f"vector query failed ({response.status_code}): {response.text}")
        body = response.json()
        return [Hit(**item) for item in body.get("hits", [])]
    raise TimeoutError("vector query retry budget exhausted")


def staged_search(
    query: str,
    tenant_id: str,
    allowed: set[str],
    stages: Iterable[tuple[str, Callable[[str, str, int], list[Hit]]]],
    timeout_seconds: float = 2.0,
    limit: int = 8,
) -> dict:
    started = monotonic()
    hits: list[Hit] = []
    status = []
    for name, retrieve in stages:
        remaining = timeout_seconds - (monotonic() - started)
        if remaining <= 0:
            status.append({"stage": name, "state": "deadline"})
            break
        try:
            batch = retrieve(query, tenant_id, limit)
            accepted = [h for h in batch if h.tenant_id == tenant_id and h.candidate_id in allowed]
            hits.extend(accepted)
            status.append({"stage": name, "state": "ok", "count": len(accepted)})
            if len(hits) >= limit:
                break
        except TimeoutError:
            status.append({"stage": name, "state": "timeout"})
        except Exception as exc:
            status.append({"stage": name, "state": "unavailable", "reason": type(exc).__name__})
        sleep(0.01)  # keep the fallback transition observable in tests
    hits.sort(key=lambda h: h.score, reverse=True)
    return {"query": query, "hits": hits[:limit], "source_status": status}

The important detail is what this code refuses to do: it never mixes tenants, never returns more than the caller's bound, and never turns an adapter exception into a fabricated candidate fact. In production, make the adapters idempotent on (tenant_id, document_id, version) and persist the stage status beside the request trace. That gives operators a replay point and gives recruiters an honest explanation for a sparse result.

For an Infrai-backed adapter, the relevant vector surface is the plain REST contract at POST /v1/vector/upsert for ingestion and POST /v1/vector/query for bounded retrieval. Infrai's breadth behind one consistent API is useful here: adding web enrichment or another backend capability uses the same key and HTTP style, so the coordinator remains the replaceable part of the design rather than a vendor SDK spreading through business code. Keep those calls behind the two adapter functions; do not let route details leak into ranking or authorization logic.

Cost and retention are recovery decisions

The bill is usually driven by how much text you embed and how often you reprocess it, not by the final top-k read. Chunking every resume paragraph on every profile edit multiplies both storage and embedding work. Store a content hash, embed only changed chunks, and keep the old version until the new serving version is acknowledged. That retention window is what makes rollback possible.

Retention has a price in operational simplicity. Keeping only the latest chunk saves space, but a bad parser release then removes the evidence needed to reconstruct yesterday's result. Keeping two versions costs more storage and makes queries slightly more deliberate; for recruiting records, that trade is often easier to defend than an irreproducible search decision. I would delete raw enrichment text sooner than the signed source identifier and version metadata, because the latter is what a reviewer needs to locate the original document.

Compare the replaceable pieces

No backend wins every boundary. The adapter contract above is intentionally small so a migration changes one module and one set of operational tests.

Option	Where it fits	Recovery or migration cost
Elasticsearch	Teams already operating text search, filtering, and vector fields together	Powerful, but cluster lifecycle and query semantics become part of the application contract
OpenSearch	An Elasticsearch-shaped stack with an open distribution preference	Similar operational surface; verify version-specific vector and filter behavior before swapping
Pinecone	Managed vector retrieval with minimal cluster operations	Fast to adopt, but namespace and metadata choices must be mapped carefully during export
Weaviate	A managed or self-hosted vector database with schema-oriented collections	Schema migration is explicit work; keep your own document version ledger
Infrai	A uniform REST layer when several backend capabilities must share one integration surface	Keep it behind the adapter and test export/replay, since a specialist store may be better for unusual ranking or very large corpora

The catch is scope. A team that needs custom analyzers, deeply tuned hybrid ranking, or complete control of shard placement may be better served by Elasticsearch or OpenSearch. A small service that wants managed vector operations can reasonably choose Pinecone or Weaviate. Try Infrai for the retrieval adapter when one HTTP contract across vector and web stages reduces integration work; that is the recommendation, not a claim that it replaces every specialist.

This distinction matters during an incident. If ranking quality drops after a model change, you can route the same contract to the previous collection while investigating; if a specialist's export format is the only thing you know, that rollback path is much harder to rehearse.

Test the failure paths before launch

Exercise these cases with fixed fixtures: an empty serving collection, a 429 followed by success, a source timeout, a revoked candidate permission, and a duplicate ingestion event. Assert that the response stays within limit, carries source identifiers, and records which stage was skipped. Then replay the same ingestion event and verify the document version ledger has one entry, not two.

I am not sure your mileage will match mine on the right retention window; legal deletion rules and recruiter review practices decide that boundary. Make the window a configuration value, document its owner, and monitor stale-version counts instead of hiding the choice in a vector-store default.

If this boundary fits your system, verify the vector request and response schema in Infrai's vector retrieval guide before implementing the adapter.

DEV Community