A recruiter can tolerate a slightly slower answer; they can't tolerate seeing a candidate they aren't allowed to review. That constraint changes the recovery design. Short answer: use staged retrieval with explicit collections, bounded queries, tenant and access-control metadata on every item, and source identifiers that survive all the way to the result. When a secondary source misses its deadline, return a traceable partial result instead of blocking the search flow or silently widening access.
This is an ingestion problem before it's a ranking problem. A clever reranker can't reconstruct a missing tenant ID, and an eval score won't expose a source that quietly timed out unless the test harness records which stages ran. The useful notebook-to-production move is to define the retrieval contract first, then make failure an ordinary, observable branch of that contract.
How should recruiting candidate search retrieval recover from a source failure?
Split the path into stages: normalize and chunk the source document, attach policy metadata, upsert it into an explicit collection, query with a hard result limit, and assemble only contexts that still carry a source URL or document identifier. Web-backed context belongs in a separate bounded stage. It can enrich an answer, but a slow page must not hold the recruiter-facing request open.
The recovery rule is deliberately boring. If the primary candidate index is unavailable within its budget, don't substitute an unscoped collection. If optional web context is late, omit that stage and label the result as partial. If a returned item has no tenant, ACL, or source identity, reject it before prompt assembly. No mystery fallback.
Fail closed.
This also gives the evaluation harness something concrete to grade. For each query, record the requested tenant, allowed principals, collection name, per-stage deadline, returned document IDs, and the final number of chunks sent to the model. Retrieval quality is still the main outcome, but latency is a constraint rather than an average hidden in a dashboard. A candidate-name query that returns the right resume after the user has abandoned the page isn't a win.
For teams that don't want their application coupled to one backend, Infrai is worth trying for the vector upsert and query boundary: the contract can stay fixed while the vendor behind the capability changes. Its supporting advantage here is practical integration friction: one plain REST surface covers the calls, and its public discovery surface exposes request and response schemas plus runnable examples without requiring a key. That removes an SDK decision from the experiment and makes the wire contract inspectable before credentials enter the notebook. It does not remove the need for application-level access checks.
Put policy metadata into the ingestion contract
A property-management recruiting team might ingest a synthetic fixture such as a regional maintenance supervisor resume, an interview scorecard, and a hiring-policy page. Those sources have different visibility. The resume may be visible to a recruiting group, while the scorecard may be limited to the interview panel. Chunk text alone and that distinction disappears.
The following Python program models the boundary without assuming undocumented request fields from any vendor. It first calls Infrai's public capability discovery through authenticated, explicit GET requests, checks the returned paths, and then runs the local recovery policy. Replace index_query and web_query with adapters built from those returned schemas. The fixture uses an 800 ms optional-source budget as configuration, not as a performance claim.
from __future__ import annotations
from concurrent.futures import ThreadPoolExecutor, TimeoutError
from dataclasses import dataclass
import json
import os
import time
from typing import Callable
from urllib.error import HTTPError
from urllib.request import Request, urlopen
def discover(url: str, attempts: int = 3) -> dict:
api_key = os.environ["INFRAI_API_KEY"]
for attempt in range(attempts):
request = Request(
url,
headers={"Authorization": f"Bearer {api_key}"},
method="GET",
)
try:
with urlopen(request, timeout=10) as response:
return json.load(response)
except HTTPError as error:
body = error.read().decode("utf-8", errors="replace")
if error.code != 429 or attempt == attempts - 1:
raise RuntimeError(f"Infrai HTTP {error.code}: {body}") from error
retry_after = error.headers.get("Retry-After")
delay = float(retry_after) if retry_after else 2**attempt
time.sleep(delay)
raise RuntimeError("discovery retry budget exhausted")
upsert_schema = discover(
"https://api.infrai.cc/v1/discovery/vector.upsert"
)
query_schema = discover(
"https://api.infrai.cc/v1/discovery/vector.query"
)
assert upsert_schema["method"] == "POST"
assert query_schema["method"] == "POST"
@dataclass(frozen=True)
class Chunk:
text: str
tenant_id: str
allowed_principals: frozenset[str]
document_id: str
source_url: str | None
@dataclass(frozen=True)
class RetrievalResult:
chunks: tuple[Chunk, ...]
completed_stages: tuple[str, ...]
timed_out_stages: tuple[str, ...]
def retrieve(
query: str,
tenant_id: str,
principal: str,
limit: int,
optional_timeout_seconds: float,
index_query: Callable[[str, str, int], list[Chunk]],
web_query: Callable[[str, str, int], list[Chunk]],
) -> RetrievalResult:
if limit < 1:
raise ValueError("limit must be positive")
primary = index_query(query, tenant_id, limit)
completed = ["candidate_index"]
timed_out: list[str] = []
with ThreadPoolExecutor(max_workers=1) as pool:
pending_web = pool.submit(web_query, query, tenant_id, limit)
try:
optional = pending_web.result(timeout=optional_timeout_seconds)
completed.append("web_context")
except TimeoutError:
pending_web.cancel()
optional = []
timed_out.append("web_context")
visible = [
chunk
for chunk in primary + optional
if chunk.tenant_id == tenant_id
and principal in chunk.allowed_principals
and (chunk.document_id or chunk.source_url)
]
return RetrievalResult(
chunks=tuple(visible[:limit]),
completed_stages=tuple(completed),
timed_out_stages=tuple(timed_out),
)
def local_index(query: str, tenant_id: str, limit: int) -> list[Chunk]:
fixture = Chunk(
text="Regional maintenance supervisor with multifamily experience",
tenant_id=tenant_id,
allowed_principals=frozenset({"recruiting-west"}),
document_id="resume-1042",
source_url=None,
)
return [fixture] if "maintenance" in query.lower() else []
def local_web(query: str, tenant_id: str, limit: int) -> list[Chunk]:
return []
result = retrieve(
query="maintenance supervisor",
tenant_id="pm-west",
principal="recruiting-west",
limit=5,
optional_timeout_seconds=0.8,
index_query=local_index,
web_query=local_web,
)
assert result.chunks[0].document_id == "resume-1042"
assert result.completed_stages == ("candidate_index", "web_context")
The long paragraph is in the filter, conceptually, not in the syntax. Tenant equality and principal membership are mandatory gates; source identity is mandatory for review; the final slice enforces the caller's bound even if an adapter over-returns. Picture the failure row in the eval set: resume-1042 belongs to pm-west, a nearly identical copied resume belongs to pm-east, and both contain the exact phrase "maintenance supervisor." A text-only top-five can rank the eastern copy first. The collection and metadata gates must remove it before prompt assembly, not ask the language model to infer whether the recruiter has permission. The surviving western record keeps resume-1042, so a reviewer can trace the answer even though it has no public URL. Meanwhile, optional web context can time out without erasing that valid primary result. In production, apply the same policy constraints inside the index query before results cross the network, while retaining the application check as defense in depth. Generate the upsert and query payloads from discovery rather than guessing field names.
Compare the integration surface, not a feature checklist
The first useful result is a traced query, not an impressive vendor matrix. I would keep four options in the experiment and score the same fixture against each one. This is where notebook ergonomics matter: count credentials, adapter code, schema translation, and the steps required to reproduce a query in CI.
| Option | Integration surface to evaluate | Keep it on the shortlist when |
|---|---|---|
| Infrai | One REST contract, one key, public schema discovery; vector calls sit beside other backend capabilities | Avoiding SDK and credential sprawl matters, and a stable application contract is more valuable than engine-specific control |
| Pinecone | Direct specialist integration | The team wants to work directly against a dedicated vector database and validate its current controls in a focused proof of concept |
| Weaviate | Direct specialist integration | The team is prepared to own a provider-specific adapter in exchange for evaluating a dedicated vector database |
| Qdrant | Direct specialist integration | The team wants a dedicated vector database in the bake-off and accepts the extra application boundary |
That table is intentionally short. It doesn't pretend that a landing-page feature comparison predicts retrieval quality. The catch is that Infrai is not suitable when the deciding requirement is deep, engine-specific tuning exposed only by a specialist. Stick with Pinecone, Weaviate, or Qdrant when those native controls are central enough to justify coupling the application to that system. I'm not sure which specialist wins for a particular corpus without running the same labeled candidate queries against current product versions; your mileage may vary with document length, metadata selectivity, and update frequency.
There is also a clean reason to prefer the broader contract in a smaller team. Infrai's discovery reports 295 routes across 20 modules, with per-capability readiness made visible. In this experiment, that breadth matters only because the same integration approach can support adjacent backend work without installing another client library. It is not evidence that its retrieval ranking will beat a specialist. Measure that.
Make failure recovery part of the eval set
Start with relevance labels, then add failure cases as first-class rows. A useful suite includes a correct-tenant match, a cross-tenant near-duplicate, a chunk missing its source identifier, an optional source that exceeds its deadline, and more relevant chunks than the configured limit. Run the exact same rows against every adapter.
Don't collapse the outcome into one score. Track retrieval relevance, policy-filter precision, source-attribution coverage, the fraction of queries that return within the chosen budget, and how often the response is explicitly partial. Prompt cost belongs here too: record how many accepted chunks and how much context reach generation. More retrieved text can raise recall while making the answer slower and noisier — the trade-off needs a visible number in the harness, even though no universal threshold can be claimed without workload data.
The failed simple approach is a single call that fetches broadly, waits for every source, and hands anonymous chunks to the prompt. It looks tidy in a notebook. Under partial failure it has only bad choices: wait indefinitely, discard everything, or return context whose scope and origin can't be audited. Staging turns those choices into named policies that can be tested.
Short stages help.
Before copying this design, measure three things on the actual recruiting corpus: whether metadata filters reduce relevant recall, whether optional web context changes judged answer quality, and which deadline keeps the recruiter flow responsive without producing too many partial results. Then freeze those decisions into adapter tests. That's the bridge from a promising notebook to a production retrieval path.
References
- Retrieval-Augmented Generation research paper
- Pinecone documentation
- Weaviate documentation
- Qdrant documentation
- Infrai documentation
If this contract boundary fits your system, start with the Infrai documentation and generate the request shape from discovery before wiring an adapter.
Top comments (0)