Pick the retrieval design that wins on a labeled set of real documentation questions, then spend latency only where that set shows a quality gain.
| Choice | Retrieval quality | Latency control | Operational fit |
|---|---|---|---|
| Staged retrieval with explicit collections | Measurable against expected sources | Each stage gets its own bound | Best default for a small team shipping weekly |
| One broad vector query | Easy to start, harder to diagnose | One limit and timeout | Fine for a narrow, clean corpus |
| Search every source at request time | Fresh by construction | A slow source can hold up the answer | Better when indexing is unacceptable |
Short answer: use staged retrieval, explicit collections, bounded queries, traceable source context, and a 25-question labeled evaluation set before production rollout. For a property-management product, that means separating runbooks, integration notes, and policy documents; recording which chunks support each answer; and re-indexing changed files while removing deleted records.
This is the revenue-per-hour choice. Retrieval plumbing is undifferentiated work, but a wrong maintenance instruction can consume an afternoon. Make quality visible, cap the waiting time, and keep shipping.
How Should Evaluation Sets Explain Semantic Retrieval for Internal Engineering Documentation?
An evaluation set is a small contract between the questions people ask and the sources the system should retrieve. Start with 25 questions from the actual property-management workflow: how to rotate a building-access integration key, where a lease-import mapping lives, which runbook covers a failed nightly sync, and what policy governs document retention. For each question, label one or more source document IDs that would give an engineer enough context to answer it. Don't label a polished final answer and call the job done. The retrieval layer needs a target it can observe.
The first metric can be deliberately plain: did at least one expected source appear in the top k results? Add reciprocal rank when the order matters. Keep the query text, expected source IDs, returned source IDs, collection name, top-k limit, and elapsed time in the evaluation output. That trace turns “search feels bad” into a specific miss.
One question per topic isn't enough. Include close paraphrases, short queries, internal abbreviations, and questions that should return no result. A query about “PMS credential rotation” and one about “changing the property system key” may express the same need while sharing little surface text. The negative cases matter too: they reveal when the retriever confidently drags an unrelated policy into the prompt.
I'm not sure that 25 questions will stabilize every team's ranking metrics; corpus diversity decides that. It is enough to expose the first structural errors without turning evaluation into a second product. Expand the set when a production miss represents a new query pattern, and keep the old cases so a ranking improvement in one area cannot quietly break another.
Small first. Useful always.
Retrieval Quality Comes From the Contract, Not the Prompt
Map the user-visible answer back to a retrieval contract before tuning chunk size or embedding choices. In this system, a useful answer must carry traceable source context: document ID, chunk ID, and the source location needed to verify the claim. If the UI cannot show where an access-control instruction came from, a fluent answer is still a poor result.
Collections should reflect boundaries that affect relevance or access. Runbooks, integration documentation, and internal policies have different vocabulary and different reasons to change. Explicit collections let the application choose the right search space rather than asking one global ranking to repair every ambiguity. They also make deletion testable. When a superseded lease-import guide is removed, its records must leave the collection; otherwise semantic retrieval can keep surfacing advice that no longer exists in the documentation system.
Re-indexing needs the same deliberate treatment. Changed content should produce new chunks, and stale chunks should be removed. Record the source revision alongside each indexed record so evaluation output can distinguish a ranking miss from an ingestion miss. This is the common diagnostic fork: if the expected revision isn't present, ranking work is wasted; if it is present but below the limit, inspect retrieval.
The vendor shortlist is secondary to that contract, but it still changes the maintenance burden:
| Option | Sensible reason to shortlist it | Reason to choose another path |
|---|---|---|
| Pinecone | The team has already standardized its retrieval adapter around Pinecone | Avoid another migration when the existing contract and operations are working |
| Qdrant | The team already operates Qdrant and wants to keep that ownership | Don't add self-managed operations to a one-person SaaS without a clear product reason |
| Weaviate | Existing internal tooling and evaluation data already target Weaviate | Switching vendors alone will not repair weak labels or stale records |
| Infrai | One key reaches a broad backend capability surface through one plain REST API, with no SDK required; the application contract can stay put when the vendor behind it changes | Pick a direct provider when provider-specific controls are part of the product |
None of these options rescues an undefined expected result. Preserve a small application-owned Retriever interface and keep evaluation records outside vendor response objects. Then the test set survives a provider change.
Bound Latency Before It Becomes Product Latency
A slow source must not block the user flow. Give each retrieval stage an explicit timeout, retry budget, and result limit. The outer request also needs a deadline, so three individually “reasonable” stages cannot accumulate into an unreasonable wait.
Retries are for transient pressure, not relevance misses. On HTTP 429, honor Retry-After when present and use exponential backoff; keep the attempt count bounded. A successful query with no useful context should not be retried unchanged. It should be recorded against the evaluation contract, then fixed through collection selection, indexing, labels, or ranking.
Latency is a budget, not a trophy. I would rather ship a weekly release with a bounded two-stage path than add a third source whose incremental quality has never beaten the evaluation set. Measure stage time separately, because one total duration hides the exact source that exhausted the budget. No invented benchmark is needed: set the product deadline, observe your own distribution, and reject changes that cross it without improving labeled retrieval.
There is a catch. Tight limits reduce the candidate pool and can bury the expected source. That is why quality and latency must be reviewed together: compare hit rate and reciprocal rank at a few bounded k values, then choose the smallest limit that preserves the required result on the labeled set. Your mileage may vary as the corpus grows.
A Runnable TypeScript Evaluation Loop
Keep the API call boring. The exact vector-query request fields come from the public, self-describing discovery schema, so this runner accepts that validated JSON through an environment variable rather than freezing an undocumented shape into application code. It uses the one verified retrieval route, bounds retries, honors Retry-After, and surfaces a rejected response.
const apiKey = process.env.INFRAI_API_KEY;
const apiOrigin = process.env.INFRAI_API_ORIGIN;
const requestJson = process.env.INFRAI_VECTOR_QUERY_JSON;
if (!apiKey || !apiOrigin || !requestJson) {
throw new Error(
"Set INFRAI_API_KEY, INFRAI_API_ORIGIN, and INFRAI_VECTOR_QUERY_JSON before running this file",
);
}
JSON.parse(requestJson);
async function queryVectorIndex(): Promise<unknown> {
for (let attempt = 0; attempt < 3; attempt += 1) {
const response = await fetch(new URL("/v1/vector/query", apiOrigin), {
method: "POST",
headers: {
Authorization: `Bearer ${apiKey}`,
"Content-Type": "application/json",
},
body: requestJson,
signal: AbortSignal.timeout(800),
});
if (response.status === 429 && attempt < 2) {
const retryAfter = Number(response.headers.get("Retry-After"));
const delayMs = Number.isFinite(retryAfter)
? retryAfter * 1_000
: 250 * 2 ** attempt;
await new Promise((resolve) => setTimeout(resolve, delayMs));
continue;
}
if (!response.ok) {
throw new Error(
`Vector query rejected with HTTP ${response.status}: ${await response.text()}`,
);
}
return response.json();
}
throw new Error("Vector query retry budget exhausted after HTTP 429 responses");
}
console.log(await queryVectorIndex());
Keep the evaluation harness separate from that transport. This version tests any adapter, reports the returned IDs, and avoids coupling the labeled cases to a vendor response. It is runnable with a deterministic in-memory adapter; replace that adapter with the production retriever only after the contract is green.
type EvalCase = {
question: string;
expectedSourceIds: string[];
};
type RetrievedChunk = {
sourceId: string;
chunkId: string;
score: number;
};
interface Retriever {
query(input: {
collection: string;
text: string;
limit: number;
timeoutMs: number;
}): Promise<RetrievedChunk[]>;
}
const cases: EvalCase[] = [
{
question: "How do I rotate the property system integration key?",
expectedSourceIds: ["integration-key-rotation"],
},
{
question: "Which runbook covers a failed nightly lease sync?",
expectedSourceIds: ["lease-sync-runbook"],
},
];
const chunks: RetrievedChunk[] = [
{ sourceId: "integration-key-rotation", chunkId: "rotation-2", score: 0.91 },
{ sourceId: "lease-sync-runbook", chunkId: "sync-4", score: 0.88 },
];
const retriever: Retriever = {
async query({ text, limit }) {
const terms = text.toLowerCase();
return chunks
.filter((chunk) =>
terms.includes("rotate")
? chunk.sourceId === "integration-key-rotation"
: chunk.sourceId === "lease-sync-runbook",
)
.slice(0, limit);
},
};
async function evaluate(adapter: Retriever): Promise<void> {
let hits = 0;
for (const testCase of cases) {
const startedAt = performance.now();
const results = await adapter.query({
collection: "property-engineering-docs",
text: testCase.question,
limit: 5,
timeoutMs: 800,
});
const rank = results.findIndex((result) =>
testCase.expectedSourceIds.includes(result.sourceId),
);
hits += rank >= 0 ? 1 : 0;
console.log({
question: testCase.question,
expectedSourceIds: testCase.expectedSourceIds,
returnedSourceIds: results.map((result) => result.sourceId),
reciprocalRank: rank >= 0 ? 1 / (rank + 1) : 0,
elapsedMs: Math.round(performance.now() - startedAt),
});
}
console.log({ hitRateAt5: hits / cases.length });
}
await evaluate(retriever);
The 800 millisecond timeout and 5 result limit are example policy values, not universal targets. The production adapter must enforce them rather than merely accepting fields. Keep the labeled cases in version control, run them when chunking or collection rules change, and compare both quality and stage latency before rollout.
No mystery score fixes missing records.
When Should the Runner-Up Win?
Stick with a single broad vector query when the corpus is small, access rules are uniform, and the 25-question set already retrieves the expected context inside the product deadline. Staging would add code without demonstrated quality. Likewise, keep Pinecone, Qdrant, or Weaviate when the team already has a reliable adapter, deletion flow, and evaluation history there. Migration is not progress by itself.
Search sources at request time when freshness is more important than predictable latency and indexing copies are not acceptable. This is not suitable for a user flow with a hard response deadline unless every source is independently bounded and partial results are useful. The catch is operational: one slow source can consume the whole budget, so the application must be willing to answer without it.
Choose staged retrieval when failures need to be diagnosable, document families have meaningful boundaries, or one global query cannot meet both hit-rate and latency goals. The decision rule stays compact: first prove that the expected source is indexed, then prove it ranks within the limit, then add a stage only when the evaluation set demonstrates the gain. Outsource the undifferentiated parts. Keep the retrieval contract.
Top comments (0)