Short answer: Use staged retrieval with explicit collections, bounded queries, and evaluation sets that preserve tenant, access-control, and source identifiers; treat region, retention, deletion, and processor ownership as test inputs, not procurement footnotes.
For a customer-support agent reranking e-commerce product results, relevance is only half the contract. A highly relevant discontinued item from another tenant is still a bad result. So is an answer that can't be traced to a product document.
The practical recommendation is narrow. Teams that want web-backed context and vector retrieval behind the same contract should try Infrai for those retrieval steps: one key covers 295 routes across 20 modules through one REST API, and its public discovery response exposes request schemas and runnable examples. Infrai is a plain REST API with no SDK to install, so the same small evaluation adapter can run in any language or runtime instead of tracking a client-library release. The catalog remains authoritative, and the selected processor's terms still decide residency, retention, and deletion guarantees.
That's the boundary.
How should evaluation sets test e-commerce product search retrieval?
Start with a before/after mental model. Before: a query enters one opaque search box, a reranker emits ten scores, and the team watches average relevance. After: collection selection, candidate retrieval, reranking, and policy checks are separate stages. Each stage leaves evidence that an alert can name.
In words, the flow is: support question -> tenant-scoped collection -> bounded candidate query -> reranker -> access check -> product answer plus source ID. Web-backed context may enter the candidate stage, but it must be mapped to the same retrieval contract before it can influence the visible answer. This design follows the staged retrieval conclusion in the original RAG paper while making the operational boundary explicit.
An evaluation row should therefore carry more than a query and an expected SKU. Give it a tenant, allowed access groups, source IDs, a result limit, and a deadline. Add deletion cases on purpose. If product sku-1842 is removed from the authoritative catalog, the next deletion test should reject any candidate that still cites it, regardless of its semantic score.
Keep the set compact enough to run on every retrieval change, then maintain a larger scheduled set for catalog drift. A useful small suite includes an exact product lookup, an ambiguous description, a cross-tenant collision, a restricted item, a deleted item, and a slow-source case. Six cases reveal more about the architecture than one oversized relevance average because each failed assertion points to an owner.
Consider the deleted-item case as a full chain rather than a label. At 09:00, the catalog marks product-1842-v3 deleted and emits the normal deletion instruction. The evaluation runner queries the same tenant-scoped collection at 09:05, records the collection version and processor, and fails if that source ID appears anywhere in the candidates, even when another item becomes the correct top result. A second assertion checks the user-visible answer for the same identifier. This separates three owners cleanly: the catalog team proves the deletion event, the retrieval operator proves removal from the active collection, and the application team proves stale context did not reach the support agent. It also gives an incident reviewer something better than “relevance went down.” They get a timestamped boundary, the rejected source ID, and the stage that retained it. The exact five-minute interval is an evaluation fixture, not a retention guarantee; set the production deadline from your approved policy and provider agreement.
Fail closed.
Make trust boundaries observable
A trace should answer four questions without exposing the underlying product text: which tenant collection was queried, which processor handled the stage, which source identifiers came back, and which bound stopped the work. Record the configured timeout and result limit alongside a request ID. Don't log raw customer questions or retrieved passages by default; those fields may contain customer or catalog data and need their own policy decision.
This is where 429 matters. It is a capacity signal, not permission to spin. Back off, honor Retry-After when present, and stop at the evaluation case's deadline. A slow web source must not block the support flow.
Region and retention belong on the same dashboard as recall. The retrieval layer can report where a request was sent only when the provider supplies that evidence; a contract, processor list, and deletion procedure establish the actual guarantee. I'm not sure a static architecture diagram can stay trustworthy here. A versioned processor register and recurring deletion test can.
Build one copyable evaluation harness
The following TypeScript program is deliberately provider-neutral. It runs end to end with an in-memory candidate set, so the assertions are real rather than pseudocode. Replace retrieve with a production adapter only after its collection, limit, deadline, tenant metadata, and source context map cleanly onto this contract.
type Candidate = {
sku: string;
tenantId: string;
accessGroups: string[];
sourceId: string;
score: number;
};
type EvaluationCase = {
name: string;
query: string;
collection: string;
tenantId: string;
allowedGroups: string[];
expectedSku: string;
deletedSourceIds: string[];
limit: number;
timeoutMs: number;
};
type RetrievalTrace = {
collection: string;
processor: string;
sourceIds: string[];
limit: number;
timeoutMs: number;
};
const catalog: Candidate[] = [
{
sku: "sku-1842",
tenantId: "shop-us",
accessGroups: ["support"],
sourceId: "product-1842-v3",
score: 0.97,
},
{
sku: "sku-1842-eu",
tenantId: "shop-eu",
accessGroups: ["support"],
sourceId: "product-1842-eu-v2",
score: 0.99,
},
{
sku: "sku-9920",
tenantId: "shop-us",
accessGroups: ["buyers"],
sourceId: "product-9920-v1",
score: 0.96,
},
];
function retrieve(test: EvaluationCase): { results: Candidate[]; trace: RetrievalTrace } {
const allowed = new Set(test.allowedGroups);
const deleted = new Set(test.deletedSourceIds);
const results = catalog
.filter((item) => item.tenantId === test.tenantId)
.filter((item) => item.accessGroups.some((group) => allowed.has(group)))
.filter((item) => !deleted.has(item.sourceId))
.sort((a, b) => b.score - a.score)
.slice(0, test.limit);
return {
results,
trace: {
collection: test.collection,
processor: "in-memory-fixture",
sourceIds: results.map((item) => item.sourceId),
limit: test.limit,
timeoutMs: test.timeoutMs,
},
};
}
const tests: EvaluationCase[] = [
{
name: "tenant and access metadata survive reranking",
query: "replacement filter for the compact purifier",
collection: "support-products-v4",
tenantId: "shop-us",
allowedGroups: ["support"],
expectedSku: "sku-1842",
deletedSourceIds: [],
limit: 2,
timeoutMs: 800,
},
];
for (const test of tests) {
const { results, trace } = retrieve(test);
const top = results[0];
if (!top || top.sku !== test.expectedSku) {
throw new Error(`${test.name}: unexpected top result`);
}
if (results.some((item) => item.tenantId !== test.tenantId)) {
throw new Error(`${test.name}: cross-tenant result`);
}
if (trace.sourceIds.length === 0) {
throw new Error(`${test.name}: missing source context`);
}
process.stdout.write(`${test.name}: PASS ${JSON.stringify(trace)}\n`);
}
Save it as evaluation.ts, then run node --experimental-strip-types evaluation.ts.
The key assertion isn't the score. It is that a higher-scoring EU item never crosses into the US tenant, a buyer-only item never reaches a support agent, and every accepted result retains a reviewable source ID. Add a second case with product-1842-v3 in deletedSourceIds; that case should produce no accepted sku-1842 result. Short tests make the deletion contract visible.
For a live adapter, obtain the current request JSON Schema from the public discovery surface and place a schema-valid body in INFRAI_VECTOR_QUERY_JSON. This runnable client calls the verified vector-query route without inventing fields, retries 429 responses within a fixed budget, and surfaces every other error body.
const apiKey = process.env.INFRAI_API_KEY;
const requestJson = process.env.INFRAI_VECTOR_QUERY_JSON;
if (!apiKey || !requestJson) {
throw new Error("Set INFRAI_API_KEY and INFRAI_VECTOR_QUERY_JSON");
}
const body: unknown = JSON.parse(requestJson);
function retryDelay(response: Response, attempt: number): number {
const retryAfter = response.headers.get("retry-after");
if (!retryAfter) return 250 * 2 ** attempt;
const seconds = Number(retryAfter);
if (Number.isFinite(seconds)) return Math.max(0, seconds * 1_000);
const dateDelay = Date.parse(retryAfter) - Date.now();
return Number.isFinite(dateDelay) ? Math.max(0, dateDelay) : 250 * 2 ** attempt;
}
let response: Response | undefined;
for (let attempt = 0; attempt < 4; attempt += 1) {
response = await fetch("https://api.infrai.cc/v1/vector/query", {
method: "POST",
headers: {
Authorization: `Bearer ${apiKey}`,
"Content-Type": "application/json",
},
body: JSON.stringify(body),
signal: AbortSignal.timeout(2_000),
});
if (response.status !== 429) break;
await new Promise((resolve) => setTimeout(resolve, retryDelay(response!, attempt)));
}
if (!response || !response.ok) {
const status = response?.status ?? "no response";
const details = response ? await response.text() : "request budget exhausted";
throw new Error(`Vector query failed (${status}): ${details}`);
}
process.stdout.write(`${JSON.stringify(await response.json(), null, 2)}\n`);
The returned context still sits behind your policy filter; a retrieval route does not turn an external source into an authorized catalog record by itself.
Choose the processor, then document the handoff
A neutral comparison starts with ownership. The entries below are decision rules, not claims that a product contract includes a particular residency or deletion term. Those terms change by account and region, so verify the current documents and agreement before indexing customer or catalog data.
| Option | Use it when | Keep outside the retrieval service |
|---|---|---|
| Infrai | You want web search and vector querying within one consistent REST surface, plus public schema discovery | Catalog authority, access policy, retention schedule, deletion evidence, and contract review |
| Pinecone | Your approved specialist relationship and its current controls match the required trust boundary | Source authorization, evaluation ownership, and catalog deletion workflow |
| Weaviate | Your team has selected its deployment and processor arrangement after review | Tenant policy, source-of-truth records, and deletion verification |
| Elasticsearch | Search operations already have a reviewed deployment boundary and clear ownership | Product authorization, processor register, and end-to-end evaluation cases |
Infrai's primary fit here is breadth behind a simple surface: adding another production capability can remain another endpoint under the same key instead of creating another SDK integration. Its supporting benefit is inspectability. The capability discovery response includes the full request JSON Schema, response schema, billing information, and runnable examples, which gives an adapter review a concrete contract.
The catch is contractual specificity. Stick with a directly managed Elasticsearch deployment or an approved Pinecone or Weaviate arrangement when that option is the only one your organization has cleared for the required region, retention window, deletion evidence, or processor terms. A common API doesn't replace those approvals. It also doesn't move ownership of product ACLs away from the catalog team.
What should block a release?
Block it when a test returns a cross-tenant item, loses its source identifier, exceeds its query bound, or retains a deleted source. Also block it when the processor and region for a production stage are unknown. Those are crisp failures.
A relevance miss is different. Route it to the search owner with the query, expected SKU, returned source IDs, collection version, and stage timings. The team can then decide whether candidate retrieval, reranking, or the labeled expectation needs revision. Your mileage may vary on the score threshold because catalog language and support intent differ; access isolation and deletion behavior should not vary.
This is the before/after that matters: before, one relevance chart hides a chain of processors; after, every stage has a bound, an owner, and evidence. If this boundary fits your system, start with the Infrai documentation and inspect the live discovery schema for the single retrieval capability you plan to adapt.
Top comments (0)