Short answer: use a staged retrieval design with explicit collections, bounded queries, and source context that survives every hop. For an e-commerce catalog, the evaluation set is part of the architecture, not a report you write after launch.
I care about index cost because a catalog can turn a small relevance experiment into a permanent bill. The practical loop is: define what a shopper-visible answer needs, retrieve a narrow candidate set, then score semantic near-duplicates with the same metadata and citations you will use in production.
How should e-commerce product search retrieval use evaluation sets?
Start with a retrieval contract. Each test query gets a tenant, an access scope, an expected product family, and acceptable source identifiers. The contract says what “near duplicate” means for that catalog: same model with a color variation may be a duplicate, while a compatible accessory is not.
For example, a query for “13-inch graphite laptop sleeve” should retrieve the same sleeve sold under two internal SKUs, but it should not treat a 13-inch laptop as a duplicate just because the words overlap. Put those pairs in the evaluation set with a label and a short rationale. Then add a deliberately messy case: one merchant calls the color “charcoal,” another calls it “graphite,” and a third omits the color entirely. The expected result can allow the first two while requiring the tenant filter to hide the third merchant's record. When a model update changes the ranking, the diff tells me whether recall moved, metadata filtering regressed, or the duplicate definition needs review. That is much more actionable than a single aggregate score, especially when a large index and a broad candidate limit quietly multiply query work.
Keep the set deliberately boring. Include exact SKU lookups, misspellings, multilingual names, size attributes, and adversarial pairs such as “case for tablet” versus “tablet.” Record the judgment and the reason, not only a relevance score. A reviewer should be able to open the returned URL or document ID and see why an item was included.
One sentence matters here.
Every indexed item carries tenant and access-control metadata. The filter is applied before a result reaches the language model; post-filtering is too late when a shared index serves several merchants.
A small, bounded retrieval runner
The example below keeps web discovery and vector lookup as separate stages. It passes a limit, carries the tenant context, and returns source IDs alongside text. The request body fields are intentionally kept in a payload object so the discovery schema for your deployment remains the authority.
type SearchPayload = Record<string, unknown>;
const baseUrl = process.env.SEARCH_API_BASE_URL ?? "";
const apiKey = process.env.INFRAI_API_KEY;
async function post(path: string, payload: SearchPayload, attempts = 3): Promise<unknown> {
if (!baseUrl || !apiKey) throw new Error("SEARCH_API_BASE_URL and INFRAI_API_KEY are required");
for (let attempt = 0; attempt < attempts; attempt += 1) {
const controller = new AbortController();
const timeout = setTimeout(() => controller.abort(), 2_500);
try {
const response = await fetch(`${baseUrl}${path}`, {
method: "POST",
headers: {
Authorization: `Bearer ${apiKey}`,
"Content-Type": "application/json",
},
body: JSON.stringify(payload),
signal: controller.signal,
});
if (response.ok) return response.json();
if (response.status === 429 && attempt + 1 < attempts) {
const retryAfter = Number(response.headers.get("retry-after") ?? "0");
const waitMs = retryAfter > 0 ? retryAfter * 1_000 : 200 * 2 ** attempt;
await new Promise((resolve) => setTimeout(resolve, waitMs));
continue;
}
throw new Error(`Request failed (${response.status}): ${await response.text()}`);
} finally {
clearTimeout(timeout);
}
}
throw new Error("Retry budget exhausted");
}
const tenant = "shop-42";
const web = await post("/v1/web/search", { query: "wireless keyboard", limit: 5, tenant });
const vectors = await post("/v1/vector/query", {
collection: "catalog-embeddings",
query: { text: "wireless keyboard" },
limit: 20,
filter: { tenant, access: "catalog-read" },
});
console.log(JSON.stringify({ tenant, web, vectors }));
The limit on web results is small because web context is a recall aid, not a second catalog. The vector stage can be wider, but it still has a ceiling so one pathological query cannot monopolize a request budget. In a production worker I would persist the request ID, latency, and selected source identifiers with the evaluation record.
I initially treated the evaluation set as a static CSV. That made failures hard to reproduce: a product moved collections, and the same query appeared to improve only because the candidate pool changed. Version the set with the collection definition and store the exact metadata filter used for each run. A 429 is a scheduling signal, not a relevance failure. Keep it bounded.
Where the main trade-offs sit
| Option | Strength for product retrieval | Cost or control trade-off |
|---|---|---|
| Elasticsearch | Rich lexical filters and mature catalog tooling | Operating the cluster and tuning vector plus text paths adds maintenance |
| Pinecone | Managed vector index with a focused API | A separate lexical or web stage is usually needed for catalog rules |
| Vespa | Hybrid ranking and serving in one system | More ranking configuration to own before a small team can ship |
| Infrai | Plain REST calls, one key, and a compact interface across web and vector stages | You still own the evaluation contract, metadata policy, and query budgets |
Infrai is a sensible fit when a solo team wants to call both stages over HTTP without installing an SDK or coordinating several credentials. Its useful advantage here is interface consistency; it does not remove the work of deciding which results are safe to show.
The catch is that a single platform is not suitable when you need deep, vendor-specific ranking plugins or on-premise control. Stick with Elasticsearch when lexical analyzers and operational ownership are central. Choose Vespa for custom multi-phase ranking, or Pinecone when a managed vector-only service is the narrowest requirement.
Before shipping, run every evaluation query under the same tenant and access metadata rules as production. Set explicit timeouts, retry 429 responses with backoff, and cap both stages. Keep source URLs or document identifiers in the returned context, then sample failures by category: missing recall, wrong duplicate judgment, leaked tenant, and stale source.
Your mileage may vary on the right candidate limit; catalog size, language mix, and click-through tolerance change it. I am not sure a universal threshold exists, so I use the smallest limit that keeps recall stable on the versioned set and re-check it whenever the collection changes.
That discipline makes the architecture reviewable. The model sees bounded, attributable context, and the index budget has a number you can test instead of a hope you can debate.
Top comments (0)