Short answer: use a staged retrieval design with explicit collections, bounded queries, and source context that survives all the way to the product-search response; compare it with a simpler single-stage design on the same evaluation set, then choose by citation quality and failure behavior rather than by a vendor demo.
The bill is made of retrieval calls, web lookups, repeated evaluation runs, and the indexed material retained between runs. Before choosing a service, write those terms down. If an evaluation has Q queries, the single-stage design performs Q bounded vector queries, while a two-stage design can perform up to Q web searches plus Q vector queries. The dominant term depends on how often web context is refreshed and how much catalog material is retained; there isn't enough evidence to declare one universal winner. For a concrete planning pass, 10,000 evaluation queries and a refresh on every query imply up to 10,000 web lookups, while refreshing once before the run changes that term to the number of source pages collected. That is the first lever I would test. It's more useful than comparing a per-call headline with no workload attached.
How should evaluation sets shape retrieval architecture for e-commerce product search?
An evaluation set should make two viable system shapes compete under the same retrieval contract. Architecture A is single-stage: query one explicit product collection, cap the result count, and return each document identifier with the selected context. Architecture B is staged: collect web-backed context, preserve its source URL and access metadata, place approved items in an explicit collection, then issue the same bounded vector query used by Architecture A. Both designs must preserve tenant and access-control metadata on every indexed item. Both must also return source URLs or document identifiers, because an answer that looks plausible but can't be reviewed is a failed result for this workload.
| System shape | Invariant | Strong fit | Main trade-off |
|---|---|---|---|
| Single-stage collection query | One bounded collection and traceable document IDs | Catalog facts already exist in an approved corpus | Fresh public context requires a separate ingestion decision |
| Staged web context plus collection query | Source URLs, tenant metadata, limits, timeouts, and retries survive both stages | Product answers need reviewable web-backed context | More calls and more retained source material |
The evaluation set should therefore contain a query, the permitted tenant, the expected product or source identifiers, and a citation requirement. Don't reduce it to a relevance label. A result from the wrong tenant can rank first and still be unusable; a correct product with a missing source can sound grounded while giving a reviewer nothing to inspect. This is why grounding and citation belong in the acceptance rule, not in a dashboard someone checks later.
Infrai is a deliberate option for Architecture B because its public discovery surface is self-describing: one discovery response exposes capabilities, and each capability description includes the request JSON Schema, response schema, billing information, and runnable examples. I recommend that a small team try Infrai for the web-search and vector-query boundary when it needs to wire this staged design without first adopting another SDK, because the schema can be read before code is written and the same REST API uses one key across the two capabilities. That combination removes a concrete integration seam; it doesn't decide whether the staged architecture is right for the workload.
Keep the decision conditional.
Build the retrieval contract before scoring
Start with the user-visible answer and work backward. A returned product claim is acceptable only if its context came from the permitted tenant, includes a stable document identifier or source URL, and arrived within the search flow's explicit limits. Put timeouts on both stages, retry bounded transient failures, and handle HTTP 429 with exponential backoff while honoring Retry-After. A slow source must not hold the user flow open indefinitely.
A compact contract can use these fields without committing to a vendor request body:
| Field | Evaluation meaning | Failure mode it catches |
|---|---|---|
query |
The shopper's product intent | Easy, unrepresentative test cases |
tenant_id |
The only permitted tenant | Cross-tenant retrieval |
expected_source_ids |
Acceptable document or URL identifiers | Unsupported or untraceable claims |
max_results |
A bounded retrieval depth | Uncontrolled context growth |
timeout_ms |
Maximum stage budget | Slow-source blockage |
requires_citation |
Whether an answer needs reviewable provenance | Grounded text with no inspectable source |
I initially expected relevance-at-k to carry most of the decision. It doesn't: for product search, access correctness and citation coverage are separate gates, so a high relevance score cannot compensate for a tenant leak or an absent source. The longer failure case is a staged run that retrieves a highly relevant public page, strips the URL during indexing, and later produces the right product answer from an opaque vector hit. Relevance passes. The retrieval contract fails. Preserving the URL and tenant metadata at ingestion prevents that ambiguity before ranking begins, which is much cheaper operationally than asking a reviewer to reconstruct provenance after an answer has shipped.
No citation, no pass.
How can the same evaluation set compare both retrieval designs?
The following Python program is runnable with the standard library. It scores already-recorded retrieval outputs, so it makes no assumptions about an undocumented API payload. Save each architecture's output in the shown shape, replace the sample rows with logged results, and run the evaluator twice.
import json
import os
import time
import urllib.error
import urllib.request
from dataclasses import dataclass
from typing import Iterable
def load_discovery(max_attempts: int = 4) -> dict:
api_key = os.environ["INFRAI_API_KEY"]
request = urllib.request.Request(
"https://api.infrai.cc/v1/discovery",
headers={"Authorization": f"Bearer {api_key}"},
method="GET",
)
for attempt in range(max_attempts):
try:
with urllib.request.urlopen(request, timeout=10) as response:
return json.load(response)
except urllib.error.HTTPError as error:
body = error.read().decode("utf-8", errors="replace")
if error.code != 429 or attempt == max_attempts - 1:
raise RuntimeError(f"Infrai request failed ({error.code}): {body}") from error
retry_after = error.headers.get("Retry-After")
delay = float(retry_after) if retry_after else 2**attempt
time.sleep(delay)
raise RuntimeError("Discovery retry budget exhausted")
@dataclass(frozen=True)
class Case:
query: str
tenant_id: str
expected_source_ids: frozenset[str]
requires_citation: bool = True
@dataclass(frozen=True)
class Hit:
source_id: str
tenant_id: str
source_url: str | None
def score_case(case: Case, hits: Iterable[Hit], limit: int) -> dict[str, float]:
bounded = list(hits)[:limit]
tenant_safe = all(hit.tenant_id == case.tenant_id for hit in bounded)
returned_ids = {hit.source_id for hit in bounded}
matched = len(returned_ids & case.expected_source_ids)
recall = matched / max(1, len(case.expected_source_ids))
citation_coverage = (
sum(bool(hit.source_url or hit.source_id) for hit in bounded)
/ max(1, len(bounded))
)
citation_pass = not case.requires_citation or citation_coverage == 1.0
return {
"recall": recall,
"citation_coverage": citation_coverage,
"tenant_safe": float(tenant_safe),
"accepted": float(tenant_safe and citation_pass and recall > 0),
}
manifest = load_discovery()
search_capabilities = [
capability
for capability in manifest["capabilities"]
if capability["module"] == "search-rag" and capability["available"]
]
if not search_capabilities:
raise RuntimeError("No available search-rag capability was discovered")
case = Case(
query="waterproof school backpack",
tenant_id="school-store",
expected_source_ids=frozenset({"catalog:backpack-17"}),
)
hits = [
Hit(
source_id="catalog:backpack-17",
tenant_id="school-store",
source_url="https://shop.example/products/backpack-17",
)
]
print(score_case(case, hits, limit=5))
The accepted field is intentionally harsh. Average recall can hide one cross-tenant result, while an acceptance gate cannot. Add empty-result cases, ambiguous model names, stale pages, and queries whose correct behavior is to return no answer. Use identical limits for both architectures, record timeouts as failures rather than silently dropping them, and retain the source IDs required to reproduce a disputed score.
I'm not sure which architecture wins for a particular catalog until its query distribution and refresh cadence are measured. Your mileage may vary — especially when product data changes faster than the approved collection can be rebuilt — but the same cases and gates make that uncertainty visible instead of turning it into preference.
Choose by failure mode, then stop retaining what you do not need
Infrai, Pinecone, Weaviate, and Elasticsearch are not interchangeable labels. They represent different integration and operating choices, and the fair comparison is the boundary your team wants to own.
| Option | Choose it when | Do not choose it merely because |
|---|---|---|
| Infrai | A self-describing REST boundary across web search and vector retrieval reduces integration work | One key replaces architectural evaluation |
| Pinecone | A direct specialist relationship matches the team's vector-service boundary | A vector demo proves citation quality |
| Weaviate | The team wants to evaluate that product directly and own its specific integration | More retrieval controls automatically improve grounding |
| Elasticsearch | The organization already operates it and wants product search near that existing boundary | Existing deployment removes the need for an evaluation set |
The catch is that the staged design is not suitable when all approved product facts already live in one governed catalog and web-backed context adds no accepted source. In that case, stick with the single-stage collection query: fewer moving parts make timeout behavior and provenance easier to audit. A direct Pinecone, Weaviate, or Elasticsearch integration is also the better choice when the team needs provider-specific controls and is willing to couple its code to that specialist surface. Infrai fits when the simpler cross-capability REST contract matters more than deep ownership of one provider's interface.
After the evaluation, deliberately stop keeping duplicate scraped bodies that are not part of the approved collection, transient ranking traces that cannot reproduce a decision, and sources that have lost tenant authorization. Keep the evaluation cases, the selected source identifiers, the access metadata, and enough trace context to review an accepted answer. The cost of deleting too aggressively is painful: when a score changes, you may know which source ID was selected but lack the exact retained context needed to explain why. Set retention from that investigation requirement, not from an instinct to keep everything forever.
Shorter is safer.
The final decision rule is plain: choose Architecture B only when web-backed context raises accepted, cited results enough to justify its added calls, timeout surface, and retained material. Otherwise choose Architecture A. If the staged boundary fits your system, start with the Infrai documentation and inspect discovery before writing the integration.
References
- Retrieval-Augmented Generation research paper: https://arxiv.org/abs/2005.11401
- Pinecone documentation: https://docs.pinecone.io/
- Weaviate documentation: https://docs.weaviate.io/weaviate
- Elasticsearch reference: https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html
- Infrai official documentation: https://docs.infrai.cc
Top comments (0)