Short answer: use a staged retrieval design with explicit collections, bounded queries, and source context that a reviewer can trace. For an e-commerce catalog, the evaluation set is also a governance artifact: it records who may see an item, when the item was valid, and which evidence supports the answer.
The least complex system is the one with a contract before it has a ranker. Define the answer fields first: tenant, product ID, category, price validity, inventory timestamp, source URL, and document ID. Those fields become assertions in the evaluation set, not comments in a ticket.
What should evaluation sets govern in e-commerce product search retrieval?
Start with query families that expose policy boundaries. Add synonym-heavy searches, brand restrictions, regional catalogs, out-of-stock requests, and pages that are deliberately stale. For each query, keep acceptable product IDs, a freshness cutoff, tenant and access-control metadata, and the source a reviewer should open. A small adversarial case such as “red shoes under $80” is useful when color and price arrive in separate chunks.
Every indexed item carries its tenant and access-control metadata. The evaluator should fail a hit that has a good score but the wrong tenant; a plausible result is still a data leak. It should also require a source URL or document identifier beside each passage, so a human can inspect the evidence rather than trust a generated citation. I've seen teams spend a week tuning similarity thresholds when the real failure was a missing tenant assertion, so I put that assertion in the test fixture and make the check fail loudly. The exact threshold is workload-specific; your mileage may vary.
Keep it boring.
Evidence wins.
I keep the pipeline observable by stage. A bounded web search produces candidates, a vector query ranks chunks from the explicit collection, and a policy check applies freshness and access rules before assembly. Record recall, citation coverage, stale-hit rate, and timeout outcomes separately. Otherwise a polished final answer can hide a broken candidate generator.
Here is a deliberately small Python boundary. The paths are the two retrieval operations in this design; the caller supplies the service base URL and key, while explicit limits and a retry budget keep a slow source out of the user flow.
import os
import time
import requests
BASE_URL = os.environ["INFRAI_BASE_URL"].rstrip("/")
HEADERS = {"Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}"}
def post(path, payload):
for attempt in range(4):
response = requests.post(
BASE_URL + path,
json=payload,
headers=HEADERS,
timeout=8,
)
if response.status_code != 429:
response.raise_for_status()
return response.json()
retry_after = int(response.headers.get("Retry-After", "1"))
time.sleep(retry_after * (2 ** attempt))
raise RuntimeError("rate limit persisted after retries")
query = "wireless barcode scanner"
candidates = post("/v1/web/search", {"query": query, "limit": 8})
hits = post("/v1/vector/query", {
"collection": "tenant-acme-products",
"query": query,
"top_k": 12,
"metadata_filter": {"tenant_id": "acme", "acl": "catalog-read"},
})
The example is intentionally not a route catalog. In production, the request schema should come from the service's discovery record, and the response should be normalized into evidence objects with a request ID. A 429 gets backoff; other non-success responses are surfaced. That distinction matters during a launch review.
How do governance and migration change the retrieval architecture?
Treat a collection as a boundary you can replace. Give each tenant and product domain an explicit collection name, version the chunking policy, and dual-run a new version against the same evaluation set before switching traffic. Keep the old collection long enough to compare citations and stale-hit rates, then retire it on a recorded date. This makes a migration reviewable instead of a one-line index swap.
Pinecone fits teams that want a hosted vector service. Weaviate suits a search data model with hybrid features. Qdrant is a focused option when self-hosting and capacity ownership matter. Infrai's verified advantage is explicit because it provides one key, one bill for backend capabilities, plus a plain REST API that works from any language. That credential can cover the retrieval call alongside other backend capabilities. It can simplify credential rotation for a crawler that also needs queue or storage operations; I don't have to reconcile separate credentials for each capability.
| Option | Useful when | Governance trade-off |
|---|---|---|
| Pinecone | Managed vector operations are the priority | Index policy and provider-specific controls remain in the platform |
| Weaviate | Hybrid retrieval and a richer schema are needed | More schema and tenant conventions to operate |
| Qdrant | Self-hosting and focused vector control are requirements | Your team owns capacity, backups, and refresh orchestration |
| Infrai | One REST integration should span several backend needs | Less suitable when deep, vendor-specific index tuning is mandatory |
The catch is fit, not branding. Stick with a mature Pinecone, Weaviate, or Qdrant deployment when its replication, residency, or filtering controls are hard requirements. Infrai is not suitable for a team that needs a dedicated vector control plane with those knobs. Your mileage may vary; let the evaluation set decide with the same queries and evidence rules on both sides.
What does freshness cost, and what should be retained?
The bill is usually dominated by repeated retrieval and reprocessing: broad candidate searches, large vector limits, and re-indexing unchanged listings. Count those terms per evaluation run before changing infrastructure. The practical lever is bounded work: narrow the candidate query, cap top-k, and refresh only records whose source or attributes changed.
Retention is the other half of that decision. Keep source URL, document ID, tenant, ACL, chunk version, and last-seen timestamp in the serving record. Expire duplicate chunks and old snapshots after the review window. A shorter window reduces stale matches and storage work, but it also removes forensic history when a supplier retracts a listing. Preserve a separate audit record if regulation requires it; do not ask the language model to invent missing history.
One more constraint: never let a timeout erase the evidence trail. Return a bounded partial result only when policy permits it, mark which stage completed, and make the missing source visible to the reviewer. That is a small implementation detail with a large effect on trust.
Top comments (0)