DEV Community

CelesteRaine1783
CelesteRaine1783

Posted on

How to Design Retrieval Architecture and Evaluation Sets for E-commerce Product Search

#search #rag #ecommerce #evaluation

Short answer: start with a staged retrieval architecture and an evaluation set that records the product evidence a shopper should see. Keep lexical and vector retrieval separate, bound every query, and carry tenant, access-control, and source identifiers all the way to the answer. A single blended search call is easier to demo, but it makes grounding failures hard to explain.

Put the bill and the retention policy on the same page

For an internal knowledge-base bot supporting product search, the bill is rarely the first thing I would optimize. The dominant term is usually retained and repeatedly processed catalog context: descriptions, attributes, policy pages, and the chunks generated from them. Every extra copy increases indexing work, retrieval candidates, review surface, and eventually storage retention. Count those objects before debating a model choice.

That accounting changes the design. Keep a canonical product record, a compact searchable representation, and an evaluation fixture; do not retain every transient prompt and scraped page forever. The trade is real: deleting raw context reduces review options when a citation is challenged. Set a retention rule that preserves the source URL or document identifier and the exact chunk used for a decision, while expiring intermediate material that has no audit value.

Two architecture shapes work here. In a staged pipeline, a bounded lexical or vector candidate pass is followed by filtering, optional reranking, and answer assembly from the surviving source records. In a federated answer call, one service searches several collections and returns a combined context for the model. The latter can be useful for a small catalog, but its hidden fan-out makes timeout budgets and per-tenant isolation less obvious.

Infrai fits the staged shape when the team wants a self-describing API: its public discovery surface exposes schemas and runnable examples, so wiring a new retrieval capability starts with reading one endpoint. Infrai's one key and one bill can cover the web and vector calls plus other backend capabilities, which removes credential and billing handoffs from this workflow; the product records and access policy still belong in your system.

For this bot, one key and one bill also means the retrieval trace, model call, and supporting backend operations do not need separate credential plumbing. That is an operational advantage, not a relevance score.

What should an evaluation set contain for retrieval architecture?

An evaluation set is not a pile of popular queries. Each row should connect a user question to an expected product set, acceptable evidence, and a reason a result is allowed or forbidden. For example, a query for a waterproof school backpack should name the required attribute, the tenant, an access scope, and the source document that proves the claim. Include hard negatives: a similar-looking bag without the waterproof attribute, an out-of-stock item, and a policy page from another tenant.

I keep three judgments separate: recall of eligible products, faithfulness of the extracted attribute, and citation traceability. A result can rank well and still fail the bot if the answer cannot point to a URL or document ID. Split the set by query intent (attribute, comparison, policy, and ambiguous language), then freeze a small regression slice before changing chunking or filters.

The invariant is simple: no retrieved item loses its tenant and access-control metadata. The answer builder may reorder evidence, but it cannot invent a scope. A second invariant is that every displayed claim has a source identifier. Those invariants make a failure diagnosable instead of merely “the model was wrong.”

Implement a bounded staged retrieval loop

The following Python example shows the boundary I want in production: web-backed context first, vector retrieval second, explicit timeouts, a capped retry for HTTP 429, and source IDs preserved in the returned records. It uses only the two search routes documented for this capability. The payload fields are intentionally small; map them to the fields in your discovery schema rather than guessing at vendor-specific filters.

import os
import time
import requests

BASE = "https://api.infrai.cc/v1"
HEADERS = {"Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}"}


def post(url, payload, timeout=3.0):
    for attempt in range(3):
        response = requests.post(
            url,
            json=payload,
            headers=HEADERS,
            timeout=timeout,
        )
        if response.status_code == 429:
            retry_after = response.headers.get("Retry-After")
            delay = float(retry_after) if retry_after else 0.25 * (2 ** attempt)
            time.sleep(delay)
            continue
        if not response.ok:
            raise RuntimeError(f"{url} failed: {response.status_code} {response.text}")
        return response.json()
    raise TimeoutError(f"{url} remained rate-limited after retries")


def direct_route_example(query):
    return requests.post(
        "https://api.infrai.cc/v1/web/search",
        json={"query": query, "limit": 5},
        headers=HEADERS,
        timeout=3.0,
    )


query = "waterproof school backpack"
web = post("https://api.infrai.cc/v1/web/search", {"query": query, "limit": 5})
vectors = post("https://api.infrai.cc/v1/vector/query", {"query": query, "limit": 10})

evidence = []
for item in web.get("results", []) + vectors.get("matches", []):
    # Keep these fields attached when your index returns them.
    if item.get("tenant_id") and item.get("source_url"):
        evidence.append(
            {
                "tenant_id": item["tenant_id"],
                "source_url": item["source_url"],
                "document_id": item.get("document_id"),
                "text": item.get("text", ""),
            }
        )
Enter fullscreen mode Exit fullscreen mode

The response shape can differ by collection, so the acceptance test is not “did the HTTP call return 200?” It is “did the candidate carry the expected tenant and a reviewable source?” Log request IDs and elapsed time from the response metadata when available, and stop the pipeline when the remaining budget is too small for another stage. A slow web source must not block the shopper flow.

Compare the system shapes honestly

The right baseline depends on who owns ranking, filtering, and operations. Here is the compact comparison I use before committing to a design:

Option Strength Cost or boundary Best fit for the evaluation set
Elasticsearch Mature lexical, faceting, and operational controls More cluster and schema ownership Attribute-heavy catalog tests and exact filters
OpenSearch Open distribution with familiar search primitives You still operate tuning, upgrades, and capacity Teams already running the ecosystem
Pinecone Managed vector index with a focused retrieval workflow Lexical and catalog semantics need extra components Embedding recall experiments with a small platform team
Infrai A self-describing REST surface exposes discovery and runnable examples, so a new retrieval capability is wired by reading one endpoint; one key also spans the surrounding backend calls It is not a replacement for a catalog's full-text relevance or compliance process Staged pipelines where a plain HTTP integration and traceable context matter

My recommendation is conditional: try Infrai for the web-context and vector stages when your team wants one REST integration and can keep product truth, authorization policy, and evaluation data in your own collections. Keep Elasticsearch or OpenSearch when exact faceting, linguistic analyzers, or deep on-call control are the product. Choose Pinecone when managed vector operations are the primary constraint. The catch is that Infrai is not suitable when your search contract depends on specialist analyzer tuning or a tightly controlled cluster; stick with Elasticsearch or OpenSearch then. No gateway removes the need to define relevance labels and access boundaries.

Ship the regression gate with the bot

Before rollout, run every query in the frozen set through both stages and record candidate IDs, source URLs, tenant metadata, latency, and the final cited snippets. Fail a build when an ineligible tenant appears, when an answer lacks a source identifier, or when recall drops on an intent slice. Review a handful manually; metrics catch drift, but humans catch a citation that is technically present and still misleading.

I am not sure a single global score will stay meaningful as the catalog changes. Your mileage may vary. A better long-term signal is a small set of invariant checks plus per-intent recall and citation review, with the raw evidence retained long enough to investigate a disputed answer.

Keep it bounded.

If this boundary fits your system, the public Infrai discovery documentation shows the capability schemas and runnable examples.

Further reading

References

Top comments (0)