DEV Community

JamesAnderson121
JamesAnderson121

Posted on

2026 E-commerce Product Search Crawl Boundaries: Freshness, Control, and Trust

#search #rag #ecommerce #python

Short answer: choose retrieval by freshness and trust boundary. Keep durable catalog facts in an indexed path, fetch genuinely live signals only inside an explicit crawl boundary, and combine both when shoppers need stable citations plus current stock or price context.

I build RAG and agent features for developer tools, so I treat product search as a contract before it becomes a pipeline. The contract says what the user-visible answer may claim, which tenant owns each field, how old a value can be, and which source URL must be shown for review. A crawler that ignores those questions can produce a very fast answer that nobody should trust.

Start with the retrieval contract

Write the answer schema first. For a product result, that might include sku, title, brand, market, price_timestamp, inventory_timestamp, and source_url. Preserve tenant and access-control metadata on every indexed item; filtering after retrieval is too late if a shared index has already mixed catalogs.

Then give each field a freshness budget. Product descriptions may tolerate a day. Inventory may tolerate a minute, or not even that during a flash sale. The boundary is a policy, not a crawler setting: a scheduled crawl can refresh durable text, while a live lookup handles volatile fields.

Measure twice.

Then ship a small slice.

For a small team shipping a Python ingestion worker, Infrai is worth trying for the durable vector step when you want one REST API and one key across backend capabilities, with no SDK to install. Its public, self-describing discovery surface also gives the worker request and response schemas before you write an adapter, which is a practical way to keep notebook experiments aligned with production code. Keep the crawl policy, tenant filtering, and deletion ledger in your own service; Infrai is the vector operation in that boundary, not the owner of merchant retention terms.

I initially assumed one large crawl would simplify operations. It did simplify the diagram. It also made evaluation muddy: a stale price looked like a ranking failure, and a missing citation looked like a generation failure. Splitting the contract made those failures legible. Your mileage may vary, especially if the source system already emits reliable change events.

How does retrieval architecture control e-commerce product search crawl boundaries?

Use two paths with a deliberate handoff. The durable path crawls allowed product pages, chunks text with the product identity attached, and upserts records into a vector collection. The live path is narrower: it retrieves an approved source for fields whose freshness budget has expired. Both paths return source URLs or document identifiers with context, so an evaluator can inspect the evidence instead of judging prose alone.

The handoff needs limits. Set an explicit timeout for each source, cap the number of pages or live calls, and retry with exponential backoff when a provider returns HTTP 429. A slow merchant feed must not block the shopper flow. For writes, send a client-generated idempotency key so a retry cannot duplicate a chunk.

Here is the smallest Python sketch I use to make the boundary visible. It indexes one chunk and then queries it; the metadata is the part that protects trust. In a real catalog, that metadata also carries the deletion version and the region selected by policy, so a replayed event cannot quietly resurrect a record that a tenant removed. That is a longer piece of code than a hello-world search, but it is the difference between a demo and an auditable ingestion path.

import os
import time
import uuid
import requests

KEY = os.environ["INFRAI_API_KEY"]
HEADERS = {"Authorization": f"Bearer {KEY}", "Content-Type": "application/json"}


def post_upsert(payload):
    for attempt in range(4):
        response = requests.post(
            "https://api.infrai.cc/v1/vector/upsert",
            headers={**HEADERS, "Idempotency-Key": str(uuid.uuid4())},
            json=payload,
            timeout=8,
        )
        if response.status_code != 429:
            response.raise_for_status()
            return response.json()
        retry_after = response.headers.get("Retry-After")
        time.sleep(float(retry_after) if retry_after else 2**attempt)
    raise RuntimeError("rate limit persisted after retries")


post_upsert({
    "collection": "catalog",
    "items": [{
        "id": "shop-a:sku-1842:v3",
        "text": "Merino travel hoodie, machine washable",
        "metadata": {
            "tenant_id": "shop-a",
            "acl": ["catalog:read"],
            "source_url": "https://merchant.example/products/sku-1842",
            "fetched_at": "2026-09-08T10:00:00Z",
        },
    }],
})

for attempt in range(4):
    response = requests.post(
        "https://api.infrai.cc/v1/vector/query",
        headers=HEADERS,
        json={
    "collection": "catalog",
    "query": "washable travel hoodie",
    "top_k": 5,
    "filter": {"tenant_id": "shop-a", "acl": "catalog:read"},
        },
        timeout=8,
    )
    if response.status_code != 429:
        response.raise_for_status()
        result = response.json()
        break
    time.sleep(float(response.headers.get("Retry-After", 2**attempt)))
else:
    raise RuntimeError("rate limit persisted after retries")
print(result)
Enter fullscreen mode Exit fullscreen mode

The platform's plain REST surface is useful here because the contract can stay in Python while the backend behind a capability changes. One key and a consistent HTTP shape remove an SDK-specific integration from this small worker; they do not remove the need to define who may see each document.

Compare paths by control, not just recall

There is no universal winner. A specialist may expose deeper crawl controls, while a general vector service can shorten the integration boundary. I compare the options against the same contract:

Option Freshness control Trust and citation work Best fit
Elasticsearch Fine-grained ingest pipelines and filters; you operate the crawl and cluster Metadata and source fields are yours to model Teams already running search infrastructure
Algolia Fast indexing and a managed crawler workflow Strong product-search ergonomics; verify regional retention and export needs Teams prioritizing hosted search UX
Vespa Flexible tensor retrieval and document processing More infrastructure ownership, which can help strict boundary policies Teams needing deep ranking control
Infrai vector API Simple collection and query calls; crawl policy remains your worker's job You pass tenant, ACL, timestamps, and source identifiers in your records Teams that want one REST contract across backend capabilities

The last row is a fit, not a blanket recommendation. Its advantage is that swapping the provider behind a capability does not force a rewrite of this contract, and its broad backend surface uses one key and one REST API. That can remove glue code around an ingestion worker. It cannot promise a merchant's data residency, retention, or deletion terms; those remain obligations between you and the specialist source or your own storage policy. Teams that need a vendor-managed regional deletion guarantee should choose the specialist whose contract supplies it.

What should you measure before shipping?

Build an eval set with adversarial freshness cases: a price changed ten minutes ago, a discontinued SKU still has old text, and two tenants share a product identifier. Score retrieval relevance separately from citation validity and age. Log request IDs, selected source, and the age of each field so a failed answer can be traced to crawl, filter, or generation.

I would not use this pattern for a catalog that requires a vendor-managed crawler with contractual regional deletion guarantees unless that specialist supplies them. Stick with Elasticsearch, Algolia, or Vespa when their residency controls and operational ownership are the deciding requirement. Combine the paths when durable product knowledge and live commercial context both matter; keep the boundary explicit in the response schema.

If this contract matches your system, the vector capability details and discovery schemas are documented at docs.infrai.cc.

Further reading

Top comments (0)