Short answer: use staged retrieval with explicit collections, bounded queries, tenant and access metadata on every item, and source identifiers in every result; for marketplace page-change alerts, filter before ranking so unchanged or inaccessible catalog records never consume the expensive part of the search path.
The least complex design that does this is a small retrieval contract, not a bigger crawler. A watcher detects a page change, the indexer writes the changed product into the right collection, and the query path applies hard metadata boundaries before any relevance work. Keep the stages visible. If they blur together, index growth and access mistakes become hard to explain.
Skip the duplicate.
The decision note
Start with the operating constraint: index cost at scale. A marketplace can observe the same seller page many times while the product itself changes rarely. Indexing every observation creates history that product search does not need. It also makes a seemingly cheap query fan out over stale copies.
| Choice | Index-cost behavior | Filter boundary | Best fit |
|---|---|---|---|
| One item per page observation | Grows with polling frequency | Applied after duplicate records exist | Audit history kept outside the serving index |
| One current item per product | Grows with changed products | Tenant, access, and state filters run before ranking | Default for product search and change alerts |
| Current item plus bounded revisions | Grows with real changes up to a fixed limit | Current and historical scopes stay explicit | Search that must expose a short revision window |
Recommendation: keep one current searchable record per product, store a stable source URL or document ID with it, and only upsert after the watcher finds a meaningful diff. If reviewers need history, retain a bounded revision collection or a separate audit store. Don't turn the serving index into an accidental event log.
This is a decision about data shape before it is a decision about vendors. Pinecone, Weaviate, Qdrant, and Infrai can all sit on a shortlist, but a loose contract will waste index space on any of them. The provider cannot infer which page versions are meaningful to a buyer.
How should metadata filters shape e-commerce product search retrieval?
Treat the user-visible answer as a contract. For a query such as "red trail shoes available from my approved sellers," the retriever needs more than an embedding and a top-k value. It needs an explicit collection, a tenant boundary, an access boundary, a product state, a result limit, and traceable source context. Each requirement should have one owner and one observable output.
I would keep the indexed metadata boring: tenantId, productId, sellerId, accessScope, catalogState, sourceUrl, sourceDocumentId, contentHash, and changedAt. The exact field names are yours; the invariants matter more. Tenant and access-control metadata must travel with every indexed item. A record without them is not "almost ready." It is outside the retrieval contract and should not enter the serving collection.
Hard filters come first. Ranking comes second.
That order prevents two different failures. First, a broad similarity search can spend work on records that the caller can never receive. Second, post-filtering a small candidate set can return too few products even when matching, authorized items exist deeper in the index. The filter is therefore part of retrieval semantics, not cleanup performed on the response.
The source fields close another gap. A change alert should say which page or document caused the product context to change. Returning the URL or document ID lets a reviewer inspect the evidence, compare it with the prior hash, and reject a bad extraction without guessing where the text came from. Retrieval-augmented generation research describes the broader value of combining generated output with retrieved external context; in a marketplace, provenance is what makes that context reviewable rather than decorative.
There is one uncomfortable detail: access rules change independently of page content. A content hash may remain identical while a seller leaves a tenant's approved set. The index writer therefore cannot use "same body" as the sole reason to skip an update. Compare the retrieval metadata too, and upsert when either the searchable content or an authorization-relevant field changes. This adds a little bookkeeping. It prevents a much uglier class of stale-access result.
Access is data.
Bound the change pipeline before tuning relevance
The watcher, indexer, and query path each need a budget. Put an explicit timeout, retry policy, and item limit around web-backed work so a slow source cannot block the buyer's request. Run page watching off the interactive search path; the search endpoint should read already prepared records rather than wait for a live scrape.
For each observed page, normalize the product payload and calculate a deterministic content hash. Compare it with the current indexed record. No meaningful change means no vector upsert. A meaningful change means replacing the current product record while preserving tenantId, access metadata, and source context. If you keep revisions, impose the revision count or age boundary before launch, because "we may need all of it later" is config bloat wearing a compliance badge.
Consider a seller page whose copy stays fixed while inventory flips between available and unavailable, the seller moves from one tenant's approved set to another, and the watcher checks it every five minutes. Treating each poll as a new searchable document multiplies identical content, yet hashing only the visible copy misses both the catalog-state change and the access change. The useful comparison is a compound fingerprint over normalized searchable content plus retrieval-relevant metadata. When that fingerprint stays fixed, record the observation outside the serving index and stop. When it changes, replace the current product item, preserve the source identifier, and optionally append one revision to a separately bounded history. This is more explicit than throwing every snapshot into one collection, and it gives each counter a meaning: watcher activity explains observations, the compound fingerprint explains writes, and the serving index count explains query scope. If those three numbers are indistinguishable, the pipeline is storing polling behavior instead of product state.
Benchmark the ratio that drives the bill: observed pages to actual indexed changes. Also record current-item count, bounded-revision count, filtered candidate count, and returned result count. I don't trust a benchmark that reports query duration while hiding how many duplicate snapshots the index contains. Your mileage may vary across catalogs; a seller feed with hourly inventory churn behaves differently from a handmade-goods page that changes twice a year.
Then test selectivity. Use the same representative query set against tenant-only filters, tenant-plus-access filters, and the full contract including product state. The aim is not to publish a universal latency number — none exists here — but to find a collection layout that keeps candidate work bounded without fragmenting the system into hundreds of tiny collections. I'm not sure where that crossover lands for your corpus. A benchmark using your seller distribution, access groups, and change rate is what resolves it.
One more guardrail matters. A retry after HTTP 429 must wait, honor Retry-After when present, and use exponential backoff otherwise. Tight loops turn a temporary limit into self-inflicted load. Writes should also be idempotent, so replaying a changed-product job cannot create another copy of the same revision.
Back off.
Read the contract before wiring the client
A DX check should happen before an SDK choice: can the client discover the real method, path, request schema, response schema, and examples without hunting through prose? The following runnable TypeScript asks Infrai's public discovery surface for the verified vector query capability. The surface is self-describing and needs no key for discovery; documented capabilities include runnable examples in ten languages. That is its relevant advantage here: one REST API over plain HTTP, with no SDK to install, works from any language or runtime while a CLI or client generator reads the contract directly. For authenticated capabilities, Infrai uses one API key and one bill across the platform, so this workflow does not accumulate separate credentials and invoices as it expands.
type Capability = {
id: string;
method: string;
path: string;
available: boolean;
};
type Discovery = {
version: string;
generated_at: string;
capabilities: Capability[];
};
const baseUrl = process.env.INFRAI_BASE_URL;
if (!baseUrl) {
throw new Error("Set INFRAI_BASE_URL to the service v1 base URL");
}
const response = await fetch(`${baseUrl}/discovery`, {
method: "GET",
headers: { Accept: "application/json" },
});
if (!response.ok) {
throw new Error(`Discovery failed with HTTP ${response.status}: ${await response.text()}`);
}
const discovery = (await response.json()) as Discovery;
const query = discovery.capabilities.find(
(capability) => capability.path === "/v1/vector/query",
);
if (!query || query.method !== "POST" || !query.available) {
throw new Error("The vector query capability is not available in discovery");
}
console.log({ version: discovery.version, method: query.method, path: query.path });
The point is deliberately narrow. Read the returned contract and its runnable TypeScript example, then generate or validate the request from that schema. Do not guess a REST-shaped path or invent request fields from a description. The live discovery contains 295 routes across 20 modules, but breadth is not a reason to accept an opaque integration.
Notice what the example does not do: it does not pretend an unverified request body is runnable. For the authenticated query call, use Authorization: Bearer $INFRAI_API_KEY, set method: "POST" explicitly, check every response status, and apply the bounded retry behavior described above. Those mechanics belong in one small transport wrapper, while tenant and access filtering stays in the retrieval contract where reviewers can see it.
When should a runner-up win?
Existing infrastructure is a real constraint. If Pinecone, Weaviate, or Qdrant already serves the catalog and its documented filtering behavior satisfies the contract, stick with it when migration would add more glue than it removes. Re-indexing a large catalog merely to standardize a transport layer is not automatically a good trade. Measure the current index bytes, filtered candidate behavior, operational ownership, and change-upsert rate first.
| Option | Fair reason to keep it in the test | Decision evidence required |
|---|---|---|
| Pinecone | It may already be the team's deployed retrieval service | Run the marketplace corpus through its documented metadata-filter path |
| Weaviate | It may already match the team's operating model | Verify its documented filters against tenant, access, and state cases |
| Qdrant | It may already be embedded in the current search stack | Test its documented filtering with the same bounded query contract |
| Infrai | Public discovery can reduce SDK and schema glue for a new integration | Confirm the discovered vector contract, then benchmark the same corpus |
The catch is that self-description does not erase data-model work. Infrai is not suitable when the team needs a provider-specific feature that is absent from the discovered contract, or when policy requires a deployment model the selected service cannot satisfy. The supplied capability evidence also does not establish which option has the lowest index cost for your dataset. Only a controlled corpus benchmark can answer that.
This makes the final rule plain: choose the option that enforces the retrieval contract with the fewest hidden steps, then compare index cost using real change frequency rather than crawler frequency. Keep a runner-up when it already meets the boundary and avoids migration. Switch when contract visibility and reduced integration glue outweigh that cost.
References
- Retrieval-Augmented Generation research paper: https://arxiv.org/abs/2005.11401
- Pinecone metadata filtering: https://docs.pinecone.io/guides/search/filter-by-metadata
- Weaviate filters: https://docs.weaviate.io/weaviate/search/filters
- Qdrant filtering: https://qdrant.tech/documentation/concepts/filtering/
Further reading
Use the four sources above to validate the retrieval model and each competitor's current filter syntax. Then run one shared marketplace corpus through every candidate; vendor examples are integration evidence, not your index-cost benchmark.
Top comments (0)