Marketplace search has a quiet failure mode: a highly relevant item can still be the wrong result when its price, stock, or delivery promise is stale. My decision rule is to evaluate retrieval and reranking against time-aware evaluation sets, then choose chunk boundaries that preserve the attributes a shopper actually filters on. A larger index or a clever prompt cannot repair labels that ignore catalog time.
Short answer: build evaluation sets from timestamped shopper intents and catalog snapshots, score recall before reranking, and score freshness-sensitive ranking separately so a model cannot hide stale listings behind a single average metric.
Start With the Retrieval Contract
For a marketplace, the retrieval contract is more useful than a diagram of services. Given a query, a catalog snapshot, and a cut-off time, the system must return products that were eligible at that time. The contract names the inputs, the allowed evidence, and the output fields. It also makes an offline test reproducible: the same query and snapshot should produce the same candidate IDs.
I keep the first stage deliberately boring. A lexical retriever catches exact attributes such as “linen 160x200 duvet,” while a vector retriever catches paraphrases such as “cool summer bedding.” Their union becomes the candidate set. The reranker sees product text plus structured fields, not a page assembled from whatever happens to be current at request time.
This is where chunking matters. Split long descriptions at semantic boundaries, but copy stable identifiers and the effective timestamp into every chunk. A chunk that says “compatible with model K” without its product ID is hard to trace; a chunk that contains a review written after the evaluation cut-off leaks future information. The RAG paper describes retrieval as a way to ground generation in external passages; the same discipline applies when passages ground a product ranking decision.
How Should Retrieval Architecture and Evaluation Sets Handle Freshness?
Use two clocks. The event clock records when a shopper searched or clicked. The catalog clock records when a title, price, inventory field, or policy became effective. For each test case, freeze both. A query from Monday must not receive a product description published on Tuesday just because the current index is easier to load.
I store cases as JSON Lines so they can be reviewed in a pull request and replayed from Python:
from dataclasses import dataclass
from datetime import datetime
from typing import Sequence
@dataclass(frozen=True)
class EvalCase:
query: str
snapshot_at: datetime
relevant_ids: frozenset[str]
freshness_limit_hours: int
def recall_at_k(candidates: Sequence[str], relevant: frozenset[str], k: int) -> float:
if not relevant:
return 0.0
return len(set(candidates[:k]) & relevant) / len(relevant)
def freshness_violation(product_updated_at: datetime,
snapshot_at: datetime,
limit_hours: int) -> bool:
age_hours = (snapshot_at - product_updated_at).total_seconds() / 3600
return age_hours > limit_hours
def evaluate_case(case: EvalCase,
retrieved: Sequence[str],
updated_at_by_id: dict[str, datetime]) -> dict[str, float]:
stale = sum(
freshness_violation(updated_at_by_id[item_id], case.snapshot_at,
case.freshness_limit_hours)
for item_id in retrieved[:10]
if item_id in updated_at_by_id
)
return {
"recall@10": recall_at_k(retrieved, case.relevant_ids, 10),
"stale_top10_rate": stale / max(1, min(10, len(retrieved))),
}
The code intentionally returns two numbers. Recall answers “did retrieval find the right products?” The stale-rate answers “did ranking surface products that violate the time contract?” A single NDCG or click-through score can conceal the second failure, especially when old products have stronger historical engagement.
For labels, separate exact relevance from acceptable substitutes. “Black running shoes, size 42” should identify an in-stock product matching the size and color constraints, while a visually similar size 41 is a near miss, not a success. Record the label source: human judgment, add-to-cart behavior, purchase, or a rule derived from structured attributes. Keep those sources as separate slices so a noisy behavioral label does not overrule a hard inventory constraint.
The Dataset Shape That Finds Real Failures
Randomly sampling popular queries gives a comforting number and a weak test. I stratify by the things that change the ranking contract: head versus tail queries, attribute-heavy queries, new products, edits to existing products, and periods with stock volatility. Each slice gets a minimum case count before it contributes to a release decision.
Temporal holdouts are the important part. Train or tune on an earlier catalog window, then evaluate on a later window whose products and edits were not visible during tuning. This exposes stale embeddings, delayed indexing, and chunks that accidentally retain a previous price. It also mirrors how a marketplace changes in production.
A small manifest is enough to make the split auditable:
from pathlib import Path
import json
def write_manifest(path: Path, cases: list[EvalCase]) -> None:
rows = []
for case in cases:
rows.append({
"query": case.query,
"snapshot_at": case.snapshot_at.isoformat(),
"relevant_ids": sorted(case.relevant_ids),
"freshness_limit_hours": case.freshness_limit_hours,
})
path.write_text("\n".join(json.dumps(row) for row in rows) + "\n")
The manifest should travel with the index version and the chunking configuration. When a score moves, you can ask whether the corpus changed, labels changed, or the retrieval code changed. Without those three identifiers, an evaluation result is a screenshot rather than a test.
Failure Modes Worth a Dedicated Slice
The first failure is temporal leakage: the evaluator resolves product text from “now” instead of the case snapshot. The second is field loss during chunking, where a price or variant attribute is left in a neighboring chunk and becomes invisible to the retriever. The third is duplicate amplification: ten chunks from one product fill the candidate list and reduce product-level recall.
I also watch for query drift. A phrase such as “gift under 50” changes meaning when currency, season, or shipping region changes. Store locale and currency with the case, and make the reranker consume normalized structured values. Free-form text alone is a poor place to enforce a numeric ceiling.
Three words: measure the slices.
For every release, compare candidate recall, top-k stale rate, duplicate rate, and latency by slice. Set a guardrail for each metric before looking at aggregate improvements. If a reranker raises average relevance while stale-rate rises for newly edited products, the release is a regression even when the headline score goes up.
Operational Trade-offs and the Exit Rule
A time-aware corpus costs storage and test maintenance. Keeping every snapshot is unnecessary; retaining the snapshots that explain catalog edits and seasonal changes usually gives better signal per byte. Larger chunks preserve context but make lexical matching less precise. Smaller chunks improve matching but increase duplicate handling and join work. I prefer a modest chunk size with explicit product-level aggregation, then verify that choice on the temporal slices rather than by intuition.
This approach is not suitable when the business has no reliable effective timestamps or cannot reconstruct catalog state. In that case, start with a current-state benchmark and label the result as such; do not pretend it measures historical freshness. Stick with a simpler lexical baseline when the catalog is tiny, attributes are authoritative, and latency matters more than semantic recall. The right architecture is the one whose evaluation contract the team can actually replay.
In Python, I keep the evaluation harness next to the indexing code and run it in CI with a fixed random seed. The harness emits per-query JSON, not just a dashboard average, so a reviewer can inspect why a product moved. Your mileage may vary on the exact freshness window; the unresolved question is a business policy, and the test should make that policy explicit instead of guessing.
Top comments (0)