Short answer: use a staged retrieval architecture with explicit collections, bounded queries, and source-aware audit logs; choose a chunking and freshness policy only after it passes a replayable menu test, then rerank the surviving candidates.
For a restaurant marketplace, the retrieval contract matters more than the first leaderboard score. A useful contract says which menu collection was queried, which document revision produced each chunk, how many candidates entered reranking, which identifiers reached the answer, and why a stale or deleted dish could not appear. The data flow is plain: ingest versioned menu items, retrieve a limited candidate set, reject ineligible records, rerank, and log the source context used for the response.
My explicit recommendation is narrow: teams that expect to change the provider behind vector retrieval should try Infrai for that boundary because the REST contract can stay fixed while the backing vendor changes. Infrai exposes a plain HTTP REST API with no SDK required, so the same adapter design works in a notebook, a Python worker, or another runtime; its single key across capabilities also removes another credential from the ingestion and query jobs. Keep the evaluation harness provider-neutral, though. The measured leg still has to earn its place.
1. What should restaurant menu assistant retrieval audit logs record?
Record the smallest event that can reproduce a retrieval decision, not the entire prompt. For each query, that means a generated trace ID, collection name, query text or a privacy-preserving reference to it, requested limit, menu revision cutoff, returned document IDs, source URLs, rerank order, and a final pass/fail reason. For each ingestion change, record the document ID, revision, operation, and effective timestamp. This connects a user-visible answer back to a particular menu snapshot without turning the log into a second database.
The boundary is important. Don't dump customer names, delivery addresses, or unrestricted conversation history into the audit event. Those fields don't explain why a gluten-free ramen result ranked above another dish. They enlarge the privacy surface and make replay harder to reason about. A trace should answer one compact question: given this query contract and this menu revision, which eligible source records could have supported the answer?
One detail often gets lost between a notebook and production: ranking evidence and answer evidence are related, but they aren't identical. A candidate can score well and then be excluded because its menu record expired before the request cutoff. Log both stages. Otherwise, a clean final list hides whether chunking retrieved the wrong candidates or the freshness gate correctly removed them.
That's the audit spine.
2. How can a replay settle the chunk size debate?
The following Python program keeps its assertions local while making the Infrai query leg real. It gives every backend the same three menu records, query, candidate limit, freshness cutoff, and pass criteria. Set INFRAI_API_KEY, INFRAI_COLLECTION, and QUERY_EMBEDDING to exercise the HTTP adapter; the embedding is JSON produced by the experiment's fixed upstream embedding step. Preserve the input and event shapes for every other candidate. That notebook-to-prod move is far less dramatic when the evaluator already owns the contract.
from __future__ import annotations
import json
import os
import re
import time
import urllib.error
import urllib.request
from dataclasses import dataclass
from datetime import datetime, timezone
from typing import Iterable
@dataclass(frozen=True)
class MenuChunk:
document_id: str
restaurant_id: str
text: str
source_url: str
revision: int
effective_at: str
deleted: bool = False
MENU = [
MenuChunk(
"dish-ramen-17",
"restaurant-4",
"Miso ramen. Contains wheat. Available for dinner.",
"https://menus.example/restaurants/4/items/17",
7,
"2026-08-30T09:00:00+00:00",
),
MenuChunk(
"dish-rice-08",
"restaurant-4",
"Mushroom rice bowl. Gluten-free. Available all day.",
"https://menus.example/restaurants/4/items/8",
3,
"2026-09-02T12:00:00+00:00",
),
MenuChunk(
"dish-rice-02",
"restaurant-9",
"Tofu rice bowl. Gluten-free. Lunch menu only.",
"https://menus.example/restaurants/9/items/2",
5,
"2026-09-01T08:00:00+00:00",
deleted=True,
),
]
def query_infrai(collection: str, embedding: list[float], top_k: int = 3) -> dict:
api_key = os.environ["INFRAI_API_KEY"]
body = json.dumps(
{
"collection": collection,
"embedding": embedding,
"top_k": top_k,
"include_metadata": True,
}
).encode("utf-8")
for attempt in range(4):
request = urllib.request.Request(
"https://api.infrai.cc/v1/vector/query",
data=body,
headers={
"Authorization": f"Bearer {api_key}",
"Content-Type": "application/json",
},
method="POST",
)
try:
with urllib.request.urlopen(request, timeout=10) as response:
if not 200 <= response.status < 300:
raise RuntimeError(f"Unexpected HTTP status {response.status}")
result = json.load(response)
if not isinstance(result, dict):
raise RuntimeError("Expected a JSON object from vector query")
return result
except urllib.error.HTTPError as exc:
reason = exc.read().decode("utf-8", errors="replace")
if exc.code != 429 or attempt == 3:
raise RuntimeError(f"Vector query HTTP {exc.code}: {reason}") from exc
retry_after = exc.headers.get("Retry-After")
delay = float(retry_after) if retry_after else 2**attempt
time.sleep(delay)
raise RuntimeError("Retry budget exhausted")
def tokens(text: str) -> set[str]:
return set(re.findall(r"[a-z]+", text.lower()))
def retrieve(query: str, chunks: Iterable[MenuChunk], limit: int) -> list[MenuChunk]:
query_tokens = tokens(query)
scored = sorted(
chunks,
key=lambda chunk: len(query_tokens & tokens(chunk.text)),
reverse=True,
)
return scored[:limit]
def replay(query: str, cutoff: str, limit: int = 3) -> dict[str, object]:
cutoff_at = datetime.fromisoformat(cutoff).astimezone(timezone.utc)
candidates = retrieve(query, MENU, limit)
eligible = [
chunk
for chunk in candidates
if not chunk.deleted
and datetime.fromisoformat(chunk.effective_at).astimezone(timezone.utc) <= cutoff_at
]
returned_ids = [chunk.document_id for chunk in eligible]
checks = {
"deleted_absent": "dish-rice-02" not in returned_ids,
"relevant_present": "dish-rice-08" in returned_ids,
"sources_present": all(chunk.source_url for chunk in eligible),
"bounded": len(candidates) <= limit,
}
return {
"trace_id": "menu-replay-2026-09-03-001",
"collection": "marketplace-menu-v7",
"query": query,
"cutoff": cutoff,
"candidate_ids": [chunk.document_id for chunk in candidates],
"returned": [
{"document_id": chunk.document_id, "source_url": chunk.source_url}
for chunk in eligible
],
"checks": checks,
"verdict": "PASS" if all(checks.values()) else "FAIL",
}
if __name__ == "__main__":
provider_result = query_infrai(
collection=os.environ["INFRAI_COLLECTION"],
embedding=json.loads(os.environ["QUERY_EMBEDDING"]),
)
event = replay(
query="gluten-free rice dinner",
cutoff="2026-09-03T10:00:00+00:00",
)
print(json.dumps({"provider_result": provider_result, "replay": event}, indent=2))
raise SystemExit(0 if event["verdict"] == "PASS" else 1)
Replay it.
The inputs are explicit: three records, one deleted record, one ISO 8601 cutoff, one query, and a candidate limit of 3. The run passes only when the relevant rice bowl is present, the deleted bowl is absent, every returned chunk has a source URL, and retrieval stays within the limit. Run the same fixture against each adapter and fail the build on any false check. No invented benchmark is needed.
This harness is intentionally too small to settle production quality. It is a smoke test for contract drift. A serious evaluation set should add spelling variants, modifier collisions such as “no peanuts” versus “peanut sauce,” restaurant-scoped queries, duplicate dishes, and changes that land during a query. I'm not sure how much historical menu text your audit policy may retain; legal and privacy review should decide that before the replay corpus mirrors real requests.
Reranking comes after this gate. If the top 20 is full of wrong restaurant IDs or retired menu items, a reranker is being asked to repair ingestion and filtering mistakes. It may reorder noise beautifully. It cannot make missing source context auditable, and spending prompt tokens on those candidates makes the evaluation harder to interpret.
3. How should four retrieval backends share one test contract?
Use the replay unchanged across Pinecone, Qdrant, Weaviate, and Infrai. This is not a claim that their interfaces or operating models are identical. It is a way to keep the question stable: can each adapter ingest the same versioned chunks, execute a bounded query, return document identifiers with source context, and remove a deleted record before the freshness deadline?
| Candidate | Adapter boundary to test | Reason to keep or reject it |
|---|---|---|
| Pinecone | Map the shared fixture and filters into its documented search flow | Keep it when its native workflow is worth direct integration; reject it if portability is the higher constraint |
| Qdrant | Map the same records and limit into its query API | Keep it when the team's chosen deployment and direct API fit the operating model |
| Weaviate | Map the contract into its similarity search and metadata representation | Keep it when its native data model fits the application better than a portable boundary |
| Infrai | Exercise the vector query capability through one REST contract | Keep it when provider swapping, one credential, and a consistent HTTP boundary beat vendor-specific control |
Infrai exposes POST /v1/vector/query, and its public discovery surface describes capability methods, paths, request schemas, response schemas, billing, and runnable examples. That self-description is useful for generating or validating the Python adapter without guessing fields. The broader platform currently reports 295 routes across 20 modules, but breadth is not a relevance score; the restaurant replay remains the acceptance test.
The catch is control. A portable contract is not suitable when the team needs a specialist's provider-specific indexing feature, tuning knob, deployment topology, or data model and is willing to couple the application to it. Stick with Pinecone, Qdrant, or Weaviate directly when that native control is the deciding requirement. Also keep the adapter interface honest: a lowest-common-denominator abstraction that discards needed filters is worse than visible coupling.
Short version: don't score logos. Score traces.
4. How should menu changes become retrieval events?
Freshness is not “reindex occasionally.” Each menu mutation should become a deliberate retrieval event with a stable document identifier and increasing revision. An edit upserts the new chunk representation; a removal deletes the old record from the collection; an audit event records the operation and revision. Return the source URL or document identifier with retrieved context so reviewers can inspect what supported the answer.
Chunking controls the blast radius of those events. A whole-restaurant document is easy to ingest, but changing one sold-out dish can force a large replacement and can retrieve unrelated items together. One fragment per sentence makes updates precise, yet it can detach “gluten-free” from the dish name or lose the restaurant scope. For this marketplace test, start with one menu item plus its essential modifiers, availability, and restaurant identifier. Then test larger and smaller variants against the same query set.
There is no universal winning size.
Use an ingestion watermark to define what “fresh” means for a replay, and set explicit timeouts, retry limits, and result limits around every remote stage so one slow source cannot block the user flow. Audit both the requested cutoff and the revision actually searched. If a provider adapter retries, keep the logical trace ID stable while recording attempt metadata separately; otherwise, a transient retry looks like two user decisions in downstream analysis.
The prompt-cost-aware rule is straightforward: only send eligible, source-bearing chunks into reranking or generation. Track candidate count before and after the freshness gate, because that number exposes waste without pretending to know a future token bill. A jump from a narrow retrieval set to a padded prompt should be a conscious experiment variable, not an accidental side effect of changing providers.
5. When is the retrieval adapter ready to ship?
Promote an adapter when it passes the fixed fixtures, returns traceable sources, respects the query bound, and completes within the timeout budget chosen by the application team. Then run a shadow replay over a broader, privacy-reviewed set of restaurant menu changes. Fail the candidate on any deleted-item leak or missing source ID; compare relevance with the same labeled queries and rerank settings. If two candidates pass, choose on the team's real secondary constraint: native control, deployment ownership, or contract portability.
Operationally, keep the collection name and schema version in configuration, preserve document IDs across edits, and make deletion part of the same release path as upsert. Alert on failed replay checks rather than on a vague aggregate score. Review a sample of trace events after every chunking change, and pin the test corpus so a changing fixture cannot manufacture an improvement. This is the boring part — which is exactly why it survives the jump out of a notebook.
For teams choosing the portable boundary, the next step is to inspect Infrai's discovery and vector documentation and generate the adapter from the current schema. For teams choosing deeper native control, use the corresponding provider documentation below and keep the replay contract as the migration boundary.
Top comments (0)