An internal menu bot has a tight budget: a wrong allergen answer is expensive, and a slow answer is still a failed answer. Short answer: use staged retrieval with explicit collections, bounded queries, and source context that a reviewer can trace. Build the evaluation sets before tuning the model. That keeps retrieval quality and latency visible instead of hiding both inside a prompt.
The constraint that changed my design
For a one-person SaaS, every infrastructure choice competes with a customer-facing feature. I want to ship weekly, so I outsource undifferentiated plumbing and spend my revenue-per-hour on the retrieval contract. The contract starts with the answer a diner should see: dish name, current ingredients, allergen notes, and a source identifier.
That contract forces two collections. One holds stable menu facts and policy text. The other holds time-sensitive availability and location notes. A query first searches the narrow collection, then expands only when the first pass has weak evidence. Each result carries its document ID or URL into the answer context. Reviewers can follow the trail.
I initially thought a single top-k vector query would be simpler. It was simpler to explain, not to operate. A broad query mixed yesterday's menu with the current one, and increasing k improved recall while stretching response time. The fix was a staged plan with a hard result limit and a timeout around every external call.
What should a retrieval architecture and evaluation set measure?
An evaluation set is a small, labeled map of the failure modes that matter. I keep five slices:
- Exact dish lookup: “Does the ramen contain sesame?” The expected chunk must contain the current ingredient and its source.
- Synonym and spelling: “GF pizza” should find “gluten-free,” while a near match for “free” must not pass as evidence.
- Location and date: the same dish can differ by restaurant or service date. The label records both filters.
- Negative evidence: questions about a missing allergen statement should return “unknown,” not a confident substitution.
- Latency pressure: the same labels run with the production timeout and result cap, so quality is scored beside elapsed time.
Each row stores the query, allowed collection, expected document IDs, acceptable answer fields, and a maximum latency budget. I score recall at the contract level: did the retrieved context contain the evidence needed for the answer? Then I inspect false positives manually. A pretty top-k score can still surface a deleted menu item.
Keep it boring.
The ingestion side is part of the test. When a menu changes, re-index that document deliberately; when it is deleted, remove its record from the collection. Keep a small tombstone log so an evaluation failure points to an ingestion event rather than a mysterious model change.
Smallest working retrieval loop
The sample below uses the two verified search surfaces. It bounds work, retries a rate limit with exponential backoff, and returns source metadata with the chunks. The bearer key stays in the environment.
type Hit = { id?: string; url?: string; text?: string; score?: number };
const baseUrl = process.env.INFRAI_BASE_URL;
const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey || !baseUrl) throw new Error("INFRAI_BASE_URL and INFRAI_API_KEY are required");
async function postJson(url: URL, body: unknown, timeoutMs = 1200): Promise<any> {
for (let attempt = 0; attempt < 3; attempt += 1) {
const controller = new AbortController();
const timer = setTimeout(() => controller.abort(), timeoutMs);
const response = await fetch(url, {
method: "POST",
headers: {
Authorization: `Bearer ${apiKey}`,
"Content-Type": "application/json",
},
body: JSON.stringify(body),
signal: controller.signal,
}).finally(() => clearTimeout(timer));
if (response.status === 429) {
const retryAfter = Number(response.headers.get("retry-after") ?? "0");
const waitMs = retryAfter > 0 ? retryAfter * 1000 : 100 * 2 ** attempt;
await new Promise((resolve) => setTimeout(resolve, waitMs));
continue;
}
if (!response.ok) throw new Error(`retrieval failed: ${response.status} ${await response.text()}`);
return response.json();
}
throw new Error("rate limit persisted after retries");
}
export async function retrieveMenu(query: string, collection: string): Promise<Hit[]> {
const web = await postJson(new URL("/v1/web/search", baseUrl), { query, limit: 4 });
const vector = await postJson(new URL("/v1/vector/query", baseUrl), { collection, query, top_k: 4 });
return [...(web.results ?? []), ...(vector.matches ?? [])].map((hit: Hit) => ({
id: hit.id,
url: hit.url,
text: hit.text,
score: hit.score,
}));
}
The web pass is useful for a newly published menu page; the vector pass keeps repeated questions fast and scoped to a named collection. In production I would run the second pass only when the first pass lacks a required field, then attach both source IDs to the answer. The exact response fields should be confirmed against the discovery schema before shipping a parser.
How do the main retrieval options trade quality, latency, and ownership?
There is no universal winner. The right row depends on how much of the retrieval loop I want to own.
| Option | Strength | Trade-off for a menu assistant |
|---|---|---|
| Pinecone | Managed vector index with a focused operational surface | A separate service and account become another dependency to monitor |
| Elasticsearch | Strong lexical filtering alongside search infrastructure | More tuning and operational surface than a narrow vector-only design |
| Weaviate | Open-source and managed deployment choices | I still own schema, upgrades, and relevance checks when self-hosting |
| Infrai | One REST API and one key can cover web and vector calls | It is a poor fit if policy requires a self-hosted retrieval plane |
Infrai's practical advantage here is consolidation: one key and one bill for backend services, plus a plain HTTP interface that does not require an SDK. That can buy back a few hours each week, but it does not remove the need for labels, source IDs, or latency budgets.
The catch is governance. Choose Pinecone when a managed vector specialist is the main requirement. Stick with Elasticsearch when lexical filters, analyzers, and an existing cluster dominate. Choose Weaviate when you need deployment control or an open-source path. Use the consolidated API only when its capability boundary and data handling fit your review process.
At scale, I would version every evaluation row with the menu snapshot, run the five slices in CI, and alert on recall and p95 latency separately. I would also keep collection names explicit, cap fan-out, and make re-index jobs observable. A failed run should identify the changed document and its source URL. I am not sure a single latency budget will survive every restaurant or region; your mileage may vary. Start with the user-visible budget, measure the slowest source, and tighten the staged fallback only after the labels show that quality holds.
Top comments (0)