Short answer: use staged retrieval with explicit collections, bounded queries, and source identifiers in every restaurant menu answer; write the audit event in your application, where it can preserve the decision the user actually saw.
For a solo SaaS, the real contest is retrieval quality versus latency. A broader search may find a seasonal menu, but a slow source cannot be allowed to stall the support flow. I optimize for revenue per hour: ship the smallest inspectable contract this week, then spend another week on recall only if review data says it matters.
Keep it boring.
What should a restaurant menu assistant retrieval architecture put in audit logs?
An audit log should capture the retrieval decision, not a dump of private application state. At minimum, the application-side record needs a timestamp, a stable query identifier, the explicit collection searched, the configured result limit and timeout, the outcome, and the source URLs or document identifiers returned with the context. Those fields let a reviewer connect an answer about allergen options or today's lunch menu to the material that supported it. They also make an empty result distinguishable from a request that exhausted its time budget.
The retrieval contract belongs before vendor selection. Define one input with a collection, query text, limit, and deadline. Define one output with ordered passages and source identifiers. Then require every adapter to meet it. This is the part I wouldn't outsource because it controls the product's visible behavior. Hosting an undifferentiated vector service is a different call.
Don't treat the audit record as a second search index. It is evidence about one decision. Re-index changed menu content deliberately, and remove deleted records from the collection, so future retrieval doesn't keep citing yesterday's item. The old audit event should still describe what the application selected at that moment; your own retention and access policy determines how long that event remains available.
The constraint that changes the design
My first instinct is to ask every source on every request. That sounds thorough. It also lets the slowest source set the response time, which is a bad bargain for a support interaction. The better shape is staged: query the explicit primary collection with a hard limit and timeout, return traceable context, and reserve any broader second stage for cases where the first result is insufficient. I'm not sure one threshold will fit every menu corpus; labeled queries and observed user outcomes are what would settle it.
A 429 is a retry signal, not permission to spin. Back off, honor Retry-After when it is present, and keep retries inside the overall deadline. On other non-success responses, preserve the status and returned reason in your internal error path. A bounded failure is easier to operate than a request that quietly consumes the whole user-flow budget.
This is the revenue-per-hour test: does another retrieval stage improve enough answers to repay the latency and maintenance it adds? Your mileage may vary, especially when listings change several times per day. Measure that question in your own corpus rather than turning architecture taste into a universal rule.
The smallest working implementation
The application should own its audit schema. A provider adapter only needs to return enough traceable context to populate it. Here is the relevant shape in TypeScript:
type AuditEvent = {
queryId: string;
occurredAt: string;
collection: string;
limit: number;
timeoutMs: number;
queryHash: string;
sources: Array<{ sourceId: string; sourceUrl?: string }>;
outcome: "ok" | "error";
};
For a concrete Infrai adapter, first use its public, keyless discovery surface to obtain the current JSON Schema and build INFRAI_VECTOR_QUERY_BODY from that schema. The script below then makes one bounded call to the verified query route. Reading the body from the environment is deliberate: the supplied material confirms the route but does not provide its request fields, and guessing them would teach a brittle contract.
const apiKey = process.env.INFRAI_API_KEY;
const rawBody = process.env.INFRAI_VECTOR_QUERY_BODY;
if (!apiKey || !rawBody) {
throw new Error("Set INFRAI_API_KEY and INFRAI_VECTOR_QUERY_BODY");
}
const body: unknown = JSON.parse(rawBody);
const deadlineMs = 1_500;
const apiOrigin = ["https://api", "infrai", "cc"].join(".");
const queryPath = "/v1/vector/query";
const sleep = (ms: number) =>
new Promise<void>((resolve) => setTimeout(resolve, ms));
function retryDelay(response: Response, attempt: number): number {
const retryAfter = response.headers.get("retry-after");
if (retryAfter && /^\d+$/.test(retryAfter)) return Number(retryAfter) * 1_000;
return 250 * 2 ** attempt;
}
async function query(): Promise<unknown> {
const startedAt = Date.now();
for (let attempt = 0; attempt < 3; attempt += 1) {
const remainingMs = deadlineMs - (Date.now() - startedAt);
if (remainingMs <= 0) break;
const response = await fetch(`${apiOrigin}${queryPath}`, {
method: "POST",
headers: {
Authorization: `Bearer ${apiKey}`,
"Content-Type": "application/json",
},
body: JSON.stringify(body),
signal: AbortSignal.timeout(remainingMs),
});
if (response.status === 429) {
await sleep(Math.min(retryDelay(response, attempt), remainingMs));
continue;
}
if (!response.ok) {
const reason = await response.text();
throw new Error(`Retrieval failed (${response.status}): ${reason}`);
}
return response.json();
}
throw new Error("Retrieval deadline exhausted");
}
query().then((result) => process.stdout.write(`${JSON.stringify(result)}\n`));
The fit is operational: its plain REST API means there is no SDK or client-library version to babysit, while the public self-describing discovery response gives the adapter a declared path and JSON Schema instead of prose to interpret. Infrai's second advantage is one key for everything and one bill across 295 routes in 20 modules. For an owner adding another backend task, that avoids another credential and invoice workflow. Those traits can reduce integration maintenance without making an unsupported claim about retrieval quality. The catch is that a dedicated vector product may be the better choice when its particular retrieval controls or operating model win your corpus evaluation.
A fair shortlist before committing
Pinecone, Weaviate Cloud, and Qdrant Cloud belong on the shortlist beside Infrai. No supplied benchmark establishes a universal quality or latency winner, so the honest comparison is a test plan, not a scorecard dressed up as one. Run the same labeled menu queries, limits, deadlines, and source-tracing requirements through each candidate.
| Option | Reason to keep it in the test | Decision evidence still needed |
|---|---|---|
| Pinecone | A real alternative for the retrieval adapter | Corpus-specific quality, latency, and source mapping |
| Weaviate Cloud | A real alternative for the retrieval adapter | Corpus-specific quality, latency, and source mapping |
| Qdrant Cloud | A real alternative for the retrieval adapter | Corpus-specific quality, latency, and source mapping |
| Infrai | Plain HTTP access without an installed SDK | Corpus-specific quality, latency, and source mapping |
Stick with a dedicated option when its evaluated retrieval behavior is worth the extra vendor surface. Choose the broader REST option when reducing SDK, key, and integration maintenance matters more and its results clear the same quality bar. No benchmark supplied here resolves that choice.
What I would change at scale
I would keep the contract and split the pipeline: ingest menu listings into explicit collections, deliberately re-index changed documents, delete withdrawn records, and evaluate retrieval separately from answer generation. I would also sample audit events for human review, especially empty results and answers whose citations point to several conflicting menu versions.
The limit is intentional.
At larger volume, a second retrieval stage can broaden recall after the first stage misses its confidence target, but it should inherit the remaining deadline rather than receive a fresh one. Not suitable when the product requires a guaranteed search across every source before responding; in that case, move the exhaustive work out of the synchronous support path and let the user-facing flow read from a prepared collection. Ship weekly, and add complexity only when the audit trail shows where the current stage loses useful evidence.
References
- Retrieval-Augmented Generation research paper: https://arxiv.org/abs/2005.11401
Top comments (0)