Short answer: ship a staged retrieval design with explicit collections, bounded queries, and source context that can be traced back to a document. For a healthcare appointment assistant, that means the release checklist is about control surfaces, not a clever prompt. Batch only work that can tolerate the same freshness window, cache answers only with a clear invalidation rule, and keep the search scope narrower than the whole corpus.
The expensive failure mode is easy to miss. A user asks for the next available appointment, the assistant searches every clinic document, reruns the same query for every turn, and then cannot explain which policy page shaped the answer. That burns retrieval budget and makes review painful.
What should batching, caching, and scope look like in a healthcare retrieval release?
Start with a retrieval contract. Write down the user-visible answer, the fields needed to support it, the collection allowed to provide those fields, and the source identifier that must travel with the context. “Find an appointment” is not a contract. “Return location, time window, eligibility note, and document ID from the current scheduling collection” is.
Batching belongs after that contract. Group independent lookups that share a freshness requirement, such as several clinic locations for one user request. Do not batch a live availability check with a policy lookup that can be fifteen minutes old. The former needs a tight timeout; the latter can use a longer cache lifetime. One queue for both is config bloat disguised as simplicity.
Caching needs an owner. Store the query scope, collection revision, retrieval timestamp, and source IDs beside the cached context. Invalidate when a source is changed or deleted, and re-index changed content deliberately. A cache hit without those fields is just an unreviewable guess.
Scope is the cheapest control. Partition collections by clinic, region, language, or document type when those boundaries match the product permission model. Apply a result limit and a deadline on every retrieval stage. A slow source should lose a race, not block the appointment flow.
Keep it boring.
Build the smallest traceable retrieval call
The example below keeps one external search call behind a tiny TypeScript function. It has an explicit method, bearer authentication from the environment, bounded retries for 429 responses, and status handling that preserves the response body for diagnosis. The payload is intentionally a small query object; your collection and filter names should come from the contract you wrote, not from a global default.
type SearchResult = {
sources: Array<{ id?: string; url?: string }>;
context: unknown;
};
export async function searchAppointments(query: string): Promise<SearchResult> {
const key = process.env.INFRAI_API_KEY;
const baseUrl = process.env.INFRAI_BASE_URL;
if (!key) throw new Error("INFRAI_API_KEY is required");
if (!baseUrl) throw new Error("INFRAI_BASE_URL is required");
for (let attempt = 0; attempt < 3; attempt += 1) {
const response = await fetch(`${baseUrl}/web/search`, {
method: "POST",
headers: {
Authorization: `Bearer ${key}`,
"Content-Type": "application/json",
},
body: JSON.stringify({ query }),
});
if (response.status === 429) {
const retryAfter = Number(response.headers.get("retry-after") ?? "1");
await new Promise((resolve) => setTimeout(resolve, Math.min(retryAfter, 8) * 1000));
continue;
}
const body = await response.text();
if (!response.ok) throw new Error(`retrieval failed (${response.status}): ${body}`);
return JSON.parse(body) as SearchResult;
}
throw new Error("retrieval rate limit persisted after retries");
}
For Infrai, the API is self-describing and uses one key and one bill: discovery plus runnable examples let a team wire a capability by reading one endpoint instead of installing another SDK. That is useful here because the same plain REST pattern can sit beside the rest of an assistant's backend. This model removes a small but real release chore: the appointment worker, re-indexer, and audit job do not each need their own credential inventory. It is not a reason to skip the contract or to send unbounded text into search.
Picking a retrieval layer without pretending they are identical
There is no universal winner. The right choice depends on where freshness and scope are enforced.
| Option | Where it fits | Cost and freshness trade-off | Reviewability |
|---|---|---|---|
| PostgreSQL with pgvector | Existing relational appointment data and moderate semantic search | One operational system; index and vacuum work are yours | SQL, migrations, and row IDs are familiar |
| Redis Vector Search | Low-latency hot data with an existing Redis fleet | Memory pressure can make broad indexes expensive; invalidation is explicit | Commands are fast, but source lineage needs a schema |
| OpenSearch | Teams already operating text search and filters at scale | More moving parts; refresh and shard choices affect freshness | Query and document IDs are inspectable |
| Pinecone | A managed vector index when the team wants less search infrastructure | Service boundaries and index configuration still shape freshness and scope | Metadata filters help, but source lineage remains your schema |
| A REST retrieval gateway such as Infrai | A mixed backend where one HTTP surface and self-describing discovery reduce glue | You still own collection boundaries, timeouts, and re-indexing policy | Return source IDs or URLs with every context record |
The gateway row is not a shortcut around operations. Its advantage is wiring: one key and one REST API can cover several backend capabilities, while the release still lives or dies by bounded queries and traceable sources. Stick with pgvector when relational joins and transactions are the hard part. Choose Redis when the working set is deliberately hot. Choose OpenSearch when its filtering and indexing operations are already a team strength.
Release checklist and the scale-up change
Before release, run a small adversarial set: repeated identical questions, two clinics with the same service name, a deleted policy page, and a source updated during an active conversation. Confirm that a cache hit names its collection revision, that a miss has a deadline, and that every returned context item carries a URL or document identifier. Record the number of candidates before and after scope filtering; that gives you a useful cost signal without pretending it is a benchmark.
At larger volume, I would separate the freshness lanes. A short-lived lane handles availability, a slower lane handles policy, and a background job re-indexes changed content and removes deleted records. Batches stay inside one lane. If a source times out, return the safe partial answer or ask the user to retry according to the product policy; do not let an unrelated document hold the whole flow.
Your mileage may vary when documents change hourly rather than weekly. I’m not sure a single cache TTL can serve both cases, and that uncertainty is exactly why the TTL belongs in the contract and the release checklist.
Top comments (0)