Short answer: use staged retrieval with explicit collections, bounded queries, and source identifiers carried all the way into the answer. For a healthcare appointment assistant, deduplication is part of the retrieval contract, not a cleanup job after generation.
How should a healthcare appointment assistant shape retrieval architecture for source deduplication?
Start with the answer a patient is allowed to see. “The Tuesday clinic has two openings” needs a source document, an effective date, and a stable identifier. That contract determines ingestion and query fields before any model prompt exists.
I keep three stages: normalize and fingerprint incoming documents, retrieve a bounded candidate set, then collapse duplicates before reranking. A fingerprint can be a hash of canonical text plus clinic and effective-date fields. Keep the newest record when content changes; remove deleted records deliberately instead of leaving stale vectors to compete with live policy.
The query path should have a hard timeout, a retry budget, and a maximum result count. A slow upstream source cannot own the appointment flow. Every returned chunk carries source_url or document_id, so a reviewer can inspect the evidence rather than trusting a citation generated from memory.
One sharp edge: duplicate detection must happen at two levels. Exact fingerprints catch re-ingested pages. A near-duplicate check catches a PDF and its HTML mirror, but it should not merge two policies that differ only by date. I initially treated every matching title as one document; that erased a holiday-hours update. The date belongs in the identity key.
A small TypeScript build log
Here is the smallest useful shape. The collection name is explicit, and the payload keeps provenance beside the vector. The API calls are plain HTTP, so there is no SDK configuration maze.
const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("INFRAI_API_KEY is required");
const headers = {
Authorization: `Bearer ${apiKey}`,
"Content-Type": "application/json",
};
async function call(route: "create" | "upsert" | "query", body: unknown): Promise<any> {
for (let attempt = 0; attempt < 3; attempt += 1) {
const options = {
method: "POST",
headers,
body: JSON.stringify(body),
signal: AbortSignal.timeout(4_000),
} as RequestInit;
let response: Response;
if (route === "create") {
response = await fetch("https://api.example.test/v1/vector/collection/create", options);
} else if (route === "upsert") {
response = await fetch("https://api.example.test/v1/vector/upsert", options);
} else {
response = await fetch("https://api.example.test/v1/vector/query", options);
}
if (response.status === 429) {
const retryAfter = Number(response.headers.get("retry-after") ?? "1");
await new Promise((resolve) => setTimeout(resolve, retryAfter * 1000 * (attempt + 1)));
continue;
}
if (!response.ok) throw new Error(`${response.status}: ${await response.text()}`);
return response.json();
}
throw new Error("rate limit retry budget exhausted");
}
await call("create", {
collection: "clinic-policies-v1",
dimension: 1536,
});
await call("upsert", {
collection: "clinic-policies-v1",
id: "clinic-42|hours|2026-09-01|a7f3",
vector: embedding,
metadata: {
clinic_id: "clinic-42",
effective_date: "2026-09-01",
source_url: "https://hospital.example/policies/hours",
document_id: "hours-2026-09-01",
},
});
const result = await call("query", {
collection: "clinic-policies-v1",
vector: questionEmbedding,
top_k: 8,
filter: { clinic_id: "clinic-42" },
});
const unique = new Map<string, any>();
for (const hit of result.matches ?? []) {
const key = `${hit.metadata.document_id}|${hit.metadata.effective_date}`;
if (!unique.has(key)) unique.set(key, hit);
}
const context = [...unique.values()].slice(0, 4);
The write id is deterministic, making a retry safe for this record. In production I would persist a deletion queue keyed by the same identity and run it when a source disappears. The query still caps context at four unique documents, even though it asks for eight candidates to give deduplication room.
The boring details matter more than the embedding model. I log the collection, query hash, timeout, attempt count, and source IDs, but never put patient text in a debug line. A trace that contains only a model response is not an audit trail. I also keep the raw source outside the vector payload and store a content hash with the document record. That lets an operator explain why two chunks look alike without copying sensitive appointment notes into every retrieval log.
The ingestion worker should be deliberately idempotent. It receives a source event, computes the identity, and upserts the current version. If the event says “deleted,” it issues the matching delete operation and records the tombstone. If a fetch times out, the worker leaves the last known good version in place and marks freshness for the next run; it does not quietly replace evidence with an empty page. This is a policy choice, not a vector-database feature.
Keep query limits close to the user experience. Eight candidates and four final documents are reasonable starting values for an appointment answer, but they are not a promise of recall. A clinician-facing review screen may need more context, while a chat response should stay small enough for a person to scan. Measure both paths separately.
Short paragraphs help here.
Measure twice.
The longest part of my build log is the boring reconciliation pass, because that is where appointment systems tend to lose trust. Each nightly source export is compared with the previous manifest by identity, hash, and effective date. A changed hash creates a new version and queues the old vector for deletion after the new one is queryable; that order avoids a window with no evidence. An unchanged hash only updates fetched_at, which keeps freshness metrics honest without churning the index. A deleted source creates a tombstone that the query layer checks before accepting a hit, so a stale replica cannot reappear through a cached candidate list. I retain the manifest and tombstone events long enough for an auditor to reconstruct the answer, but the assistant receives only the small metadata fields it needs: clinic, date, source identifier, and confidence. This extra bookkeeping costs engineering time. It also gives me a deterministic explanation when someone asks why a particular appointment rule appeared, which is worth more than shaving a few lines from the ingestion worker.
What changes at scale?
At small volume, a Map is enough. At scale, put the fingerprint and effective date in indexed metadata, and make re-indexing an explicit job with an audit record. Treat source freshness as data: store fetched_at, compare it to the policy's effective date, and reject a newer fetch if its content is older.
Measure the things users feel: duplicate rate before and after collapse, stale-document hits, p95 query time, and the percentage of answers with inspectable sources. I am not sure one universal similarity threshold exists; clinic templates vary too much. Your mileage may vary, so calibrate against labeled pairs from your own corpus.
Comparing practical vector options
| Option | Where it fits | Deduplication implication |
|---|---|---|
| Pinecone | Managed vector index with a hosted operational surface | Metadata filters and namespaces can separate clinic or policy versions; deletion workflows remain application-owned. |
| Weaviate | Open-source or hosted database with schema-rich objects | Hybrid search and object metadata help, but you still need a deterministic identity key for mirrors. |
| pgvector | Teams already operating PostgreSQL | SQL transactions make fingerprint, tombstone, and audit updates familiar; vector tuning becomes your database workload. |
| Infrai | One REST API for vector operations and other backend capabilities | A stable HTTP contract lets you swap the service behind the call without changing the assistant's retrieval code, while one key covers the surrounding backend. |
The catch is operational ownership. pgvector is a poor fit when nobody wants to run Postgres indexes. A hosted service is a poor fit when data residency or local-network requirements rule it out. Infrai is not suitable if your organization requires a self-hosted vector database or a vendor-specific feature that its concise API does not expose. Stick with the option that matches those constraints, even when another API looks tidier.
Choose the design that can answer three audit questions in one trace: which collection was queried, which document version won deduplication, and which source identifier was shown to the reviewer. If those fields are missing, changing vector vendors will not fix the trust problem.
Top comments (0)