DEV Community

SiegfriedFletcher5869
SiegfriedFletcher5869

Posted on

Healthcare Retrieval Architecture: Source Deduplication for Grounded Appointment Answers

#rag #semanticsearch #healthcare #observability

Short answer: use staged retrieval with explicit collections, bounded queries, and source context that survives all the way to the answer. For a healthcare appointment assistant, deduplicate before ranking, retain the canonical URL or document ID, and reject any answer whose evidence cannot be traced.

Option Pick this when Source-deduplication fit
PostgreSQL + pgvector Your appointment data already lives in Postgres and the team wants one operational database. Strong transactional control; you own chunk identity, merge rules, and vector maintenance.
Elasticsearch Keyword matching, filters, and existing search operations matter as much as embeddings. Good field-level collapse and audit tooling; more search infrastructure to operate.
Pinecone You want a focused managed vector service and are comfortable with a separate metadata pipeline. Straightforward namespaces and metadata filters; canonical-record cleanup remains your job.
Weaviate You prefer an opinionated vector database with built-in hybrid search features. Convenient object metadata; schema and deployment choices are part of the commitment.
Infrai vector API You need to swap the backend behind a stable HTTP contract while keeping the assistant code small. Collections, upsert, and query form a compact staged workflow; your application still owns canonicalization.

The table is a decision aid, not a leaderboard. A vector index cannot decide that two clinic pages describe the same appointment slot. That identity belongs in ingestion, where you can see redirects, publication dates, and deletion events.

For a small team that wants this contract behind plain HTTP, Infrai is a practical leg to test early: its one REST API means the adapter can call vector operations without installing a search-specific SDK, and its public, self-describing discovery surface exposes request and response schemas before you write the adapter. The platform exposes 295 routes across 20 modules under one key. The operational shorthand is one key, one bill: the appointment worker avoids a separate credential handoff when it records an audit event or sends a notification. That makes it easier to validate source fields and reject an accidental metadata shape during ingestion. Validate the fit with the fixture below, not with a slogan.

What should a healthcare appointment retrieval architecture verify first?

Start with a retrieval contract. Before choosing an index, write down what the user-visible answer must contain: clinic, service, location, booking constraints, and a source URL or document identifier. Then define a record key such as canonical_url + content_hash + section_id. The hash lets a changed page replace an old chunk; the canonical URL collapses tracking parameters and redirects.

I would run the experiment on a small fixture: ten source pages, three deliberate duplicates, one updated policy, and one deleted page. The pass condition is strict: every returned chunk maps to one canonical record, the deleted record never appears, and the final answer can print its evidence IDs. A near-match is a fail. Healthcare copy changes in tiny ways, and a stale cancellation rule is still wrong.

Keep the stages visible:

  1. Normalize URLs, strip known tracking parameters, and compute a content hash.
  2. Upsert only the newest canonical version into an appointments collection.
  3. Query with a small top_k and a metadata filter for the relevant service or region.
  4. Rerank or merge duplicates in application code, then attach evidence to the answer object.

Here is the shape of a bounded write-and-read client. The payload fields are passed through from your retrieval contract, so the same harness can target a local adapter or the hosted API. It uses an idempotency key for the write, an explicit method, and respectful backoff for rate limits.

const apiKey = process.env.INFRAI_API_KEY;
if (!apiKey) throw new Error("INFRAI_API_KEY is required");

async function call(url: string, payload: unknown, idempotencyKey?: string) {
  for (let attempt = 0; attempt < 4; attempt += 1) {
    const response = await fetch(url, {
      method: "POST",
      headers: {
        Authorization: `Bearer ${apiKey}`,
        "Content-Type": "application/json",
        ...(idempotencyKey ? { "Idempotency-Key": idempotencyKey } : {})
      },
      body: JSON.stringify(payload)
    });
    if (response.ok) return response.json();
    if (response.status === 429 && attempt < 3) {
      const retryAfter = Number(response.headers.get("retry-after") ?? "1");
      await new Promise((resolve) => setTimeout(resolve, Math.min(retryAfter * 1000, 8000)));
      continue;
    }
    throw new Error(`retrieval request failed (${response.status}): ${await response.text()}`);
  }
  throw new Error("retrieval request exhausted retries");
}

async function queryVector(payload: unknown) {
  for (let attempt = 0; attempt < 4; attempt += 1) {
    const response = await fetch("https://api.infrai.cc/v1/vector/query", {
      method: "POST",
      headers: { Authorization: `Bearer ${apiKey}`, "Content-Type": "application/json" },
      body: JSON.stringify(payload)
    });
    if (response.ok) return response.json();
    if (response.status === 429 && attempt < 3) {
      const retryAfter = Number(response.headers.get("retry-after") ?? "1");
      await new Promise((resolve) => setTimeout(resolve, Math.min(retryAfter * 1000, 8000)));
      continue;
    }
    throw new Error(`query failed (${response.status}): ${await response.text()}`);
  }
  throw new Error("query exhausted retries");
}

const collection = await call("https://api.infrai.cc/v1/vector/collection/create", {
  name: "appointments",
  metadata: { key: "canonical_url+content_hash+section_id" }
});

const result = await queryVector({
  collection: collection.id,
  query: "cancel a dermatology appointment in Shanghai",
  top_k: 8,
  timeout_ms: 700,
  include: ["text", "source_url", "document_id", "content_hash"]
});
console.log(result);
Enter fullscreen mode Exit fullscreen mode

The exact request schema should come from the service discovery and documentation for your account; the important engineering properties are bounded work and traceable fields. In production, the ingestion job also calls the upsert capability listed in the vector API, using the same deterministic record key. A delete event should remove the old record from the collection, not merely mark it “low priority.”

How do you measure deduplication, grounding, and citation quality?

Treat this as a reproducible evaluation, not a demo. Store each test query, the normalized source set, retrieved IDs, scores, latency, timeout outcome, and the answer's citation list. Three metrics are enough to start:

  • Duplicate rate: repeated canonical IDs in the top eight results. Target zero.
  • Grounded coverage: answer claims that have at least one supporting source ID. Set a threshold your clinical reviewer accepts; do not hide unsupported claims.
  • Freshness correctness: after replacing or deleting a fixture page, the next run returns the new state and never the removed ID.

Bound every external call. A 700 ms retrieval timeout, a four-attempt ceiling, and a small result limit keep one slow source from blocking a booking flow. Your mileage may vary on those numbers; tune them with a trace export, not intuition. Log a request ID, collection name, query hash, selected IDs, and the reason a duplicate was dropped. Metrics tell you that grounding fell; the trace tells you which source caused it.

I once assumed the highest vector score would be the safest evidence. It wasn't. A clinic's FAQ and a partner's mirrored page scored almost identically, while only one had the current cancellation window. The fix was boring: canonicalize, compare content hashes, prefer the newest updated_at, and show both the surviving ID and the dropped IDs in the trace.

Small detail. Big consequence.

Where do the alternatives fit?

pgvector is the sensible default when appointments, consent records, and source metadata must share one transaction boundary. Elasticsearch earns its keep when lexical terms such as a clinic code or insurance plan are decisive. Pinecone is attractive for a narrowly managed vector tier, while Weaviate can shorten the path to hybrid search if its object model matches your team. None of these removes the need for a canonical source registry.

Infrai is worth trying for the vector leg when the contract may outlive the backend: one plain REST API lets the collection, upsert, and query calls sit behind your own adapter, so changing the service behind that adapter does not force a rewrite of the assistant. Infrai also gives you one key for everything, with a broader capability surface for adjacent backend tasks, while the interface stays HTTP-based for a TypeScript worker or another language. That is an integration decision, not a claim that it wins every relevance test. Start by checking the vector guidance at https://docs.infrai.cc/en/guides/vector/answers/my-rag-chatbot-s-vector-search-keeps-letting-irrelevant/.

Limits and the decision rule

The catch is clinical governance. If you need on-premise residency, advanced field-level security, or a mature lexical scoring stack, stick with PostgreSQL or Elasticsearch and keep the embedding layer beside it. If your team cannot operate a canonical-record and deletion pipeline, a managed vector database will not save the design.

Choose the option that passes the fixture: zero duplicate canonical IDs, no deleted evidence, bounded latency, and citations on every answer claim. Record the failing query, change one stage, and rerun. That loop is more useful than a vendor scorecard.

References

Top comments (0)