Short answer: rank appointment evidence by policy and time validity first, then blend lexical, semantic, and operational signals; the assistant should answer only when the winning passages carry a citation that a support agent can open.
That ordering is a safety control, not a tuning preference. A patient asking “Can I move my Tuesday visit?” needs the current cancellation window and the clinic’s location, not a semantically similar paragraph from last year. I build the retrieval trace around that distinction: every candidate gets a document revision, effective interval, source type, and reason code before a language model sees it.
Start with the healthcare appointment constraint
Appointment content is a small, messy corpus: scheduling rules in PDFs, clinic-specific exceptions in a database, and messages that may contain protected health information. Split those sources before indexing. Parse each PDF into sections with page numbers, preserve table headers, and attach clinic_id, locale, service line, and effective_from metadata. A chunk without those fields is not citable evidence.
Use hard filters for facts that cannot be negotiated. A query for a pediatric clinic must not retrieve an adult-only rule because its wording is close. Filter by clinic, service, language, and validity window first; rank the survivors second. If a filter removes every candidate, return a clarification request instead of filling the gap with a plausible sentence.
Keep identifiers separate from patient text. Store a keyed digest for correlation and redact names, phone numbers, and appointment IDs in logs. The retrieval layer needs enough context to choose evidence, while observability needs enough structure to explain a decision without becoming a second medical record.
How do ranking signals make appointment answers grounded and citable?
Treat ranking as a transparent score with gates. One practical shape is:
from dataclasses import dataclass
@dataclass
class Candidate:
lexical: float
semantic: float
freshness: float
authority: float
citation_ready: bool
policy_match: bool
def rank_score(c: Candidate) -> float | None:
if not c.policy_match or not c.citation_ready:
return None
return (
0.35 * c.lexical
+ 0.30 * c.semantic
+ 0.20 * c.freshness
+ 0.15 * c.authority
)
The weights are a starting hypothesis, not a clinical standard. Lexical overlap catches exact service names and time windows; semantic similarity handles “reschedule” versus “change an appointment.” Freshness should decay after a rule expires, and authority can distinguish an approved clinic policy from an unreviewed draft. The gate matters more than the decimals.
No source, no answer.
Return evidence beside the generated text. A compact record might include source_id, page, section heading, revision, and the exact character span used. The model prompt should require those IDs and reject claims that cannot be mapped back to a span. This is where Retrieval-Augmented Generation (RAG) earns its keep: the generator is constrained by retrieved material rather than asked to recall scheduling policy from parameters (Lewis et al., 2020).
I keep a two-stage path for difficult queries. Retrieve 30 candidates with hybrid search, apply policy filters, then rerank the remaining 8 to 12 with the richer signals above. A second pass costs latency, so measure p95 and abandonment together. Your mileage may vary across languages and clinics; a labeled set of 500 appointment questions will tell you more than a leaderboard score.
Failure modes and tests that catch them
The nastiest defect is a correct-looking answer supported by the wrong clinic. Build one adversarial fixture with two nearly identical PDFs: Clinic North permits changes until 17:00 two business days before a visit, while Clinic South uses a different window; then add an expired North policy whose wording is closer to the question than the current policy. Ask the same rescheduling question with each clinic scope, once just before the boundary and once just after it. The test should inspect every stage: the expired document must fail the validity filter, the other clinic must fail the scope filter, the current passage must survive retrieval, and the emitted citation must identify the exact page and span. Also remove the clinic name from a body row while leaving it in the repeated table header. If parsing discarded that header, the chunk should be rejected during ingestion instead of entering the ranker with guessed metadata. This fixture catches three failures that a generic relevance benchmark can miss: stale evidence winning on similarity, tenant-like scope leaking across clinics, and a citation resolving to the right PDF but the wrong rule.
Test temporal boundaries explicitly. At 23:59 local time, an “48 hours before” rule can cross a daylight-saving transition. Normalize timestamps to the clinic’s timezone, retain the original offset for audit, and make the effective interval part of the filter expression. A one-hour discrepancy can create a missed appointment and a support escalation.
Observe retrieval separately from generation. Log candidate counts, filter rejection reasons, score components, citation coverage, and abstention rate. Never log raw patient questions by default. Alert on a sudden rise in NO_CITATION or EXPIRED_POLICY; those are actionable signals even when the final prose sounds fluent.
Choosing an architecture and rolling it out
A relational full-text index is often enough when policies are short and terminology is stable. Add vector retrieval when labeled queries show consistent lexical misses, and keep a metadata filter in front of both paths. Managed search services can reduce operations work; self-hosted search gives tighter control over residency and upgrade timing. The trade-off is real: choose the option your compliance team can inspect and your on-call team can repair at 03:00.
Start in shadow mode. For two weeks, run the new ranker beside the existing one, compare citation validity and filter violations, and sample disagreements with a clinical operations reviewer. Ship only after the abstention path, deletion policy, and rollback artifact are tested. Stick with the simpler lexical path when semantic retrieval adds latency without improving recall on your labeled appointment set.
The result is intentionally modest: a grounded answer, a page-level citation, or a clear request for clarification. That behavior is more useful than a confident paragraph that cannot show its source.
Top comments (0)