DEV Community

SladeBarrett9642
SladeBarrett9642

Posted on

How to Bound Stale Evidence in Insurance Claims Retrieval (A Practical Record)

Short answer: use separate freshness windows for policy wording, claim events, and reference guidance, then make the window part of every retrieval filter and answer citation. For an insurance claims intake bot, this keeps a newly submitted loss from being explained with yesterday's status while avoiding a wasteful full re-index on every upload.

I would record the decision as an architecture decision record (ADR): freshness is a correctness constraint, not a tuning knob. The bot may retrieve an older document only when its source explicitly permits that age and the answer labels the document's effective time.

The decision record: what must stay true

The system has four invariants. Every chunk carries effective_at, observed_at, expires_at, claim_id (when applicable), and a source revision. Retrieval rejects expired chunks before ranking. A claim event can narrow a search, but it cannot silently widen a policy window. Finally, the generated answer cites the revision and timestamp that actually won retrieval.

These boundaries matter because claims intake mixes clocks. A policy endorsement can be effective on Monday, a notice of loss can arrive on Tuesday, and an adjuster's note can be corrected on Wednesday. A single global “last 24 hours” filter makes at least one of those records wrong.

Option Freshness rule Where it fits Failure boundary
Global time-to-live One age limit for every chunk Small prototypes with one source cadence Slow policy documents disappear too soon, while fast claim events stay too long
Per-source windows Window selected from document type and authority Mixed policy, event, and guidance corpora Requires source classification and explicit defaults
Event-triggered invalidation Re-index or invalidate after a claim change High-volume claims with reliable change events Missed events leave stale evidence unless a periodic sweep backs them up
Version pinning Query against a declared policy revision Audits, disputes, and replayable decisions A pinned revision can be unavailable if retention is too short

The recommended baseline is per-source windows plus version pinning for adjudication-sensitive answers. Event triggers are an accelerator, not the only guard.

Freshness is a clock problem.

Measure it.

How should insurance claims intake retrieval enforce freshness windows?

Start with a small policy table owned by the ingestion service. Keep durations in configuration, but store the selected window with each indexed record so an audit does not depend on today's configuration. In this example, policy text is allowed to age for 30 days, claim events for 15 minutes, and general guidance for 7 days. Those are example defaults for the design, not universal insurance rules; compliance and legal teams must approve the actual values.

from dataclasses import dataclass
from datetime import datetime, timedelta, timezone

@dataclass(frozen=True)
class FreshnessPolicy:
    window: timedelta
    require_claim_id: bool

POLICIES = {
    "policy": FreshnessPolicy(timedelta(days=30), False),
    "claim_event": FreshnessPolicy(timedelta(minutes=15), True),
    "guidance": FreshnessPolicy(timedelta(days=7), False),
}

def freshness_cutoff(kind: str, now: datetime) -> datetime:
    policy = POLICIES[kind]
    return now.astimezone(timezone.utc) - policy.window

def build_filter(kind: str, claim_id: str | None, now: datetime) -> dict:
    policy = POLICIES[kind]
    if policy.require_claim_id and not claim_id:
        raise ValueError("claim_event retrieval requires claim_id")
    result = {
        "kind": kind,
        "observed_at_gte": freshness_cutoff(kind, now).isoformat(),
        "expires_at_gt": now.astimezone(timezone.utc).isoformat(),
    }
    if claim_id:
        result["claim_id"] = claim_id
    return result
Enter fullscreen mode Exit fullscreen mode

The important detail is the two-sided check. observed_at prevents an old copy from looking current after a re-embedding job; expires_at protects against a source that has declared an earlier cutoff. A vector score never overrides either condition.

I once debugged a retrieval trace showing a perfect semantic match that was 42 minutes old. The claim feed's contract was 15 minutes. The ranker was innocent; the missing metadata filter was the bug in my design. The fix was a 400 response for an invalid filter during development and a metric for rejected stale chunks in production.

That incident changed the shape of my traces. I stopped logging only the top score and started recording the query time, selected policy kind, calculated cutoff, source revision, and rejection reason for every candidate that crossed the filter. During a replay, this makes it possible to answer a very specific question: was the chunk unavailable because it was stale, because its source clock was malformed, or because the claim scope did not match? Those are different incidents with different owners. A single “no answer” counter hides all three and sends an engineer looking in the vector store when the ingestion contract is the real boundary.

What should the indexing pipeline preserve for an auditable answer?

Normalize source records before chunking. Capture the source revision, effective date, observed date, and expiry decision in the same event that creates embeddings. Use deterministic chunk IDs such as claim-123:rev-7:chunk-04; this makes replays idempotent and lets a new revision invalidate its predecessor without deleting history needed for an investigation.

The ingestion path should be boring: parse, classify, validate timestamps, chunk, embed, upsert, and emit a receipt. If classification is uncertain, route the record to a review queue instead of assigning the shortest window. A false “claim event” label can hide a valid policy clause; a false “policy” label can preserve stale loss details.

Keep raw documents outside the vector index and retain a cryptographic digest in the receipt. Embeddings are derived data. They can be regenerated; the source revision and decision about eligibility must remain reproducible.

How do retrieval tests expose stale claim evidence before release?

Build fixtures around boundary times, not average documents: exactly at the cutoff, one second before it, a future effective_at, a superseded revision, and a claim event with no claim ID. Assert that stale chunks are absent before ranking and that the answer payload includes the winning revision. Test the clock as an injected dependency so a CI run can replay the same instant.

For observability, emit retrieval_stale_rejected, retrieval_revision_miss, and answer_citation_missing counters. Sample the rejected IDs, but redact claimant names and policy numbers. A dashboard should show rejection rate by source kind and window; a sudden drop to zero can mean the feed stopped sending timestamps, not that the corpus became fresher.

Where does this design stop being suitable?

The catch is operational scope. Per-source windows are not suitable when a regulator requires a provider-enforced immutable archive, a formally versioned rules engine, or a legal hold that outlives normal index retention. In those cases, keep the retrieval layer as a locator and hand the authoritative decision to the governed system. Stick with a simpler global TTL when the corpus has one trusted update cadence and no claim-level clock; the extra metadata and review queue are overhead you will not recover.

Freshness also cannot repair bad source authority. If two carriers publish conflicting endorsements, retrieval can expose both revisions and their timestamps, but a claims policy must define which authority wins. Your mileage may vary across jurisdictions; I am not sure a single default window can survive every filing schedule, so document who can change each value and require a replay before changing it.

The practical rule is compact: filter by an approved window before ranking, preserve the revision that justified the answer, and rehearse the boundary cases with a frozen clock. That gives an intake bot a defensible explanation without pretending that semantic similarity is proof of currency.

References

Top comments (0)