If you've built a retrieval-augmented generation (RAG) system for general documents and are now pointing it at insurance policy wordings, endorsements, and schedules — a warning: the naive pipeline that demos beautifully on a blog corpus falls apart on policy documents, and it fails in ways that look like success.
Here's what actually matters.
1. Chunking by token count destroys policy semantics
The default RecursiveCharacterTextSplitter(chunk_size=1000) will happily split a coverage clause from its exclusions, or a definition from the term it defines. In a policy, meaning is structural: a limit means nothing without the peril it applies to, and an exclusion means nothing without the grant of cover it carves out.
Chunk on document structure, not character count — heading-aware splitting that keeps a clause, its sub-clauses, and its cross-references together. If your splitter can't see that "subject to the Exclusions in Section 4" points somewhere, your retriever will confidently return half the answer.
2. Retrieval relevance ≠ answer correctness
The dangerous failure mode: you ask "is flood covered?" The retriever pulls the flood grant of cover (high similarity to "flood") and misses the flood exclusion three pages later (lower similarity, different wording). The LLM answers "yes." It is wrong, and it is confident, and nothing in your eval caught it because the retrieved chunk was relevant.
Two mitigations that work:
- Hybrid search (dense vectors + BM25, fused with RRF). Exclusions often use precise legal terms that lexical search nails and embeddings blur.
- Force the model to retrieve exclusions as a separate step. Don't trust one similarity pass to surface both the grant and the carve-out.
3. Tables and schedules are where the real data lives
Limits, deductibles, sub-limits, endorsement effective dates — the facts a user actually wants — live in tables, and most extraction pipelines flatten tables into word soup. A schedule that reads Buildings £250,000 / Contents £50,000 becomes Buildings 250000 Contents 50000 and the model cheerfully attaches the wrong number to the wrong peril.
Preserve table structure end to end. Extract to structured rows before embedding, and store the structured version alongside the prose.
4. Cite or don't ship
In insurance, an answer without a source is a liability, not a feature. Every response has to point back to the exact clause and document version it came from — not because it's nice UX, but because a wrong coverage answer with no audit trail is the kind of thing that ends up in front of a regulator. Wire source labels and scores through retrieval so every generated sentence is traceable to a chunk.
The unglamorous conclusion
RAG over policy documents is 20% prompt engineering and 80% document engineering. The teams that win aren't the ones with the cleverest reranker — they're the ones who did the boring work of structure-aware chunking, hybrid retrieval, table preservation, and citations.
I wrote up the full breakdown, including the exclusion-retrieval pattern and why relevance scores lie, here:
RAG Over Policy Documents: What Works and What Doesn't →
If you're building this on top of a real book of business, the data-foundation work matters more than the model choice — that's the thread across everything IntelliBooks writes about insurance AI.
What's broken in your policy-document pipeline? Curious whether others hit the exclusion-retrieval trap.
Top comments (0)