The four deliberate decisions that separate a working RAG demo from a reliable RAG system are chunking, retrieval, reranking, and evaluation. Tutorials that stop at "load PDFs → fixed-size chunks → embed → top-k into a prompt" produce fluent answers that fail silently on the exact queries that matter most - error codes, SKUs, function names, specific clauses. The gap is almost never a fancier model; it's these four choices made by default instead of by design.
This post is grounded in a real Confluence-backed system I built and run — hybrid dense + BM25 with RRF, cross-encoder reranking, structure-preserving ingestion, retrieval-informed routing, sentence-level groundedness verification, and a golden-set eval harness. The accompanying repository (github.com/chaudharyviv/confluence-rag-platform) implements exactly that architecture.
Decision 1: Chunking sets the hard ceiling
You cannot retrieve context that was split across a boundary, and you cannot generate a faithful answer from a chunk that lost half its meaning. Chunking is therefore not a preprocessing detail; it determines the maximum achievable quality of everything downstream.
Practical default that works across most corpora: recursive character/token splitting at natural boundaries (paragraphs → sentences → words) with a hierarchy of separators, targeting roughly 400–512 tokens and 10–20% overlap. Independent benchmarks including Chroma's, place this in the 85–90% recall band for typical content, which is why it's the sane starting point rather than a compromise (Firecrawl).
When to deviate - match the structure of the source, don't impose a uniform token window:
Page-level (or section-level) units win on paginated documents that contain tables and figures. NVIDIA's 2024 benchmarks showed this beating more elaborate methods precisely because mid-table or mid-figure splits destroy meaning (Firecrawl).
Semantic chunking - splitting where sentence-embedding similarity actually shifts topic can add up to ~9% recall, but it costs an embedding pass over every sentence at ingestion time. Use it when topic boundaries are genuinely fuzzy and the extra cost is justified (Firecrawl).
LLM-based chunking works but is prohibitively expensive at production scale; reserve it for small, high-value corpora.
Structure-preserving / domain-specific: for Confluence XHTML (or any hierarchical format), respect the actual document tree headings, tables, code blocks rather than flattening everything into a token stream. My own system does exactly this: heading breadcrumbs, Markdown tables, and fenced code blocks are preserved, then a token-aware sliding window runs inside each section so tables and code are never split mid-block. This is simply the page/section principle applied to the real source format.
Rule of thumb: start with recursive splitting unless you have a concrete, named reason to change mostly tables means page-level, fuzzy topics with high recall value means semantic. Don't default to LLM chunking.
One ingestion-scoping tip that matters before any of this: my own implementation currently pulls pages from a single space via Confluence's REST content API (spaceKey as a query param), which is fine for one knowledge base but doesn't scale to an org with many spaces. You'd either have to loop over every space key or accept everything in each space, tables of contents and archived junk included. The better approach once you're past one space is CQL (Confluence Query Language) with label-based filtering - a query like label = "rag-kb" AND type = page (via the /rest/api/content/search endpoint, or /rest/api/search on newer Cloud instances) finds every page tagged with a given label across every space in one call, regardless of which space it lives in. That turns "which pages belong in this knowledge base" into an explicit, auditable label you attach in Confluence, rather than an implicit "whatever's in this space" assumption baked into the ingestion code. It's the same principle as the chunking rule above, one level up: don't let your source system's default unit (a whole space) define your corpus boundary define the boundary on purpose, then query for it.
Decision 2: Retrieval - pure dense search has a predictable, silent failure mode
Dense (embedding) retrieval is excellent at semantic matching: "cancel subscription" finds "terminate your plan." It systematically degrades on lexically exact, rare tokens — error codes like ERR_SSL_VERSION_OR_CIPHER_MISMATCH, product SKUs, function names like torch.nn.functional.cross_entropy. Embedding pooling averages the rare token's signal into the surrounding context; BM25's inverted index does not. The failure is dangerous because the system still returns something, the LLM still produces a fluent answer, and the answer is simply grounded in the wrong document (TianPan.co).
The practical fix is hybrid retrieval: run BM25 (sparse) and dense search in parallel, then fuse. Reciprocal Rank Fusion (RRF) is the right default. Each document's fused score is the sum, across every ranked list it appears in, of 1 / (k + rank) with k = 60 by convention. RRF is score-scale agnostic (you never have to normalize cosine similarity against BM25's TF-IDF scores) and needs no labeled training data, which is why it ships as the default in Elasticsearch, Weaviate, and Qdrant (TianPan.co).
Caveat from the literature: RRF typically adds only ~1.3% NDCG over BM25 alone. A properly tuned weighted (convex) combination of the two scores can reach ~7.5%. Start with RRF because it requires zero tuning data; once you have a golden set (Decision 4) you can calibrate weights and measure the gain (TianPan.co). My own system uses exactly this pattern - Chroma for dense, rank_bm25 for sparse, hand-rolled RRF because self-hosted Chroma has no built-in hybrid search.
Decision 3: Reranking is retrieval's second opinion
Hybrid retrieval gives you a solid candidate pool, typically the top 20–50. It does not give you the best possible ordering of the 3–5 chunks that will actually enter the LLM's context window. Both bag-of-words and bi-encoder similarity are cheap approximations computed without ever looking at the query and document together.
A cross-encoder takes the query and each candidate as a single joint input and scores relevance directly. It's too expensive to run over the whole corpus, which is why retrieval happens first to narrow the field, but running it over 20–50 candidates is cheap and consistently improves the final ranking. My own system uses the small, self-hostable cross-encoder/ms-marco-MiniLM-L-6-v2 — no API call and no large model required, which matters for latency and cost when a weekend project turns into something you run daily.
Decision 4: Evaluation turns "seems to work" into a measurable fact
Most weekend RAG projects skip this entirely. Without it you cannot distinguish a real improvement from noise when you change chunk size, reranker, or prompt.
Measure three layers separately rather than one end-to-end "did it answer correctly" number that can't tell you where it broke (FutureAGI):
Retrieval - context precision/recall, MRR, hit-rate@k, NDCG for graded relevance.
Generation - faithfulness/groundedness (is the answer actually supported by the retrieved context, or did the model fall back on parametric knowledge?), answer relevance, context utilization.
End-to-end - answer correctness against a labeled reference, helpfulness, and refusal calibration (does it correctly decline when the corpus has no answer?).
Ragas remains the de facto reference implementation for the middle layer — faithfulness, answer relevance, context precision, context recall — and newer tools still measure themselves against it (FutureAGI).
The concrete advice that actually moves the needle:
Build a hand-authored golden set of 200–500 queries with ground-truth answers and relevance-labeled chunks before you trust any LLM-as-judge score. An uncalibrated judge grading against nothing is just a second opinion with no anchor (FutureAGI).
Gate every reindexing or pipeline change behind a regression run on that set.
Track rolling-mean scores per layer over time so drift appears before users notice it.
Once offline faithfulness is reliable, add a live, sentence-level groundedness check before any answer reaches the user: verify each claim traces to a specific retrieved passage and flag or block the answer if it doesn't. This turns a metric into a runtime gate.
My own system does precisely this: a golden set is run through the live retrieval-to-generation graph, every run is logged to a relational table for trend tracking, and a Claude-based sentence-level groundedness verifier runs on every real query, not just the ones in the golden set.
What "solid" actually means
None of the above is exotic technology. Recursive (or structure-aware) chunking, hybrid retrieval because dense-only has a known blind spot, a cheap cross-encoder pass because top-k ordering is only a rough draft, and a golden-set-plus-live-groundedness harness so you can tell whether the next change is an improvement these are a weekend's worth of deliberate decisions. The demo-to-system gap is almost never about needing a fancier model; it's about refusing to default on the four choices above.
A few more production-oriented details from my own implementation, worth adopting once the four core decisions above are solid:
Retrieval runs before any LLM router; routing strength is the primary signal, and a Claude router is consulted only on weak or empty matches. This avoids misrouting questions the knowledge base can actually answer.
Structured tool-use outputs for both routing and groundedness decisions - typed JSON rather than substring parsing or "ask for JSON and hope."
Host-agnostic configuration (
STORAGE_MODE,DATABASE_URL) so the same code runs on a laptop, Streamlit Community Cloud, or a VPS without forking anything.Incremental reindexing keyed on Confluence page versions, with an optional git-commit of index artifacts for hosts with ephemeral disks.
A full audit trail of every query and every eval run in a real relational schema (SQLite by default).
If you want to go further, the natural next steps are: implement or adapt the hybrid-plus-rerank-plus-groundedness pipeline for your own corpus, build the golden set first so every subsequent change is measurable, or go straight to the source and adapt the repo that already embodies these choices.
Sources: Firecrawl — Best Chunking Strategies for RAG, TianPan.co — Hybrid Search in Production, FutureAGI — What is RAG Evaluation? | Code: github.com/chaudharyviv/confluence-rag-platform
Top comments (0)