I built a RAG assistant, then found out my architecture change made it worse, and I'm glad it happened
I recently built a hybrid RAG (retrieval-augmented generation) support assistant for a fictional B2B SaaS platform, "Helix," designed to answer customer-success questions grounded in a 100-document knowledge base of product docs, runbooks, and resolved support tickets. It cleared production-readiness evaluation thresholds comfortably: 0.939 faithfulness and 0.775 context precision on a 50-query RAGAs test set, against required floors of 0.70 and 0.60.
But the most useful thing that came out of the project wasn't the passing score. It was a hypothesis that turned out to be wrong, and what I did after finding that out.
The setup
The pipeline ingests a mixed-format 100-document corpus (Markdown product docs, PDF runbooks, HTML support tickets) into a Pinecone vector index, retrieves relevant context, and generates a grounded, citation-backed answer with an explicit confidence rating via an LCEL chain. Structured output is enforced with Pydantic (answer, sources, confidence), using gpt-4o-mini at temperature=0, because a support assistant answering the same question against the same context should give the same answer every time. Determinism mattered more than creative variation here.
Chunking wasn't one-size-fits-all. Three formats needed three strategies:
- Markdown docs were split by header first, so a chunk never crosses a topic boundary, with a recursive splitter as a fallback for long sections.
- PDF runbooks (no header structure to exploit) got a straight recursive character split.
- HTML tickets were kept as one whole chunk per ticket whenever possible, because a resolution often only shows up in the final turn of the conversation, and splitting a ticket risks separating the question from its answer.
- 5 scanned PDFs with no extractable text layer were detected and skipped gracefully rather than OCR'd, a conscious call I'll come back to.
Result: 95 of 100 documents ingested cleanly, 426 chunks produced.
The hypothesis I was confident about
Going in, I expected hybrid search (BM25 keyword matching combined with vector search via Reciprocal Rank Fusion) to outperform vector-only retrieval. The reasoning seemed solid: pure vector search can miss queries that hinge on an exact term (an error code, a specific field name), which get blurred by semantic embedding. BM25 should catch those, while vector search catches paraphrased matches BM25 would miss. Combining both should be strictly better.
I measured it properly, running a controlled before/after comparison across all 50 test queries, tracking Hit@5 (whether the expected source document appeared in the top 5 results) and Doc-Precision@5.
| Method | Hit@5 | Doc-Precision@5 |
|---|---|---|
| Vector-only | 100.0% (50/50) | 0.288 |
| Hybrid (RRF) | 94.0% (47/50) | 0.248 |
Hybrid search didn't improve retrieval on this corpus. Vector-only was slightly better on both measures. My hypothesis was wrong.
Why I think that happened
My read: this corpus is mostly natural, well-formed prose, product docs and runbook text, rather than dense with the exact-match triggers (IDs, codes, rare tokens) where BM25 usually earns its keep. RRF fusion appears to occasionally push a correct vector hit that ranked just outside the top candidates out of the final top-5, in favor of a BM25 match that shares surface keywords but isn't the right document. That's a plausible explanation for the 3 queries hybrid lost that vector alone caught.
The end-to-end generation-level eval (context precision via RAGAs, judged at chunk-relevance rather than strict doc-ID match) still passed comfortably at 0.775. The two precision metrics aren't directly comparable, but it suggests hybrid isn't badly hurting overall answer quality even where it underperforms on the stricter doc-hit measure.
I could have buried this, called hybrid a win because it's the more sophisticated-sounding architecture, and moved on. Instead I reported what the data actually said. If I ship this for real, that's the difference between a system that works and one that just looks like it does.
Three failure cases, three different root causes
Digging into the actual misses mattered more than the aggregate scores:
1. Pure retrieval miss. A query about an HTTP 401 error, the exact kind of exact-term query hybrid was supposed to help with, still failed. The expected source docs were never retrieved. The system correctly said "the provided context doesn't cover this" rather than guessing (faithfulness scored 1.00 on the non-answer), but relevance and precision both scored 0.00. Root cause: retrieval, not generation.
2. Retrieval succeeded, generation still failed. A webhook-configuration query correctly retrieved the right source document (context precision 1.00), but the model still said it couldn't answer. The specific chunk pulled from that file didn't happen to contain the actual instructions. Retrieving the right document doesn't guarantee retrieving the right chunk within it: a chunk-granularity gap, not a retrieval gap.
3. Incomplete synthesis on a hard, multi-document query. A billing-dispute question needed four sources spanning billing docs, a refund runbook, and a resolved ticket. The system cited only one and produced a plausible but under-grounded answer (faithfulness 0.65). Top-k=5 didn't surface all four relevant documents at once for a query that genuinely needed cross-corpus synthesis.
A fourth issue, not query-specific: citations occasionally leaked the literal string "doc_id: ..." instead of a clean path, because the model copied the context block's label formatting. Fixed by rewording the prompt rule and changing the context label format.
What I'd do with another week
Ranked by expected impact:
- Investigate the hybrid-search regression properly: tune the RRF constant and candidate pool size, only invoke BM25 for queries with likely exact-match tokens instead of always blending it, and try reranking (Cohere Rerank) as a direct comparison point.
- Fix the chunk-granularity gap: smaller chunks with more overlap for dense procedural docs, or a parent-document retrieval strategy where I'd retrieve small chunks for precision but pass the full parent section to generation for completeness.
- Improve hard-query synthesis: increase k for queries classified as needing cross-corpus synthesis, or retrieve separately per source category and merge, rather than one flat top-k across everything.
- Add OCR for the 5 skipped scanned PDFs, now that the rest of the pipeline is proven, using Tesseract with a quality check before trusting the extracted text.
Why this mattered more than the passing score
It would have been easy to write "I built a hybrid RAG system, here are the great numbers" and stop there. What I think actually demonstrates engineering judgment is the part where the more sophisticated architecture underperformed the simpler one, I noticed, I said so, and I could point to exactly which three queries broke and why. Faithfulness, context precision, and Hit@5 are useful summary numbers, but the failure analysis underneath them is where the real understanding of the system lives.
Repo, full eval results, and code: github.com/saftyy/hybrid-rag-enterprise-support
Top comments (2)
This is the RAG failure I see most often. The trace says the right document was nearby, then the actual answer falls between chunks and the model looks worse than it is. I usually add a query set that asserts the exact chunk, not just the source doc, before changing the architecture. Did you end up tuning chunk boundaries by file type or by failure case?
I like that you kept the losing variant in the write-up. The RRF result is a useful reminder that hybrid search is still a bet on the query distribution, not a free upgrade. For this corpus, I would probably test a routing rule for exact-token queries before paying the complexity tax everywhere.