This was called to my attention by one of my readers which tell me to check for the router path.
I really appreciate my readers which take their time to read, drop their suggestions and correct me to do the right thing. Thank you all...
The Short Version of how Vicquant answer financial question
Vicquant has two pathways to answer a financial question:
A RAG path (rag_vault) that searches the user's uploaded documents (such as a 401(k) statement or tax summary) and cites the exact source.
A chat path (fast_chat) that answers from the model's general financial training data.
A small classification heuristic (the query router ) decides which path each incoming question takes.
My quality run scores four metrics (including faithfulness and context recall) on 25 golden questions. Every one of those questions goes straight into the RAG path. So those four numbers describe RAG only after a question has already arrived there. They say nothing about whether the router picked the right path.
That was a blind spot. When I finally audited the router, I found that 18 out of 20 document-grounded questions never reached the document vault at all.
Why This Matters
Imagine a user uploads their 401(k) statement and asks: "What formula does my employer use to match my contributions?"
If routed to RAG: The system inspects the statement, extracts the matching tier, and cites page 2 of the document.
If routed to Chat: The model answers from general knowledge. It might sound completely confident, stating a standard 50% match up to 6% even though the employer's actual formula is 100% up to 4%.
In the second case, my quality metrics didn't drop at all, because the question never entered the RAG pipeline. The failure was completely invisible to my evaluation harness.
The Two Errors Do Not Cost the Same
The one sent to RAG and the one sent to Chat cost two different things which are;
Because the failure costs are completely lopsided, the router should lean heavily toward retrieval. When in doubt, the question must go to RAG.
The Metric I Added
Alongside latency benchmarks, I now track:
The share of document-grounded inquiries that mistakenly route to the chat path.
I also track the reverse (general inquiries routed to RAG) so the engineering trade-off remains transparent.
How the Check Was Run
A Clean Benchmark Set: 40 questions (20 document-grounded inquiries about a sample 401(k) statement and 20 general personal finance questions).
Pre-Labelled: The expected path for every question was locked in before running the benchmark.
Automated End-to-End Execution: A test user uploaded the sample document, dispatched each query through the API, and recorded the path taken directly from the streaming response metadata.
Baseline Results (Pre-Fix)
The results before the fix was carried out;
Why Did 18 Document Questions Miss the Vault?
Personal Possessive Pronouns (8 misses): Questions like "What formula does my employer use?" or "Who is listed as my primary beneficiary?" used personal references (my employer, my balance) instead of generic keywords. The router mistook them for general conversational chat.
Missing Document Nouns (4 misses): Questions referencing "the statement" or "my plan summary" lacked exact token matches in the old keyword list.
Unrecognized Plan Terminology (6 misses): Operational benefits terms (graded vesting schedule, in-service rollover, plan loan origination fee) had no heuristic triggers.
The Fix: "Retrieval by Default"
Instead of "send to RAG only if an explicit keyword matches," the architecture flipped to retrieval by default:
A query is sent to fast_chat only if it matches an unequivocal, positive general financial signal (such as "How does compound interest work?" or "Explain the 50/30/20 budget").
Any ambiguous, borderline, or pronoun-anchored question automatically defaults to rag_vault.
If a user has not uploaded any documents, the sub-millisecond retrieval check returns zero matching chunks, and the pipeline gracefully streams a standard answer without failing or injecting empty context.
Before and After: Validation Across Two Sets
Set 1: Benchmark Set (40 Questions)
Set 2: Fresh Evaluation Set (30 Unseen Questions, marked not independent)
To test generalizability against new phrasing, a second set of 30 questions (15 document, 15 general) was run once without tuning:
(Note: In a 15-question sample, 0 misses statistically rules out a true miss rate above roughly ~20%. A larger ongoing test corpus will narrow this confidence interval).
The Cost of Fallback
Local Compute: Negligible. Local hybrid BM25 and vector search takes 13.8 ms mean compute time.
Context Overhead: A fallback question incurs ~400 additional tokens of retrieved context, costing approximately +$0.000015 USD on OpenRouter.
The Asymmetric Trade-Off: Sending 73% of borderline general inquiries through the RAG pipeline is an intentional design choice. Paying 14 ms of compute and a fraction of a cent is incomparably better than allowing a participant's retirement question to reach a chat model that cannot see their statement.
What I Learned
A quality metric only measures what flows through it.
My four RAG evaluation scores were mathematically sound, yet completely blind to our most dangerous failure mode because the failure occurred before the pipeline began.
If your AI system has a router, a classifier, or an intent gate standing in front of your RAG pipeline, evaluate the gate first, and weight its errors by what they actually cost the user.
In your spare time, kindly check out vicquant(It is in the beta testing phase ) so i am in need of all the help i can get and send your feedback. So I can Improve it and get it to work better.
The Link: https://budget-planner-advisor-rho.vercel.app/
Thank you!
Top comments (0)