A client called us last month with a simple complaint: "Our support agent confidently quotes the wrong refund policy." The model was fine. The prom...
For further actions, you may consider blocking this person and/or reporting abuse
Good list. The stale-document case is the one I see teams underestimate most, because the retrieval stack looks healthy while the corpus is quietly wrong.
One detail I would add: I like separating retrieval confidence from source validity. A chunk can be highly relevant and still invalid because its effective date, product version, or permission scope is wrong. So I usually treat metadata filters as a hard gate before reranking, not just extra text for the model to reason over.
For the "I don't know" path, thresholding also works better when it is evaluated per intent. A low-confidence pricing answer and a low-confidence FAQ answer should not have the same handoff threshold.
The metadata-as-hard-gate point is one we end up making to clients directly - a confidently wrong answer on stale pricing erodes trust far more than a system that admits it isn't sure. Per-intent thresholds are underrated for the same reason: the cost of being wrong isn't uniform across question types, so the handoff bar shouldn't be either. Have you found many teams willing to tune thresholds per intent, or do most still ship one global number and hope?
Most teams still ship one global number, honestly. Per-intent thresholds get resisted less because they're technically hard and more because nobody owns the decision - it needs someone who understands both the cost of a wrong answer per intent and the retrieval scores, and that's usually split across a PM and an ML engineer who never sit in the same meeting. The teams that do tune per-intent almost always got there after a specific incident (a bad pricing answer, a compliance-adjacent question answered wrong), not proactively. Same pattern as most RAG maturity, honestly - measurement discipline shows up after the first embarrassing failure, not before.
You've named the real blocker, and it's organizational more than technical. The PM knows the cost of a wrong answer, the ML engineer knows the retrieval scores, but nobody owns the number, so it defaults to one global threshold no one has to defend. The incident that finally forces the issue is usually what frees up the budget too, because now there's a name attached to the failure. When we get in early, half the job is just getting someone to own that call before the bad answer instead of after. Have you seen making the PM the owner actually stick, or does it tend to drift back to the ML side?
Drift is the more common outcome in my experience. Making it stick usually needs the ownership tied to a metric someone is actually evaluated on, not just a title on a doc. If the PM "owns" the threshold but nothing measures whether their calls were right, it slides back to whoever's on call during the next incident. The setups where it held were the ones where per-intent retrieval quality showed up in a weekly review, not just in postmortems after something broke.
That weekly-review point is the part most teams skip. We ended up putting per-intent recall on the same dashboard the on-call engineer already stares at, so a bad retrieval class surfaces before anyone files a ticket about a wrong answer. Tying it to a number in someone's review is what stopped the ping-pong for us too — postmortems only ever caught the incidents loud enough to notice. Do you sample the weekly eval set, or run it across the full intent taxonomy?