DEV Community

Ali Baizhanov
Ali Baizhanov

Posted on • Originally published at mengram.io

Our Monitoring Said 62% of Retrievals Were Failing. The Bug Was Two Score Scales in One Column.

#rag #webdev #ai #postgres

The scare

Hybrid retrieval over personal memory — vector similarity + BM25, fused with Reciprocal Rank Fusion, optional cross-encoder rerank on some tiers. Every search logs top_score for quality monitoring. Analyzing 10,706 logged searches, I applied the obvious threshold — top_score < 0.3 = weak retrieval. Result: 62% "failures," a dozen users at "100% failure with avg score 0.017," and a terrifying month-over-month "degradation." One of the "100% failed" users was a paying customer with a thousand searches. I was halfway into incident mode.

The tell

A search for an exact entity name — a guaranteed hit — logged top_score 0.0426. And the "failing" users all averaged 0.016–0.021. Then it clicked: RRF scores are 1/(k + rank) with the standard k=60. Top rank = 1/60 ≈ 0.0167. My "catastrophic" users weren't failing — their top result was rank-1 almost every time. An average of 0.017 is what perfect RRF retrieval looks like.

What actually happened

Requests that go through the reranker log cosine-style scores (0–1 scale, 0.3+ = good). Requests on the raw RRF path log fusion scores (0.016–0.05 scale, where 0.017 = excellent). Both landed in the same top_score column with no scale tag. Every aggregate over that column — means, z-scores, my failure thresholds, even the health-monitoring cron — was averaging apples with orbital velocities. The "month-over-month degradation" was just the RRF-path share growing as more traffic moved to hybrid.

What survived scale-correction: true failure (zero results) was 9–13%, driven mostly by two accounts whose agents were querying literally empty stores — a real problem, but a completely different one than "retrieval is broken."

Lessons that generalize

  1. A fused ranking score is not a similarity. RRF outputs rank information, not confidence. The moment you fuse, the score's absolute value stops meaning what your dashboards think it means.
  2. Never store scores from different scoring regimes in one unlabeled column. Log a score_kind (or a scale-aware quality label computed at write time) — analysis-time guessing is how you get 3am false incidents.
  3. The only scale-free failure signal is emptiness. Zero results means the same thing on every path. When in doubt, count zeros, not thresholds.
  4. Validate your alarm against a known-good query before believing it. One exact-match search that "scored 0.04" saved me from paging myself.

The k=60 default everyone inherits comes from Cormack, Clarke & Buettcher (2009), "Reciprocal Rank Fusion outperforms Condorcet and individual rank learning methods." The trap applies to any RAG stack mixing rerankers with fusion scoring — grep your score column and look for a bimodal cluster around 1/60.

I build Mengram, a memory layer for AI agents. The fix — a scale-aware quality label written alongside every search — is in the public commit history. Questions welcome.

Top comments (0)