Lesson 3 is the kind of thing that costs a day and shows up in no tutorial.
One thing I would pressure-test before trusting the semantic cache, complementary to Ivan's scoping point rather than a repeat of it: cosine 0.92 between an incoming query and a cached query measures topical similarity, not answer equivalence. Two questions that differ by a single entity, a date, or a negation usually sit very close in embedding space, because one token moves the vector far less than it moves the correct answer. In clinical Spanish that is not a corner case, it is the normal shape of a question: "pacientes con fiebre" versus "pacientes sin fiebre", or the same query with the year changed.
Cheap way to falsify it, maybe twenty minutes. Take twenty real queries from your logs, write one near twin of each that differs only by an entity or a negation, embed both pairs with nomic-embed-text, and look at the cosine distribution of the twins. If any twin pair clears your threshold, the cache will serve a confidently wrong answer, and it fails silently, since a cache hit never reaches the LLM to be checked. Whatever the top of that distribution turns out to be is your real floor for the threshold, not 0.92.
That probe also fits the embedding evaluation on Spanish clinical text you say is next. Retrieval quality and negation sensitivity are separate axes, and an embedder can score well on the first while being useless on the second.
On lesson 5 my experience matches yours: I benchmarked a set of retrieval-side changes on a different corpus and several standard tricks came back null, while prompt wording moved the number. Caveat firmly on my own result, different corpus and a different question distribution, so treat it as a second data point rather than a property of RAG in general.
One question, since you are CPU-only: of the 60 to 120 seconds on 4 vCPU, how much is prefill over the retrieved context versus generation? That split usually decides whether the next win comes from shrinking top-k or from a smaller model.
Systems engineer. 19 years in Costa Rica's national health system. I write about migrating legacy .NET at scale, T-SQL under real hospital load, and running LLMs on-premise for regulated data.
Location
San José, Costa Rica
Education
MSc in Applied AI, TECH Global University. BSc in Systems Engineering, U. Catolica de Costa Rica.
Work
Systems Analyst at CCSS (Costa Rica's national health service) + Lecturer at UNED
Giulio, I ran the twenty-pair probe you suggested. The result is worse than I expected, and I want to publish it here before I write it up properly.
Setup: nomic-embed-text, twenty pairs on Spanish administrative-domain queries, five per category — negation, temporal, entity, and paraphrase-control. Cosine similarity, same math the cache uses.
The numbers, in descending order of concern:
Negation pairs (5/5): cosine range 0.9702–0.9984, mean 0.9837. All five would cache-hit at 0.92. The worst case, "con goce salarial" versus "sin goce salarial", scored 0.9984. Practically identical to the embedder.
Temporal pairs (5/5): cosine range 0.9054–0.9646, mean 0.9372. Three of five would cache-hit at 0.92.
Entity pairs (5/5): cosine range 0.7498–0.9210, mean 0.8641. One of five would cache-hit at 0.92.
Paraphrase controls (5/5): cosine range 0.7470–0.9060, mean 0.8067. Zero of five would cache-hit at 0.92.
The distributions overlap catastrophically. The highest adverse similarity (0.9984) is well above the lowest paraphrase similarity (0.7470). There is no cosine threshold that separates the two categories with nomic-embed-text on Spanish text of this structure. Any threshold high enough to reject the adverse pairs also rejects every genuine paraphrase.
You were right about the mechanism, and the mechanism turned out to be much more severe than the article implied. What I described as "aggressive caching" is, in this configuration, a source of silent wrong answers for exactly the class of question users are most likely to ask.
Two things I want to say before I close this comment:
First, the two-layer cache design I mentioned earlier doesn't fix this on its own — separating retrieval cache from answer cache reduces the leak surface but doesn't help with the underlying semantic collapse. Any answer cache keyed on query embedding has the same problem in this domain.
Second, I'm not going to commit to a specific fix in this reply. I want to sit with the evidence for a day before I decide whether the right move is to disable the cache by default, gate it behind explicit configuration with strong warnings, add symbolic checks on top, or something else. I'll come back to this thread when the PR is up.
The next article will be the full write-up with the twenty pairs, the code, and the distribution. It's a much stronger piece of evidence than anything I could have written from principles.
Thanks for pushing on this. This is exactly the kind of comment that separates readers who engage from readers who ship.
That is a stronger result than I expected too, and the negation row is the one I would lead with. 0.9984 for "con goce salarial" against "sin goce salarial" is not a near miss, it is the embedder telling you the negation particle carried almost no weight in the pooled vector.
One caveat on my own suggestion: with five pairs per category, the counts at 0.92 ("three of five", "one of five") are fragile and readers will quote them as rates. I would report the distributions and the best separation achievable across all thresholds instead. The claim that survives scrutiny is the one you already made: no threshold separates the two populations.
The scope I would state explicitly is nomic-embed-text on Spanish administrative text. Whether the collapse is the embedder, the language, or the domain is cheap to settle: the same twenty pairs through bge-m3 or multilingual-e5-large, plus the negation five in English through nomic. If negation collapses in English too, this stops being about Spanish and becomes a statement about mean-pooled embeddings, which is a considerably bigger piece.
Agreed on the two layers. I raised that as a leak-surface argument, not a fix, and any key derived from the query embedding inherits the same failure.
Waiting a day seems right. My instinct is that the decisive question is not which of your four options you pick, but whether the cache can be made to fail loudly rather than silently, since a wrong cached answer with no signal is far worse than a miss.
Systems engineer. 19 years in Costa Rica's national health system. I write about migrating legacy .NET at scale, T-SQL under real hospital load, and running LLMs on-premise for regulated data.
Location
San José, Costa Rica
Education
MSc in Applied AI, TECH Global University. BSc in Systems Engineering, U. Catolica de Costa Rica.
Work
Systems Analyst at CCSS (Costa Rica's national health service) + Lecturer at UNED
Giulio I ran both models you suggested. Before the results, two corrections to my own work, because they change how the first report should be read.
First: two of my five negation pairs were mislabelled. "¿Es obligatorio X?" against "¿No es obligatorio X?" is a confirmatory negative interrogative in Spanish it doesn't invert the answer, and a correct system responds the same to both. I had them as pairs the cache must reject. They should have been accepts. The published minimum for the negation row, 0.9702, was one of those. The report now carries an erratum: github.com/psychohub/rag-onpremise...
Second, and this is the one I'd want you to look at: taking your caveat seriously exposed a confound in my design, not just in my reporting. My negation pairs differed by one token. My paraphrase controls differed by most of their tokens. Any separation I reported could have been surface form rather than semantics. So I added paraphrases with matched lexical overlap one-token synonym swaps and pre-specified that contrast.
The control was not decorative. bge-m3 gives AUC 0.3556 against low-overlap paraphrases and 0.9333 against matched ones, on the same adversarial pairs. The uncontrolled version was measuring lexical distance. My original headline rested on it.
Matched contrast, n=5 against n=9, exact enumeration over all 2002 label assignments:
nomic on Spanish AUC 0.1333, margin −0.1017. The error-minimising threshold accepts nothing: the optimal cache configuration is no cache.
nomic on English AUC 0.4444, p=0.797. Not significant. The honest reading is that the score carries no usable information here, not that English does better. Negation similarity averages 0.9520 across nine pairs, comparable to Spanish. I've retracted the language-based explanation from the original report. Nine pairs won't establish a claim about mean pooling, but the Spanish attribution doesn't survive.
bge-m3 on Spanish AUC 0.9333, p=0.0070. It orders correctly, and it cleanly resolves temporal and entity distinctions that nomic could not. But the margin is −0.0086, and removing one pair flips its sign. Not "bge-m3 works" undetermined, and I can say how far from determined.
On the reranker: ms-marco-MiniLM-L-6-v2, scored in both directions since a cache needs a symmetric relation. Negative. In English the score is significantly inverted (AUC 0.0667, p=0.0070, margin −6.27 logits); in Spanish the primary contrast doesn't reach significance. The reason is structural and reads straight off the distributions in English the two populations that must be accepted sit one below and one above the population that must be rejected (matched paraphrases 4.58, negations 7.69, confirmatory 9.31). There is no cut point, at any threshold.
That ordering tracks lexical overlap almost exactly: confirmatory inserts a token, negation swaps a particle, matched paraphrase swaps a content word. Which is to say the cross-encoder fails the same way the bi-encoder does, not a different way. A relevance objective is well approximated by surface overlap, and negation preserves topical relevance nearly intact. That's evidence against this model for this task, not against cross-encoders — but it does make me expect BGE-reranker-v2-m3 to be more inverted rather than less, since it's trained on the same objective. That's a prediction, not a result, and your 240k-pair benchmark is better positioned to test it than I am.
Latency, since you asked whether it fits the budget: 32.6 ms per pair on CPU covering both directions, no batching. Cost wasn't the obstacle. Signal was.
On failing loudly rather than silently I agree that's the decisive question and I still don't have a clean design. It's the piece I most want to think about properly rather than ship.
Scripts and raw similarities are in docs/experiments/. The JSON holds the scores, so the reanalysis runs without Ollama.
Appreciate the corrections and the erratum. One thing that could save you a step and give you a comparison: I built a RAG called RE-call (github.com/GiulioDER/RE-call) that self-hosts, so your corpus stays inside your own data protection constraints, and its scope matches what you're already testing here.
It can be tailored to your own requirements and hardware, backed by Postgres with pgvector. Point it at your corpus, and it gives you a same-conditions comparison without having to build a new harness for it. I'd be glad to walk through the setup with you if that's useful.
On the reranker finding, the surface-overlap explanation is convincing given the population numbers you laid out, and I think your prediction about BGE-reranker-v2-m3 is testable rather than speculative, so I'd be curious what you find if you run it.
Systems engineer. 19 years in Costa Rica's national health system. I write about migrating legacy .NET at scale, T-SQL under real hospital load, and running LLMs on-premise for regulated data.
Location
San José, Costa Rica
Education
MSc in Applied AI, TECH Global University. BSc in Systems Engineering, U. Catolica de Costa Rica.
Work
Systems Analyst at CCSS (Costa Rica's national health service) + Lecturer at UNED
Giulio — I ran BGE-reranker-v2-m3. My prediction failed.
I'd said it should be more inverted than MiniLM on the primary contrast, since both train on the same relevance objective. It isn't. AUC 0.7667 in Spanish and 0.8222 in English, against MiniLM's 0.3333 and 0.0667. Wrong side of 0.50 in both languages. And MiniLM's English inversion — the single significant result in that whole run, p=0.0070 — does not replicate with another cross-encoder.
What I can't claim is that BGE works. Neither Spanish result reaches significance (p=0.1174 and p=0.3636) and English lands at p=0.0599. The defensible statement is that the prediction failed, not that the model succeeded. Same brake I had to apply to bge-m3 earlier.
The operating point doesn't move regardless. Best threshold in Spanish gives 3 errors out of 14, English 2 out of 14. Ordering well and cutting well are different things, and a cache needs the second.
One thing I'd point you at, because it's more interesting than my failed prediction. BGE scores AUC 1.0000 on temporal and entity contrasts and still doesn't resolve polarity — which is the same failure profile I measured for bge-m3 as a bi-encoder. Two models from the same family, two different architectures, the same shape of failure. That's an observation, not a conclusion: two models won't distinguish "property of the training objective" from "property of this family," and I'd need models from other families to say which. Writeup here: github.com/psychohub/rag-onpremise...
Two methodological notes in case they save you time. The two models resolve different activations — ms-marco-MiniLM to Identity, so raw logits; BGE to Sigmoid, so probabilities. AUC and permutation p-values compare across them because both depend only on ordering. Margins don't. I nearly published probabilities labelled as logits. And I checked that the perfect AUCs weren't saturation artifacts: maximum observed is 0.99998, one exact tie in 45 comparisons.
Latency, since you flagged the cost question: 571.1 ms per pair against MiniLM's 36.4, roughly 15.7x on CPU. 568M parameters against 22M. That's operating cost, not evidence — a more expensive model isn't more or less inverted for being expensive.
On RE-call — thanks, and I looked. I don't think it fits this particular experiment, and the reason might be worth saying: what I'm measuring isn't a RAG system. There's no harness, no vector store, no retrieval. It's Ollama, a similarity function, and 47 hand-built pairs. Running it through a full pipeline would reintroduce every variable I spent two weeks isolating.
Where I'd actually want your input is upstream of that. If RE-call caches at all, how do you handle the case where a cached question and the incoming one differ only in polarity? That's the question I still don't have a clean answer to, and it's closer to your "fail loudly rather than silently" framing than any of the model comparisons have been.
Edited: the latency paragraph originally said 32.6 ms and 17.5x. Auditing the repo turned up that 32.6 came from a run whose output no longer exists — the reproduction command I'd documented redirected with >, so each run overwrote the previous file. The figures above are the ones recalculable from the tracked JSON. Fixed in c3f0928, along with the command that was destroying the evidence.
Systems engineer. 19 years in Costa Rica's national health system. I write about migrating legacy .NET at scale, T-SQL under real hospital load, and running LLMs on-premise for regulated data.
Location
San José, Costa Rica
Education
MSc in Applied AI, TECH Global University. BSc in Systems Engineering, U. Catolica de Costa Rica.
Work
Systems Analyst at CCSS (Costa Rica's national health service) + Lecturer at UNED
SemanticCacheEnabled is now a flag with default false, and the README says explicitly why: your hypothesis was correct, and the twenty-pair probe made the case concrete enough that leaving the cache on by default was untenable.
The full experiment report, including the twenty pairs, the raw distributions by category, the reproducibility scripts, and the decision reasoning, is at docs/experiments/threshold-safety.md. Your comment is credited in section 8.
Next thing on the roadmap is the proper embedding evaluation on Spanish clinical text that you flagged as the necessary follow-up — building a real eval set instead of the twenty synthetic pairs. That's the next article.
Hubert, good outcome, you did the right step of running the probe, rather than just taking my word for the mechanism.
One thing I would add to the roadmap now that the flag is off by default: two free models worth putting through the same twenty pairs. For the embedder, BAAI/bge-m3 is the one I would try first. It is multilingual, open weight, and supports dense plus sparse plus late interaction scoring. So it gives you a second axis to check whether the negation collapse, is a property of dense mean pooling specifically, or of the embedder generally.
For a reranker, cross-encoder/ms-marco-MiniLM-L-6-v2 is the one I would reach, given your CPU constraint. It is small enough to stay inside the latency budget you already measured, and it scores a query against a candidate directly, which is closer to the fail loudly, not silently property you said you wanted.
I will say plainly where it fell short for me: on my own reranking benchmark, about 240000 query candidate pairs, this exact model improved recall at 100 but the gain did not reliably convert into a top 5 ranking improvement, not Holm significant in my run. A heavier cross-encoder, BGE-reranker-v2-m3, did convert, but at roughly ten times the inference cost of MiniLM on the same hardware. On CPU that is not a free upgrade, so it is worth timing both on your own queries before picking one, especially if a reranker ends up as your confirmation step ahead of serving a cache hit, rather than only as a retrieval quality tool.
Neither of these fixes the negation problem by itself, but they are free, on premise, and cheap enough to add as two more columns on the twenty pair table you already built.
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
Lesson 3 is the kind of thing that costs a day and shows up in no tutorial.
One thing I would pressure-test before trusting the semantic cache, complementary to Ivan's scoping point rather than a repeat of it: cosine 0.92 between an incoming query and a cached query measures topical similarity, not answer equivalence. Two questions that differ by a single entity, a date, or a negation usually sit very close in embedding space, because one token moves the vector far less than it moves the correct answer. In clinical Spanish that is not a corner case, it is the normal shape of a question: "pacientes con fiebre" versus "pacientes sin fiebre", or the same query with the year changed.
Cheap way to falsify it, maybe twenty minutes. Take twenty real queries from your logs, write one near twin of each that differs only by an entity or a negation, embed both pairs with nomic-embed-text, and look at the cosine distribution of the twins. If any twin pair clears your threshold, the cache will serve a confidently wrong answer, and it fails silently, since a cache hit never reaches the LLM to be checked. Whatever the top of that distribution turns out to be is your real floor for the threshold, not 0.92.
That probe also fits the embedding evaluation on Spanish clinical text you say is next. Retrieval quality and negation sensitivity are separate axes, and an embedder can score well on the first while being useless on the second.
On lesson 5 my experience matches yours: I benchmarked a set of retrieval-side changes on a different corpus and several standard tricks came back null, while prompt wording moved the number. Caveat firmly on my own result, different corpus and a different question distribution, so treat it as a second data point rather than a property of RAG in general.
One question, since you are CPU-only: of the 60 to 120 seconds on 4 vCPU, how much is prefill over the retrieved context versus generation? That split usually decides whether the next win comes from shrinking top-k or from a smaller model.
Giulio, I ran the twenty-pair probe you suggested. The result is worse than I expected, and I want to publish it here before I write it up properly.
Setup: nomic-embed-text, twenty pairs on Spanish administrative-domain queries, five per category — negation, temporal, entity, and paraphrase-control. Cosine similarity, same math the cache uses.
The numbers, in descending order of concern:
Negation pairs (5/5): cosine range 0.9702–0.9984, mean 0.9837. All five would cache-hit at 0.92. The worst case, "con goce salarial" versus "sin goce salarial", scored 0.9984. Practically identical to the embedder.
Temporal pairs (5/5): cosine range 0.9054–0.9646, mean 0.9372. Three of five would cache-hit at 0.92.
Entity pairs (5/5): cosine range 0.7498–0.9210, mean 0.8641. One of five would cache-hit at 0.92.
Paraphrase controls (5/5): cosine range 0.7470–0.9060, mean 0.8067. Zero of five would cache-hit at 0.92.
The distributions overlap catastrophically. The highest adverse similarity (0.9984) is well above the lowest paraphrase similarity (0.7470). There is no cosine threshold that separates the two categories with nomic-embed-text on Spanish text of this structure. Any threshold high enough to reject the adverse pairs also rejects every genuine paraphrase.
You were right about the mechanism, and the mechanism turned out to be much more severe than the article implied. What I described as "aggressive caching" is, in this configuration, a source of silent wrong answers for exactly the class of question users are most likely to ask.
Two things I want to say before I close this comment:
First, the two-layer cache design I mentioned earlier doesn't fix this on its own — separating retrieval cache from answer cache reduces the leak surface but doesn't help with the underlying semantic collapse. Any answer cache keyed on query embedding has the same problem in this domain.
Second, I'm not going to commit to a specific fix in this reply. I want to sit with the evidence for a day before I decide whether the right move is to disable the cache by default, gate it behind explicit configuration with strong warnings, add symbolic checks on top, or something else. I'll come back to this thread when the PR is up.
The next article will be the full write-up with the twenty pairs, the code, and the distribution. It's a much stronger piece of evidence than anything I could have written from principles.
Thanks for pushing on this. This is exactly the kind of comment that separates readers who engage from readers who ship.
That is a stronger result than I expected too, and the negation row is the one I would lead with. 0.9984 for "con goce salarial" against "sin goce salarial" is not a near miss, it is the embedder telling you the negation particle carried almost no weight in the pooled vector.
One caveat on my own suggestion: with five pairs per category, the counts at 0.92 ("three of five", "one of five") are fragile and readers will quote them as rates. I would report the distributions and the best separation achievable across all thresholds instead. The claim that survives scrutiny is the one you already made: no threshold separates the two populations.
The scope I would state explicitly is nomic-embed-text on Spanish administrative text. Whether the collapse is the embedder, the language, or the domain is cheap to settle: the same twenty pairs through bge-m3 or multilingual-e5-large, plus the negation five in English through nomic. If negation collapses in English too, this stops being about Spanish and becomes a statement about mean-pooled embeddings, which is a considerably bigger piece.
Agreed on the two layers. I raised that as a leak-surface argument, not a fix, and any key derived from the query embedding inherits the same failure.
Waiting a day seems right. My instinct is that the decisive question is not which of your four options you pick, but whether the cache can be made to fail loudly rather than silently, since a wrong cached answer with no signal is far worse than a miss.
Giulio I ran both models you suggested. Before the results, two corrections to my own work, because they change how the first report should be read.
First: two of my five negation pairs were mislabelled. "¿Es obligatorio X?" against "¿No es obligatorio X?" is a confirmatory negative interrogative in Spanish it doesn't invert the answer, and a correct system responds the same to both. I had them as pairs the cache must reject. They should have been accepts. The published minimum for the negation row, 0.9702, was one of those. The report now carries an erratum: github.com/psychohub/rag-onpremise...
Second, and this is the one I'd want you to look at: taking your caveat seriously exposed a confound in my design, not just in my reporting. My negation pairs differed by one token. My paraphrase controls differed by most of their tokens. Any separation I reported could have been surface form rather than semantics. So I added paraphrases with matched lexical overlap one-token synonym swaps and pre-specified that contrast.
The control was not decorative. bge-m3 gives AUC 0.3556 against low-overlap paraphrases and 0.9333 against matched ones, on the same adversarial pairs. The uncontrolled version was measuring lexical distance. My original headline rested on it.
Matched contrast, n=5 against n=9, exact enumeration over all 2002 label assignments:
nomic on Spanish AUC 0.1333, margin −0.1017. The error-minimising threshold accepts nothing: the optimal cache configuration is no cache.
nomic on English AUC 0.4444, p=0.797. Not significant. The honest reading is that the score carries no usable information here, not that English does better. Negation similarity averages 0.9520 across nine pairs, comparable to Spanish. I've retracted the language-based explanation from the original report. Nine pairs won't establish a claim about mean pooling, but the Spanish attribution doesn't survive.
bge-m3 on Spanish AUC 0.9333, p=0.0070. It orders correctly, and it cleanly resolves temporal and entity distinctions that nomic could not. But the margin is −0.0086, and removing one pair flips its sign. Not "bge-m3 works" undetermined, and I can say how far from determined.
On the reranker: ms-marco-MiniLM-L-6-v2, scored in both directions since a cache needs a symmetric relation. Negative. In English the score is significantly inverted (AUC 0.0667, p=0.0070, margin −6.27 logits); in Spanish the primary contrast doesn't reach significance. The reason is structural and reads straight off the distributions in English the two populations that must be accepted sit one below and one above the population that must be rejected (matched paraphrases 4.58, negations 7.69, confirmatory 9.31). There is no cut point, at any threshold.
That ordering tracks lexical overlap almost exactly: confirmatory inserts a token, negation swaps a particle, matched paraphrase swaps a content word. Which is to say the cross-encoder fails the same way the bi-encoder does, not a different way. A relevance objective is well approximated by surface overlap, and negation preserves topical relevance nearly intact. That's evidence against this model for this task, not against cross-encoders — but it does make me expect BGE-reranker-v2-m3 to be more inverted rather than less, since it's trained on the same objective. That's a prediction, not a result, and your 240k-pair benchmark is better positioned to test it than I am.
Latency, since you asked whether it fits the budget: 32.6 ms per pair on CPU covering both directions, no batching. Cost wasn't the obstacle. Signal was.
On failing loudly rather than silently I agree that's the decisive question and I still don't have a clean design. It's the piece I most want to think about properly rather than ship.
Scripts and raw similarities are in docs/experiments/. The JSON holds the scores, so the reanalysis runs without Ollama.
Appreciate the corrections and the erratum. One thing that could save you a step and give you a comparison: I built a RAG called RE-call (github.com/GiulioDER/RE-call) that self-hosts, so your corpus stays inside your own data protection constraints, and its scope matches what you're already testing here.
It can be tailored to your own requirements and hardware, backed by Postgres with pgvector. Point it at your corpus, and it gives you a same-conditions comparison without having to build a new harness for it. I'd be glad to walk through the setup with you if that's useful.
On the reranker finding, the surface-overlap explanation is convincing given the population numbers you laid out, and I think your prediction about BGE-reranker-v2-m3 is testable rather than speculative, so I'd be curious what you find if you run it.
Giulio — I ran BGE-reranker-v2-m3. My prediction failed.
I'd said it should be more inverted than MiniLM on the primary contrast, since both train on the same relevance objective. It isn't. AUC 0.7667 in Spanish and 0.8222 in English, against MiniLM's 0.3333 and 0.0667. Wrong side of 0.50 in both languages. And MiniLM's English inversion — the single significant result in that whole run, p=0.0070 — does not replicate with another cross-encoder.
What I can't claim is that BGE works. Neither Spanish result reaches significance (p=0.1174 and p=0.3636) and English lands at p=0.0599. The defensible statement is that the prediction failed, not that the model succeeded. Same brake I had to apply to bge-m3 earlier.
The operating point doesn't move regardless. Best threshold in Spanish gives 3 errors out of 14, English 2 out of 14. Ordering well and cutting well are different things, and a cache needs the second.
One thing I'd point you at, because it's more interesting than my failed prediction. BGE scores AUC 1.0000 on temporal and entity contrasts and still doesn't resolve polarity — which is the same failure profile I measured for bge-m3 as a bi-encoder. Two models from the same family, two different architectures, the same shape of failure. That's an observation, not a conclusion: two models won't distinguish "property of the training objective" from "property of this family," and I'd need models from other families to say which. Writeup here: github.com/psychohub/rag-onpremise...
Two methodological notes in case they save you time. The two models resolve different activations — ms-marco-MiniLM to Identity, so raw logits; BGE to Sigmoid, so probabilities. AUC and permutation p-values compare across them because both depend only on ordering. Margins don't. I nearly published probabilities labelled as logits. And I checked that the perfect AUCs weren't saturation artifacts: maximum observed is 0.99998, one exact tie in 45 comparisons.
Latency, since you flagged the cost question: 571.1 ms per pair against MiniLM's 36.4, roughly 15.7x on CPU. 568M parameters against 22M. That's operating cost, not evidence — a more expensive model isn't more or less inverted for being expensive.
On RE-call — thanks, and I looked. I don't think it fits this particular experiment, and the reason might be worth saying: what I'm measuring isn't a RAG system. There's no harness, no vector store, no retrieval. It's Ollama, a similarity function, and 47 hand-built pairs. Running it through a full pipeline would reintroduce every variable I spent two weeks isolating.
Where I'd actually want your input is upstream of that. If RE-call caches at all, how do you handle the case where a cached question and the incoming one differ only in polarity? That's the question I still don't have a clean answer to, and it's closer to your "fail loudly rather than silently" framing than any of the model comparisons have been.
Edited: the latency paragraph originally said 32.6 ms and 17.5x. Auditing the repo turned up that 32.6 came from a run whose output no longer exists — the reproduction command I'd documented redirected with >, so each run overwrote the previous file. The figures above are the ones recalculable from the tracked JSON. Fixed in c3f0928, along with the command that was destroying the evidence.
Update: shipped the fix and the write-up.
Commit: github.com/psychohub/rag-onpremise/commit/6f22c11
SemanticCacheEnabled is now a flag with default false, and the README says explicitly why: your hypothesis was correct, and the twenty-pair probe made the case concrete enough that leaving the cache on by default was untenable.
The full experiment report, including the twenty pairs, the raw distributions by category, the reproducibility scripts, and the decision reasoning, is at docs/experiments/threshold-safety.md. Your comment is credited in section 8.
Next thing on the roadmap is the proper embedding evaluation on Spanish clinical text that you flagged as the necessary follow-up — building a real eval set instead of the twenty synthetic pairs. That's the next article.
Hubert, good outcome, you did the right step of running the probe, rather than just taking my word for the mechanism.
One thing I would add to the roadmap now that the flag is off by default: two free models worth putting through the same twenty pairs. For the embedder, BAAI/bge-m3 is the one I would try first. It is multilingual, open weight, and supports dense plus sparse plus late interaction scoring. So it gives you a second axis to check whether the negation collapse, is a property of dense mean pooling specifically, or of the embedder generally.
For a reranker, cross-encoder/ms-marco-MiniLM-L-6-v2 is the one I would reach, given your CPU constraint. It is small enough to stay inside the latency budget you already measured, and it scores a query against a candidate directly, which is closer to the fail loudly, not silently property you said you wanted.
I will say plainly where it fell short for me: on my own reranking benchmark, about 240000 query candidate pairs, this exact model improved recall at 100 but the gain did not reliably convert into a top 5 ranking improvement, not Holm significant in my run. A heavier cross-encoder, BGE-reranker-v2-m3, did convert, but at roughly ten times the inference cost of MiniLM on the same hardware. On CPU that is not a free upgrade, so it is worth timing both on your own queries before picking one, especially if a reranker ends up as your confirmation step ahead of serving a cache hit, rather than only as a retrieval quality tool.
Neither of these fixes the negation problem by itself, but they are free, on premise, and cheap enough to add as two more columns on the twenty pair table you already built.