Every RAG tutorial I've read makes the same two assumptions: you have a GPU, and you can call a cloud API. For the environments I build for, both assumptions are wrong.
I work on health information systems in the public sector. The stack has to run inside institutional infrastructure — no data leaves the network — and the hardware I get is whatever the procurement cycle produced two years ago. In practice that means Windows Server, CPU only, and open-weight models running locally.
So I built a RAG stack that runs entirely on-premise, no GPU, no cloud, no Docker. It's open source at github.com/psychohub/rag-onpremise: ASP.NET Core 9 for orchestration, Ollama for local inference, Qdrant for vectors, Python for the ingest pipeline, Mistral 7B as the LLM, nomic-embed-text for embeddings.
Getting it into production took longer than the design did, because five things broke that no tutorial had warned me about. This is the field report.
The environment, and why it matters
Before the lessons, it's worth being precise about the constraint, because it changes what "good" looks like.
The stack has to run on a Windows Server, not a Linux workstation. Docker is not available on many of the target machines — either because it wasn't approved, because GPO policies restrict it, or because ops teams already run everything as Windows services and adding a container runtime is a new operational surface nobody wants to own. GPUs are aspirational. In the meantime, you have CPU inference and you have to make it work.
None of this is exotic. It's the default reality in a lot of public sector, healthcare, and legacy enterprise environments. It's also the reality most RAG content on the internet quietly assumes away.
The overall shape of the system:
Documents (PDF / Word / Excel)
│
▼
[ Python ingest ]
├─ Text extraction (pdfplumber, python-docx, openpyxl)
├─ Chunking (500 tokens, 50 overlap)
├─ Embeddings (nomic-embed-text via Ollama)
└─ Store (Qdrant, cosine similarity)
│
User query │
│ │
▼ │
[ ASP.NET Core 9 API ] ──────────────────────────── ┘
├─ 1. Embed the question
├─ 2. Retrieve top-K chunks from Qdrant
├─ 3. Assemble prompt with context
├─ 4. Call Mistral 7B via Ollama
└─ 5. Return answer + cited sources
Lesson 1: The Qdrant .NET SDK uses gRPC. Use REST directly.
The first thing I tried was the official Qdrant SDK for .NET. Clean API, well documented, felt like the right choice. It also failed in a way that took me a day to diagnose, because the failure wasn't obvious: connections were being established, then dropped, with error messages that pointed at everything except the actual cause.
The cause: the SDK talks to Qdrant over gRPC, and the network path between the .NET application and the Qdrant instance was HTTP/1.1 only. gRPC needs HTTP/2. Some intermediate proxy or load balancer downgraded the connection, and the SDK didn't degrade gracefully — it just failed.
The fix was to skip the SDK entirely and talk to Qdrant's REST API directly with HttpClient:
// Not this — the SDK uses gRPC under the hood
// var client = new QdrantClient(new Uri(url));
// This — plain REST works everywhere
var response = await _httpClient.PostAsync(
$"{qdrantUrl}/collections/{collection}/points/search",
content);
Qdrant's REST API is complete enough for RAG workloads. You lose type safety and some ergonomics; you gain the ability to deploy without arguing about HTTP/2 support at every network hop. In a corporate or public sector network, that's a good trade.
Lesson 2: The default HttpClient timeout will kill your responses.
HttpClient in .NET defaults to a 100-second timeout. That's fine for most HTTP work. It is not fine when the thing on the other end is Mistral 7B running on CPU.
On a modest server — 4 vCPU, 16 GB RAM — Mistral 7B takes between 60 and 120 seconds to produce a full response. The first time I ran an end-to-end query, it worked. The second time it worked. The third time the model happened to generate a longer answer and the client timed out, mid-stream, leaving the user staring at a generic error while the server continued generating an answer nobody would ever see.
The fix is two lines:
// Not this — 100s default, will cut you off
var client = new HttpClient();
// This — set the ceiling explicitly, above your worst-case
var client = new HttpClient { Timeout = TimeSpan.FromSeconds(300) };
The number itself matters less than the discipline. If you're calling a local LLM on CPU, measure the worst case on your actual hardware, and set the timeout comfortably above it. And if you're building a UI on top of this, put a progress indicator. Ninety seconds of silence looks like a broken system, even when it's working exactly as designed.
Lesson 3: Ollama only listens on localhost by default.
This one I discovered when I moved from developing on my laptop to deploying on the server, and the .NET application on a different machine couldn't reach Ollama.
Ollama, out of the box, binds to 127.0.0.1:11434. Fine for local development. Useless for any deployment where the LLM host is separate from the application host, or even where the application runs under a service account that doesn't share the loopback context with the interactive user.
The fix is an environment variable:
$env:OLLAMA_HOST = "0.0.0.0:11434"
ollama serve
Which is simple, once you know. The trap is that Ollama's error messages when it's unreachable are generic connection errors, not "hey, I'm only listening on loopback." I spent an afternoon reading firewall rules before I checked the binding.
If you're deploying Ollama as a Windows service — which you probably should — that environment variable needs to be set at service level, not user level. Setting it in a PowerShell prompt won't affect the service. Small detail, real time cost.
Lesson 4: The Python MSI installer fails under corporate GPO. Use the embeddable package.
The ingest pipeline is Python. On a locked-down Windows Server with corporate Group Policy Objects controlling what installers can run, the standard Python MSI would not install. It failed in ways that ranged from silent "operation completed" with nothing on disk, to loud errors about elevation that the actual admin account couldn't resolve either.
The fix, which is not obvious the first time: use the embeddable Python package. It's a ZIP file, not an installer, so it sidesteps most of the GPO surface.
The setup is a little more manual than the installer:
- Download
python-3.x.x-amd64-embed.zipfrom python.org. - Extract to a folder —
C:\Python311\, wherever. - In that folder, open
python3xx._pthand uncomment theimport siteline. Without this,pipwon't work. - Download
get-pip.pyand runpython get-pip.pyfrom that folder. - From there,
pip install -r requirements.txtworks normally.
Nothing here is hard. It's just not documented as the default path, so if you don't know it exists you spend two days fighting an installer that will never succeed.
Lesson 5: The prompt is where most of the quality lives.
I spent weeks tuning chunking, embedding parameters, and retrieval top-K, and got single-digit percentage improvements each time. Then I rewrote the prompt template and got a step-change in response quality that made all the retrieval tuning look like rounding error.
The two failure modes I kept oscillating between:
Too restrictive. "Answer ONLY from the context. If the context does not contain the answer, say you don't know." The model became allergic to context. It would refuse to answer questions that were partially covered, refuse to make reasonable inferences, and pepper the user with "I don't know" for questions any human reading the same documents could answer.
Too permissive. "Use the context to help you answer the question." The model started hallucinating confidently, filling in gaps in the retrieved chunks with plausible-sounding invention. In a regulated environment, that's not a quality problem. It's a liability.
What ended up working, roughly:
Answer BASED on the provided context.
If the information is partially relevant, use it and be explicit
about what the context does and does not say.
Only if there is absolutely nothing related to the question,
say so clearly.
Do NOT invent data that is not in the context.
The keywords that mattered were "partially relevant" (permission to reason from incomplete context) and "be explicit about what the context does and does not say" (forcing the model to distinguish what it read from what it inferred). Neither is a magic incantation. But together they moved the balance from "refuses to answer" and "makes things up" to "answers when it can, defers when it can't, and tells you which."
What CPU inference actually looks like
The other thing tutorials skip: numbers. Everything above assumes latency you can live with. Here's what I actually measured on the hardware I had:
| Hardware | Model | Response time |
|---|---|---|
| 4 vCPU / 16 GB RAM | Mistral 7B | 60–120 seconds |
| 16 vCPU / 32 GB RAM | Mistral 7B | 20–45 seconds |
| 4 vCPU / 8 GB RAM | phi3:mini | 15–30 seconds |
| GPU 8 GB+ | Mistral 7B | 3–8 seconds |
The CPU rows are sustained measurements on the servers I actually deploy on. The GPU row is from a single test on borrowed hardware, not sustained production measurements — take that one as a reference point, not a promise.
Two things worth calling out. First, phi3:mini on modest hardware is competitive with Mistral 7B on much better hardware, for latency. If your quality bar allows it, downgrade the model before you upgrade the hardware. Second, the jump from CPU to GPU is roughly 10×. If you can get one 8 GB GPU into your environment, do it — it changes what interactions are possible.
Because CPU latency is what it is, the repo includes a semantic cache in front of the LLM: cosine similarity between the incoming query and cached queries, with a threshold of 0.92. When a user asks something semantically close to a previous query, they get the cached answer in under a second. When they ask something new, they wait for the model. On a moderately busy internal system, cache hit rates got high enough that the average user experience felt reasonable, even though the worst case was still ninety seconds.
One warning I learned by breaking it: clear the cache when you change the LLM. Cached answers are pinned to whoever generated them. When you swap Mistral for a newer model, the cache is now returning answers from a model you're no longer running, and users will notice the personality change before you do.
What I'd tell someone starting today
If you're building on-premise RAG on constrained hardware, the compressed version:
- Talk to Qdrant over REST, not gRPC. Fewer surprises on corporate networks.
- Set your HTTP timeouts explicitly. The defaults were designed for web traffic, not local LLMs on CPU.
- Configure Ollama's binding for your deployment, not your laptop. And set the environment variable at service level if you're running it as a service.
- Use Python's embeddable package on locked-down Windows. The MSI is not your friend under GPO.
- Tune the prompt before the retrieval. Chunking and top-K matter, but the prompt is where the quality bar actually sits.
- Cache aggressively when your LLM is slow, and remember to invalidate when you change models.
-
Downgrade the model before upgrading the hardware.
phi3:minion 8 GB RAM beats Mistral 7B on a machine you can't afford.
None of this is exotic. It's the part of RAG that gets skipped when the tutorial assumes GPU, cloud, and a Linux dev box. When you don't have any of those, this is the reality you build against.
The next thing on my roadmap is proper embedding evaluation on Spanish clinical text — because "it works" and "it works well in your language on your corpus" are not the same thing, and I haven't measured the gap yet. That's the next article.
This article describes the design and implementation of my personal open-source project rag-onpremise. The measurements are from my own test hardware and my own project, not from any specific institutional deployment. The views expressed here are my own.
Hubert García Gordon works on health information systems in the Costa Rican public sector and teaches at UNED Costa Rica. He maintains rag-onpremise and writes about applied AI in constrained environments.
Top comments (15)
Two production guardrails stood out around the semantic cache. In the sample, cache lookup appears before the requested collection is selected, so a semantically similar question for collection B could reuse an answer and sources produced for collection A. I’d partition entries by authorization scope, collection or corpus revision, embedding model, prompt and chat-model versions, and retrieval settings. With document-level permissions, I’d also retain stable source IDs and re-authorize them on every hit. The trade-off is hit rate: per-user partitions are safest but fragment the cache, while role or group partitions with hit-time source checks preserve more sharing.
The timeout lesson has a second half too: flow HttpContext.RequestAborted through QueryAsync, Qdrant, Ollama, and response reads. Raising HttpClient.Timeout avoids premature failures, but cancellation stops disconnected requests consuming scarce CPU. Would you consider a reusable retrieval cache beneath an authorization-aware answer cache?
Ivan, thanks — this is exactly the kind of comment I hoped this piece would attract.
Both guardrails are fair, and I want to be honest about where the current repo stands versus where it needs to go.
On the cache and authorization scope. You're right that the cache in the reference implementation looks up by semantic similarity of the question first, without partitioning by the requested collection, embedding model, prompt version, or retrieval settings. In the single-collection reference scenario the repo demonstrates, this is not currently exploitable — but the moment someone extends it to multi-collection or document-level permissions (which is exactly the direction any serious deployment goes), the sharing you described becomes a real cross-tenant leak vector. That's a design gap, and calling it out publicly is the right call. I'm going to add a compound cache key covering at minimum: collection, embedding model, chat model, prompt version, and retrieval top-K. Per-user partitioning versus role-based with hit-time re-authorization is the interesting trade-off you raised, and I'll write that up as an ADR in the repo when I ship the change so the reasoning is visible, not just the code.
On timeouts and cancellation. Also right. Raising HttpClient.Timeout was lesson two in the article, but I stopped at "don't get cut off." The second half — propagating HttpContext.RequestAborted as CancellationToken through QueryAsync, the Qdrant call, the Ollama call, and the stream reads — is what actually protects CPU on disconnected clients. On CPU inference that matters much more than it does on GPU, because a 90-second orphaned generation is 90 seconds of a scarce resource going to nobody. I'll add the cancellation propagation in the same pass.
On the two-layer cache proposal. Yes, I would consider it, and I think it's the right shape. A retrieval cache below the answer cache — keyed on the embedded query plus collection plus embedding-model version — can be shared across users because chunks carry their own authorization metadata and get re-checked at hit time. The answer cache above stays authorization-partitioned and much smaller. You get most of the sharing benefit on the expensive part (embedding + vector search) without leaking generated answers across scopes. I'll open an issue with that design and link it here when it's up.
Appreciate you engaging with the specifics rather than the surface — this is exactly the follow-up conversation I wanted this article to open.
Thanks, Hubert — I appreciate the clear distinction between the safe single-collection reference scenario and what changes in a permissioned deployment.
The two-layer design is exactly what I had in mind. One detail I’d make explicit in the ADR: share cached candidate source IDs and scores, then re-resolve the current corpus revision and ACLs on every retrieval-cache hit. If revocation removes candidates, backfill from vector search before generation. I’d also include corpus and authorization-policy revisions in the invalidation strategy.
A useful acceptance test would repeat the same query across collections A and B, then change a role or revoke a document and bump the prompt, model, and top-K versions. No answer should cross scope, stale candidates should be rejected, and disconnecting a request should release the Ollama generation slot within a measured bound.
Ivan — the ADR is up with your refinements in it: github.com/psychohub/rag-onpremise...
It has an implementation status table, because writing it exposed a gap between what I told you in August and what the code does.
Shipped: five of the eight key components — collection, embedding model, chat model, prompt version, retrieval top-K. Not shipped: authorization scope, corpus revision, authorization-policy revision. Those aren't omissions in the key builder, they're inputs the repository doesn't have — no authenticated caller, no revision counter, no versioned policy. The retrieval cache of section 3 doesn't exist either; it's written as proposed and credited to you.
A correction I owe you: I said I'd propagated RequestAborted. I propagated it inside the service and left the interface and controller untouched, so the token reaching Qdrant and Ollama was always CancellationToken.None. The service also stopped satisfying its own interface — uncaught because the repository has no project file and so no build anywhere in the loop. Fixed in d93eab5, cache lock waits included. Your criterion 5 is still unmet: the wiring is a precondition for measuring the bound, not a substitute.
The part I'd have gotten wrong is re-resolving ACLs on every retrieval hit instead of trusting the cached outcome — my instinct was to cache it alongside the candidates, which turns revocation into a delayed leak.
One simplification: here a role grants a whole collection, not documents within one. That makes role-partitioned sharing sound rather than convenient, and revocation needs no sweep — the role set is in the key, so losing a role stops matching those partitions on the next request. Under document-level permissions none of that holds; the ADR names that as the condition that would supersede it.
And one update that makes your two-layer proposal more relevant, not less: the answer cache is disabled by default for reasons unrelated to authorization — cosine similarity doesn't separate a question from its negation, so a hit can serve a confidently wrong answer with no signal. Partitioning bounds what a wrong hit reaches; it doesn't stop it being wrong. Your split does what one layer can't: a retrieval hit returns candidates and the model still reads the actual question against them, so a near-miss degrades retrieval instead of fabricating an answer.
Hubert — thank you for the candid follow-through and generous credit. The implementation-status table is the most valuable part of the ADR: it makes the boundary between shipped, designed, and unmeasured work auditable.
I agree that whole-collection RBAC makes role-set partitioning sound within the stated boundary, with document-level permissions correctly treated as a superseding design. One condition worth keeping explicit at the authentication boundary is that “immediate” revocation requires a freshly resolved role set, or versioned claims that cannot remain valid after revocation.
The answer/retrieval distinction is also much clearer now. Partitioning limits the blast radius of a wrong answer-cache hit; it cannot make the hit correct. A retrieval hit keeps the actual question in the generation path. If retrieval hits are matched semantically, I would also re-score the candidate IDs against the incoming embedding rather than reuse scores produced for the earlier query.
I checked d93eab5: cancellation now reaches the interface, Qdrant, Ollama, deserialization, and the cache-lock waits. Agreed that this proves the plumbing, not slot release. The acceptance test should disconnect a deliberately slow request and measure when Ollama capacity becomes available again.
One concrete integration gap surfaced while checking it: the README registers a client named ollama, while RagService requests ollama-embedding, ollama-generation, and qdrant. Without the missing Program.cs configuration, those receive default client settings, leaving generation on the 100-second default timeout. A minimal buildable sample plus CI and integration tests would catch both that and the interface regression.
This is strong work. Thank you for documenting the corrections and boundaries as carefully as the design.
Ivan — verifying that turned up two more things. The install guide registered no named clients at all, so all three fell through to defaults, not just generation. And lesson two in the article showed a bare new HttpClient as the correct pattern, which the service never uses — a reader who understood the lesson perfectly had nowhere to apply it.
Fixed in 061aa3b: the three names registered with their timeouts, install guide and README now byte-identical, and lesson two rewritten around the named registration.
Worth noting for its own sake: the repository already documented this exact failure in its troubleshooting page, transcribed error message and all, while the happy path walked straight into it. Documentation and code drifted apart in opposite directions and nothing in between noticed.
Which is where your build-versus-integration distinction lands. I had filed the missing project file as the cause of the interface regression, and it was — but CreateClient with an unregistered name compiles and runs fine. Two different holes, and I'd been treating them as one.
Your three design points are in the ADR as dated amendments rather than a superseding record, since none of them changes the decision: github.com/psychohub/rag-onpremise...
The revocation condition is the one I had wrong. The ADR claimed immediacy as a property of the cache key. It belongs to the authentication boundary: with cached claims the revoked role stays valid until the token expires and the partition stays reachable for that interval, key or no key. The amendment states the condition rather than the guarantee — either the role set is resolved afresh per request, or claims are versioned so they cannot survive a revocation.
Re-scoring candidates against the incoming embedding is now explicit in the retrieval layer. Sharing the IDs keeps the saving, since vector search is the expensive part; recomputing the score avoids ranking by relevance to a question nobody asked. Still design — no retrieval cache exists.
Criterion 5 carries your procedure now: disconnect a deliberately slow request, measure when generation capacity returns. It stays unmet, which I'd rather leave visible than quietly satisfy with the wiring.
On the minimal buildable sample — that's the third argument for it this week, and the first one that separates it from making the repository look clonable, which was my objection to it. No date from me yet.
Thank you for reviewing the ADR as carefully as the code. The boundary claims were the parts most likely to be wrong, and one of them was.
Hubert — thank you. I checked 061aa3b and the ADR amendments; they preserve the distinctions accurately.
The most useful lesson here is the separation between verification layers. A build would have caught the IRagService mismatch. It would not have caught the named-client problem because IHttpClientFactory legally returns an unconfigured default client. That needs startup or integration validation which resolves the exact client names and verifies their effective settings. The troubleshooting page describing the failure while the happy path recreated it is a particularly good example of why documentation needs an executable anchor.
I agree that amendments are the right treatment for the three design points: they clarify conditions and measurement without changing the underlying decision. I also appreciate criterion 5 remaining visibly unmet. Observable acceptance criteria are valuable precisely because correct plumbing cannot satisfy them by declaration.
On the minimal sample, I do not think it needs to present the repository as a clone-and-deploy application. It could be explicitly framed as a verification harness for the reference fragments: compile the contracts, validate the named-client configuration, and host the small set of executable safety checks. That would preserve the repository’s scope while closing the gap that allowed both failures through.
Thank you for engaging with this so rigorously and for the generous credit. The resulting ADR, corrections, and record of what remains unproven are substantially more valuable than a thread in which everyone simply agreed.
Lesson 3 is the kind of thing that costs a day and shows up in no tutorial.
One thing I would pressure-test before trusting the semantic cache, complementary to Ivan's scoping point rather than a repeat of it: cosine 0.92 between an incoming query and a cached query measures topical similarity, not answer equivalence. Two questions that differ by a single entity, a date, or a negation usually sit very close in embedding space, because one token moves the vector far less than it moves the correct answer. In clinical Spanish that is not a corner case, it is the normal shape of a question: "pacientes con fiebre" versus "pacientes sin fiebre", or the same query with the year changed.
Cheap way to falsify it, maybe twenty minutes. Take twenty real queries from your logs, write one near twin of each that differs only by an entity or a negation, embed both pairs with nomic-embed-text, and look at the cosine distribution of the twins. If any twin pair clears your threshold, the cache will serve a confidently wrong answer, and it fails silently, since a cache hit never reaches the LLM to be checked. Whatever the top of that distribution turns out to be is your real floor for the threshold, not 0.92.
That probe also fits the embedding evaluation on Spanish clinical text you say is next. Retrieval quality and negation sensitivity are separate axes, and an embedder can score well on the first while being useless on the second.
On lesson 5 my experience matches yours: I benchmarked a set of retrieval-side changes on a different corpus and several standard tricks came back null, while prompt wording moved the number. Caveat firmly on my own result, different corpus and a different question distribution, so treat it as a second data point rather than a property of RAG in general.
One question, since you are CPU-only: of the 60 to 120 seconds on 4 vCPU, how much is prefill over the retrieved context versus generation? That split usually decides whether the next win comes from shrinking top-k or from a smaller model.
Giulio, I ran the twenty-pair probe you suggested. The result is worse than I expected, and I want to publish it here before I write it up properly.
Setup: nomic-embed-text, twenty pairs on Spanish administrative-domain queries, five per category — negation, temporal, entity, and paraphrase-control. Cosine similarity, same math the cache uses.
The numbers, in descending order of concern:
Negation pairs (5/5): cosine range 0.9702–0.9984, mean 0.9837. All five would cache-hit at 0.92. The worst case, "con goce salarial" versus "sin goce salarial", scored 0.9984. Practically identical to the embedder.
Temporal pairs (5/5): cosine range 0.9054–0.9646, mean 0.9372. Three of five would cache-hit at 0.92.
Entity pairs (5/5): cosine range 0.7498–0.9210, mean 0.8641. One of five would cache-hit at 0.92.
Paraphrase controls (5/5): cosine range 0.7470–0.9060, mean 0.8067. Zero of five would cache-hit at 0.92.
The distributions overlap catastrophically. The highest adverse similarity (0.9984) is well above the lowest paraphrase similarity (0.7470). There is no cosine threshold that separates the two categories with nomic-embed-text on Spanish text of this structure. Any threshold high enough to reject the adverse pairs also rejects every genuine paraphrase.
You were right about the mechanism, and the mechanism turned out to be much more severe than the article implied. What I described as "aggressive caching" is, in this configuration, a source of silent wrong answers for exactly the class of question users are most likely to ask.
Two things I want to say before I close this comment:
First, the two-layer cache design I mentioned earlier doesn't fix this on its own — separating retrieval cache from answer cache reduces the leak surface but doesn't help with the underlying semantic collapse. Any answer cache keyed on query embedding has the same problem in this domain.
Second, I'm not going to commit to a specific fix in this reply. I want to sit with the evidence for a day before I decide whether the right move is to disable the cache by default, gate it behind explicit configuration with strong warnings, add symbolic checks on top, or something else. I'll come back to this thread when the PR is up.
The next article will be the full write-up with the twenty pairs, the code, and the distribution. It's a much stronger piece of evidence than anything I could have written from principles.
Thanks for pushing on this. This is exactly the kind of comment that separates readers who engage from readers who ship.
Update: shipped the fix and the write-up.
Commit: github.com/psychohub/rag-onpremise/commit/6f22c11
SemanticCacheEnabled is now a flag with default false, and the README says explicitly why: your hypothesis was correct, and the twenty-pair probe made the case concrete enough that leaving the cache on by default was untenable.
The full experiment report, including the twenty pairs, the raw distributions by category, the reproducibility scripts, and the decision reasoning, is at docs/experiments/threshold-safety.md. Your comment is credited in section 8.
Next thing on the roadmap is the proper embedding evaluation on Spanish clinical text that you flagged as the necessary follow-up — building a real eval set instead of the twenty synthetic pairs. That's the next article.
Hubert, good outcome, you did the right step of running the probe, rather than just taking my word for the mechanism.
One thing I would add to the roadmap now that the flag is off by default: two free models worth putting through the same twenty pairs. For the embedder, BAAI/bge-m3 is the one I would try first. It is multilingual, open weight, and supports dense plus sparse plus late interaction scoring. So it gives you a second axis to check whether the negation collapse, is a property of dense mean pooling specifically, or of the embedder generally.
For a reranker, cross-encoder/ms-marco-MiniLM-L-6-v2 is the one I would reach, given your CPU constraint. It is small enough to stay inside the latency budget you already measured, and it scores a query against a candidate directly, which is closer to the fail loudly, not silently property you said you wanted.
I will say plainly where it fell short for me: on my own reranking benchmark, about 240000 query candidate pairs, this exact model improved recall at 100 but the gain did not reliably convert into a top 5 ranking improvement, not Holm significant in my run. A heavier cross-encoder, BGE-reranker-v2-m3, did convert, but at roughly ten times the inference cost of MiniLM on the same hardware. On CPU that is not a free upgrade, so it is worth timing both on your own queries before picking one, especially if a reranker ends up as your confirmation step ahead of serving a cache hit, rather than only as a retrieval quality tool.
Neither of these fixes the negation problem by itself, but they are free, on premise, and cheap enough to add as two more columns on the twenty pair table you already built.
That is a stronger result than I expected too, and the negation row is the one I would lead with. 0.9984 for "con goce salarial" against "sin goce salarial" is not a near miss, it is the embedder telling you the negation particle carried almost no weight in the pooled vector.
One caveat on my own suggestion: with five pairs per category, the counts at 0.92 ("three of five", "one of five") are fragile and readers will quote them as rates. I would report the distributions and the best separation achievable across all thresholds instead. The claim that survives scrutiny is the one you already made: no threshold separates the two populations.
The scope I would state explicitly is nomic-embed-text on Spanish administrative text. Whether the collapse is the embedder, the language, or the domain is cheap to settle: the same twenty pairs through bge-m3 or multilingual-e5-large, plus the negation five in English through nomic. If negation collapses in English too, this stops being about Spanish and becomes a statement about mean-pooled embeddings, which is a considerably bigger piece.
Agreed on the two layers. I raised that as a leak-surface argument, not a fix, and any key derived from the query embedding inherits the same failure.
Waiting a day seems right. My instinct is that the decisive question is not which of your four options you pick, but whether the cache can be made to fail loudly rather than silently, since a wrong cached answer with no signal is far worse than a miss.
Giulio I ran both models you suggested. Before the results, two corrections to my own work, because they change how the first report should be read.
First: two of my five negation pairs were mislabelled. "¿Es obligatorio X?" against "¿No es obligatorio X?" is a confirmatory negative interrogative in Spanish it doesn't invert the answer, and a correct system responds the same to both. I had them as pairs the cache must reject. They should have been accepts. The published minimum for the negation row, 0.9702, was one of those. The report now carries an erratum: github.com/psychohub/rag-onpremise...
Second, and this is the one I'd want you to look at: taking your caveat seriously exposed a confound in my design, not just in my reporting. My negation pairs differed by one token. My paraphrase controls differed by most of their tokens. Any separation I reported could have been surface form rather than semantics. So I added paraphrases with matched lexical overlap one-token synonym swaps and pre-specified that contrast.
The control was not decorative. bge-m3 gives AUC 0.3556 against low-overlap paraphrases and 0.9333 against matched ones, on the same adversarial pairs. The uncontrolled version was measuring lexical distance. My original headline rested on it.
Matched contrast, n=5 against n=9, exact enumeration over all 2002 label assignments:
nomic on Spanish AUC 0.1333, margin −0.1017. The error-minimising threshold accepts nothing: the optimal cache configuration is no cache.
nomic on English AUC 0.4444, p=0.797. Not significant. The honest reading is that the score carries no usable information here, not that English does better. Negation similarity averages 0.9520 across nine pairs, comparable to Spanish. I've retracted the language-based explanation from the original report. Nine pairs won't establish a claim about mean pooling, but the Spanish attribution doesn't survive.
bge-m3 on Spanish AUC 0.9333, p=0.0070. It orders correctly, and it cleanly resolves temporal and entity distinctions that nomic could not. But the margin is −0.0086, and removing one pair flips its sign. Not "bge-m3 works" undetermined, and I can say how far from determined.
On the reranker: ms-marco-MiniLM-L-6-v2, scored in both directions since a cache needs a symmetric relation. Negative. In English the score is significantly inverted (AUC 0.0667, p=0.0070, margin −6.27 logits); in Spanish the primary contrast doesn't reach significance. The reason is structural and reads straight off the distributions in English the two populations that must be accepted sit one below and one above the population that must be rejected (matched paraphrases 4.58, negations 7.69, confirmatory 9.31). There is no cut point, at any threshold.
That ordering tracks lexical overlap almost exactly: confirmatory inserts a token, negation swaps a particle, matched paraphrase swaps a content word. Which is to say the cross-encoder fails the same way the bi-encoder does, not a different way. A relevance objective is well approximated by surface overlap, and negation preserves topical relevance nearly intact. That's evidence against this model for this task, not against cross-encoders — but it does make me expect BGE-reranker-v2-m3 to be more inverted rather than less, since it's trained on the same objective. That's a prediction, not a result, and your 240k-pair benchmark is better positioned to test it than I am.
Latency, since you asked whether it fits the budget: 32.6 ms per pair on CPU covering both directions, no batching. Cost wasn't the obstacle. Signal was.
On failing loudly rather than silently I agree that's the decisive question and I still don't have a clean design. It's the piece I most want to think about properly rather than ship.
Scripts and raw similarities are in docs/experiments/. The JSON holds the scores, so the reanalysis runs without Ollama.
Appreciate the corrections and the erratum. One thing that could save you a step and give you a comparison: I built a RAG called RE-call (github.com/GiulioDER/RE-call) that self-hosts, so your corpus stays inside your own data protection constraints, and its scope matches what you're already testing here.
It can be tailored to your own requirements and hardware, backed by Postgres with pgvector. Point it at your corpus, and it gives you a same-conditions comparison without having to build a new harness for it. I'd be glad to walk through the setup with you if that's useful.
On the reranker finding, the surface-overlap explanation is convincing given the population numbers you laid out, and I think your prediction about BGE-reranker-v2-m3 is testable rather than speculative, so I'd be curious what you find if you run it.
Giulio — I ran BGE-reranker-v2-m3. My prediction failed.
I'd said it should be more inverted than MiniLM on the primary contrast, since both train on the same relevance objective. It isn't. AUC 0.7667 in Spanish and 0.8222 in English, against MiniLM's 0.3333 and 0.0667. Wrong side of 0.50 in both languages. And MiniLM's English inversion — the single significant result in that whole run, p=0.0070 — does not replicate with another cross-encoder.
What I can't claim is that BGE works. Neither Spanish result reaches significance (p=0.1174 and p=0.3636) and English lands at p=0.0599. The defensible statement is that the prediction failed, not that the model succeeded. Same brake I had to apply to bge-m3 earlier.
The operating point doesn't move regardless. Best threshold in Spanish gives 3 errors out of 14, English 2 out of 14. Ordering well and cutting well are different things, and a cache needs the second.
One thing I'd point you at, because it's more interesting than my failed prediction. BGE scores AUC 1.0000 on temporal and entity contrasts and still doesn't resolve polarity — which is the same failure profile I measured for bge-m3 as a bi-encoder. Two models from the same family, two different architectures, the same shape of failure. That's an observation, not a conclusion: two models won't distinguish "property of the training objective" from "property of this family," and I'd need models from other families to say which. Writeup here: github.com/psychohub/rag-onpremise...
Two methodological notes in case they save you time. The two models resolve different activations — ms-marco-MiniLM to Identity, so raw logits; BGE to Sigmoid, so probabilities. AUC and permutation p-values compare across them because both depend only on ordering. Margins don't. I nearly published probabilities labelled as logits. And I checked that the perfect AUCs weren't saturation artifacts: maximum observed is 0.99998, one exact tie in 45 comparisons.
Latency, since you flagged the cost question: 571.1 ms per pair against MiniLM's 36.4, roughly 15.7x on CPU. 568M parameters against 22M. That's operating cost, not evidence — a more expensive model isn't more or less inverted for being expensive.
On RE-call — thanks, and I looked. I don't think it fits this particular experiment, and the reason might be worth saying: what I'm measuring isn't a RAG system. There's no harness, no vector store, no retrieval. It's Ollama, a similarity function, and 47 hand-built pairs. Running it through a full pipeline would reintroduce every variable I spent two weeks isolating.
Where I'd actually want your input is upstream of that. If RE-call caches at all, how do you handle the case where a cached question and the incoming one differ only in polarity? That's the question I still don't have a clean answer to, and it's closer to your "fail loudly rather than silently" framing than any of the model comparisons have been.
Edited: the latency paragraph originally said 32.6 ms and 17.5x. Auditing the repo turned up that 32.6 came from a run whose output no longer exists — the reproduction command I'd documented redirected with >, so each run overwrote the previous file. The figures above are the ones recalculable from the tracked JSON. Fixed in c3f0928, along with the command that was destroying the evidence.