DEV Community: Hubert García Gordon

Three things I got wrong measuring my own cache

Hubert García Gordon — Tue, 25 Aug 2026 01:48:00 +0000

A team that produces regulatory documents kept getting the same kind of question from other teams: does the current rule allow X? Answering meant someone reading through memos, manuals and regulations to find where the rule was stated, then writing a summary with the source cited. The work was real, and the answer was almost always already written down somewhere.

Retrieval fits this exactly. The user asks, the system summarises, and it cites which document the answer came from. The citation is the whole point — a summary without a source is useless when the question is about a rule.

It also fits the constraint from Part 1: the documents can't leave the perimeter, there's no GPU, and each generated answer costs tens of seconds on CPU.

Which is why a semantic cache looked like the most obvious optimisation in the system. Several teams ask about the same regulation. If two questions mean the same thing, serve the stored answer and skip the model entirely. I implemented it with cosine similarity between question embeddings and a threshold of 0.92, and it went out in the reference implementation this repository publishes — always on, with no flag to turn it off.

Then Giulio D'Erme asked me to prove that the threshold was safe. His example was clinical rather than administrative — pacientes con fiebre against pacientes sin fiebre, one token apart, opposite correct answers — but it transfers to every corpus I care about.

The interesting question turned out not to be whether embedders handle negation badly — they do, and the literature has said so for years. It's what happens when that failure lands in a component that suppresses generation rather than one that reorders results. A bad ranking degrades an answer the user can still see and judge. A cache hit skips the model entirely and returns the opposite rule with full confidence and no signal that anything happened.

Answering him took six days — the commit history runs from 6 to 11 August and every step is in it. The cache turned out to be unsafe, and so did the way I had measured it.

What the first run showed

The design is simple enough to break in one sentence: two questions that differ only in polarity mean opposite things, and a cosine similarity between their embeddings doesn't know that.

So I built twenty pairs of Spanish administrative questions in four categories — negation, temporal, entity, and paraphrase as a control — and measured cosine similarity with nomic-embed-text running on Ollama. The pairs are written by hand for the experiment, not drawn from any query log: leave, sick notes, permits, budgets, the vocabulary any public administration shares. No harness, no vector store, no retrieval. Just an embedder, a similarity function, and a script.

The result was blunt. The highest adversarial similarity was 0.9984, for a pair whose two questions have opposite correct answers. The lowest genuine paraphrase sat at 0.7470. Any threshold high enough to reject the adversarial pairs also rejected every paraphrase the cache existed to catch. No safe threshold existed, so I disabled the cache by default and wrote it up.

That conclusion still stands. Everything else about how I got there needed correcting.

Mistake one: two of my five negation pairs weren't negations

Giulio came back with a caveat about my own numbers: with five pairs per category, counts at a fixed threshold are fragile and readers will quote them as rates. He was right, and building a larger corpus to answer him is where I found the first problem.

Two of my five "negation" pairs were these:

¿Es obligatorio presentar la solicitud con anticipación? vs ¿No es obligatorio presentar la solicitud con anticipación?
¿Los contratos temporales tienen derecho a aguinaldo? vs ¿Los contratos temporales no tienen derecho a aguinaldo?

In Spanish, a negative interrogative like this is confirmatory. It asks for confirmation of the same fact rather than its opposite, and a correct system answers both identically. I had labelled them as pairs the cache must reject when they should have been accepts.

This was a linguistics error, not a code error, and no amount of testing would have caught it. The published minimum for my negation row, 0.9702, belongs to one of those invalid pairs. The count I'd reported as "9 of 15 adversarial pairs" is 7 of 13.

The conclusion survived because the highest adversarial similarity, 0.9984, belongs to a valid pair. But the row was wrong, so the report now carries an erratum and the two pairs live on as a separate confirmatory category — useful, as it turns out, because they're the cleanest possible control: same surface change as a negation, opposite required behaviour.

Mistake two: my control was too easy

This is the one worth the article. It's also the one I'd have defended if someone had raised it, right up until I sat down and wrote out the two populations I was comparing.

Here they are.

The pairs I wanted the cache to reject differed by a single token. Con against sin. Incluye against excluye. Fourteen words, thirteen of them identical, and the correct answer inverts.

The pairs I wanted the cache to accept differed by nearly all of their tokens. ¿Cómo solicito vacaciones? against ¿Cuál es el procedimiento para pedir vacaciones? — one content word in common, vacaciones, and the rest rewritten from scratch.

So one population was near-identical strings that must be rejected. The other was barely-overlapping strings that must be accepted. I measured cosine similarity across both, found that the first scored high and the second scored low, and published that as evidence that the embedder cannot tell a negation from a paraphrase.

Now ask what else would produce that exact table. A function that counted shared tokens and knew nothing about meaning would produce it. Character n-gram overlap would produce it. My experiment could not distinguish the embedder failing at semantics from the embedder succeeding at surface form while I misread the output as semantics — because I had built the two candidate explanations to predict the same result.

That's what a control is for, and mine wasn't one. A control population has to differ from the adversarial population in the thing you're testing and match it in everything else. Mine differed in both, so the contrast couldn't isolate anything. It looked like a clean result because the numbers were far apart, and distance between numbers is not the same as evidence for a claim.

The fix was to build five paraphrases with matched lexical overlap — one-token synonym swaps (solicito → pido, me corresponden → me tocan), the same surface distance as the negations, the opposite required behaviour — and to pre-specify that contrast as the primary one, in writing, before looking at any number it produced.

It mattered more than I expected — and the model that showed me how much wasn't mine either. Giulio's comment of 7 August recommended two: bge-m3 as a second embedder, and cross-encoder/ms-marco-MiniLM-L-6-v2 as a reranker, that one picked explicitly for running on CPU. I ran the embedder first, and it turned the control problem from an argument into a measurement.

Running the same adversarial pairs against both control populations with bge-m3:

against low-overlap paraphrases: AUC 0.3556, p=0.4376
against matched-overlap paraphrases: AUC 0.9333, p=0.0070

Same embedder, same negations, same nine rejects. The only thing that changed was how many words the accepted pairs shared with their twins, and the reading went from no usable information at this sample size to the ordering is right — which is emphatically not the same as the cache works, and I'll come back to that in a moment.

My original headline rested on the uncontrolled version.

The numbers that survive the control

Primary contrast, matched overlap, n=5 accepts against n=9 rejects. The p-values come from enumerating every one of the 2002 ways those labels could be reassigned to the same scores — no normal approximation, no distributional assumption, just counting how often chance produces a result this extreme.

Scope	AUC	Margin	p (exact)
nomic-embed-text, Spanish	0.1333	−0.1017	0.0290
bge-m3, Spanish	0.9333	−0.0086	0.0070
nomic-embed-text, English	0.4444	−0.0707	0.7972

Three different readings, and the discipline is in not collapsing them:

nomic in Spanish is inverted, not merely blind. An adversarial pair outranks a genuine paraphrase most of the time. The error-minimising threshold is the degenerate one that accepts nothing — the optimal cache configuration is no cache. I'd rather lean on the margin of −0.1017 than on a p-value of 0.0290 that sits close to the line. Drop any single pair from the set and recompute, and the margin keeps its sign every time; the p-value is one borderline observation away from moving.

nomic in English doesn't reject the null. The honest statement is that the score carries no usable information in this contrast — not that English performs better, which the data doesn't support. Negation similarity averages 0.9520 across the nine translated pairs against 0.9821 for the same pairs in Spanish: lower, and still nowhere near low enough for a cut point to fit between the negations and the paraphrases. Whatever this is, it isn't about Spanish. I retracted that explanation from the original report.

bge-m3 is undetermined, and I can say how undetermined. It orders correctly, and it cleanly resolves the temporal and entity distinctions that nomic couldn't. But the margin is negative, and dropping one single pair flips its sign. Ordering well and cutting well are different problems; a cache needs a cut point, not a ranking. "We don't know" is the right answer here, and it took building the tooling to be able to say it precisely.

The row that isn't in the table

There is a fourth number, and it is the best one I have.

The unmatched contrast for nomic — the same nine negations, scored against the original low-overlap paraphrases — gives AUC 0.0000 at p=0.0010. Perfect inversion: every single adversarial pair outranks every single paraphrase. It is the lowest p-value anywhere in the run, sitting at the floor of what an exact test on fourteen pairs can report at all.

It is also the confounded one. It is the row my original report rested on, and it is the row this section exists to retract.

Leaving it out of the table is deliberate. Printed alongside three controlled contrasts it would read as the strongest of four findings, when what it actually measures is lexical distance wearing the costume of meaning. The number is in the artefacts, with the script that produced it, for anyone who wants to check that I'm characterising it fairly. It just isn't evidence for the thing I originally said it was evidence for — and a striking number that answers the wrong question is worse than no number, because it stops you looking.

A prediction that failed

The reranker was the other half of Giulio's recommendation, and the reason to try it is real: a cross-encoder scores the two texts jointly instead of comparing two embeddings computed in isolation, so it can see the token that inverts the meaning rather than averaging it away. A confirmation step before serving a cache hit is the obvious place to put that.

He also drew the boundary around his own suggestion. In his reranking benchmark, ms-marco-MiniLM-L-6-v2 improved recall@100 without converting that gain into top-5 — better at dragging the right candidate into the pool than at deciding which one wins. I tested it knowing that, which is the only reason a negative result was worth the run at all: negative, and significantly inverted in English.

I then predicted, in writing and before measuring, that BAAI/bge-reranker-v2-m3 would be more inverted, since it trains on the same relevance objective. It isn't. AUC 0.7667 in Spanish and 0.8222 in English, against MiniLM's 0.3333 and 0.0667 — wrong side of 0.50 in both languages. The prediction failed cleanly.

What I can't claim is that BGE works. Neither Spanish result reaches significance, English lands at p=0.0599, and its best available threshold in Spanish still costs 3 errors out of 14 — one false hit and two of five legitimate hits thrown away; in English, 2 out of 14. Registering the prediction beforehand is what makes the failure worth reporting; a prediction written after seeing the result isn't one.

For anyone weighing the cost: 571.1 ms per pair on CPU against MiniLM's 36.4, about 15.7×. Cost was never the obstacle. Signal was.

Mistake three: the literature had this in 2021

I designed and ran five versions of this experiment without a literature review. I did the review afterwards, and it changes what I can claim.

NevIR (EACL 2024) builds essentially my experiment at scale: it asks retrieval models to rank two documents that differ only by negation, and finds that most models — including state-of-the-art ones — perform at or below random ranking, with cross-encoders barely above chance and bi-encoders below it. My nomic result is a bi-encoder below chance, and my two cross-encoders land on opposite sides of chance, which is what you'd expect from a signal hovering around it.

Consistent with, not confirmed by. NevIR ranks two documents against one query; I'm asking whether two queries share an answer. Those aren't the same task, and fourteen pairs can't confirm anything about a paper's benchmark.

A 2021 biomedical paper found the inversion itself: across every model tested, mean cosine similarity for negation and antonym subsets ran higher than for sentence pairs human experts rated as highly similar. That's my finding, in another domain, five years earlier.

And HEROS offers a candidate for the mechanism I went looking for and didn't find: encoders fine-tuned on paraphrase datasets with contrastive learning come out highly sensitive to negation, while fine-tuning only on question-answer pairs leaves a model insensitive to it. On that account the training objective predicts the failure — not the architecture, and not the language. nomic-embed-text trains mostly on retrieval pairs, which puts it on the insensitive side, which is exactly where I measured it.

That's a tidy story and I haven't earned it, so I wrote down what would falsify it before running anything further: an NLI-tuned encoder should separate the primary contrast where nomic doesn't, AUC materially above 0.50 instead of 0.1333. If it doesn't separate, HEROS doesn't explain this case and the cause is still open. That prediction is dated, unmeasured, and sitting in the report where anyone can hold me to it.

So the phenomenon isn't mine. What none of these papers covers is the operating context I opened with: they measure ranking. Ranking failures are visible and recoverable. A cache hit is neither.

Three corrections, then, and not one of them moved the decision. Two pairs left the negation set; the control was rebuilt from scratch, and the same rebuilt corpus took my Spanish explanation down with it; the literature review shrank what was left to what nobody had published already — and the cache is disabled today for the same reason it was disabled on day one. That distinction is the whole point: the method was wrong and the answer was right, and I only know which was which because I measured it again instead of defending it.

What survived

The cache stays disabled by default, and the evidence for that is stronger now than when I published it, not weaker.

What changed is everything about how much I can claim. Not "cosine similarity fails at negation" — that was known. Just: with lexical overlap controlled, across three scopes, no usable cut point appeared, and in one of them the score was actively misleading.

If you're about to add a semantic cache to a retrieval system, the thing I'd take from this isn't the finding. It's that a control which shares no words with the thing it controls for isn't a control. Mine looked like evidence right up to the day after I published it. It was measuring string distance, and I only found out because Giulio asked a harder question about my own numbers than I had asked myself.

The pairs, the scripts, the raw scores and the erratum are in docs/experiments/. The JSON holds the similarities, so the reanalysis runs without Ollama and without repeating a single embedding.

This is a personal open-source project. It describes no institutional deployment and uses no data from any production system. Written in Spanish, self-translated, with Claude as a language editor; the experiments, the measurements and the mistakes are mine.

On-premise RAG without GPU, cloud, or Docker: five lessons that cost me a week each

Hubert García Gordon — Sat, 01 Aug 2026 03:10:27 +0000

Every RAG tutorial I've read makes the same two assumptions: you have a GPU, and you can call a cloud API. For the environments I build for, both assumptions are wrong.

I work on health information systems in the public sector. The stack has to run inside institutional infrastructure — no data leaves the network — and the hardware I get is whatever the procurement cycle produced two years ago. In practice that means Windows Server, CPU only, and open-weight models running locally.

So I built a RAG stack that runs entirely on-premise, no GPU, no cloud, no Docker. It's open source at github.com/psychohub/rag-onpremise: ASP.NET Core 9 for orchestration, Ollama for local inference, Qdrant for vectors, Python for the ingest pipeline, Mistral 7B as the LLM, nomic-embed-text for embeddings.

Getting it into production took longer than the design did, because five things broke that no tutorial had warned me about. This is the field report.

The environment, and why it matters

Before the lessons, it's worth being precise about the constraint, because it changes what "good" looks like.

The stack has to run on a Windows Server, not a Linux workstation. Docker is not available on many of the target machines — either because it wasn't approved, because GPO policies restrict it, or because ops teams already run everything as Windows services and adding a container runtime is a new operational surface nobody wants to own. GPUs are aspirational. In the meantime, you have CPU inference and you have to make it work.

None of this is exotic. It's the default reality in a lot of public sector, healthcare, and legacy enterprise environments. It's also the reality most RAG content on the internet quietly assumes away.

The overall shape of the system:

Documents (PDF / Word / Excel)
    │
    ▼
[ Python ingest ]
    ├─ Text extraction  (pdfplumber, python-docx, openpyxl)
    ├─ Chunking         (500 tokens, 50 overlap)
    ├─ Embeddings       (nomic-embed-text via Ollama)
    └─ Store            (Qdrant, cosine similarity)
                                                      │
User query                                            │
    │                                                 │
    ▼                                                 │
[ ASP.NET Core 9 API ]  ────────────────────────────  ┘
    ├─ 1. Embed the question
    ├─ 2. Retrieve top-K chunks from Qdrant
    ├─ 3. Assemble prompt with context
    ├─ 4. Call Mistral 7B via Ollama
    └─ 5. Return answer + cited sources

Lesson 1: The Qdrant .NET SDK uses gRPC. Use REST directly.

The first thing I tried was the official Qdrant SDK for .NET. Clean API, well documented, felt like the right choice. It also failed in a way that took me a day to diagnose, because the failure wasn't obvious: connections were being established, then dropped, with error messages that pointed at everything except the actual cause.

The cause: the SDK talks to Qdrant over gRPC, and the network path between the .NET application and the Qdrant instance was HTTP/1.1 only. gRPC needs HTTP/2. Some intermediate proxy or load balancer downgraded the connection, and the SDK didn't degrade gracefully — it just failed.

The fix was to skip the SDK entirely and talk to Qdrant's REST API directly with HttpClient:

// Not this — the SDK uses gRPC under the hood
// var client = new QdrantClient(new Uri(url));

// This — plain REST works everywhere
var response = await _httpClient.PostAsync(
    $"{qdrantUrl}/collections/{collection}/points/search",
    content);

Qdrant's REST API is complete enough for RAG workloads. You lose type safety and some ergonomics; you gain the ability to deploy without arguing about HTTP/2 support at every network hop. In a corporate or public sector network, that's a good trade.

Lesson 2: The default HttpClient timeout will kill your responses.

HttpClient in .NET defaults to a 100-second timeout. That's fine for most HTTP work. It is not fine when the thing on the other end is Mistral 7B running on CPU.

On a modest server — 4 vCPU, 16 GB RAM — Mistral 7B takes between 60 and 120 seconds to produce a full response. The first time I ran an end-to-end query, it worked. The second time it worked. The third time the model happened to generate a longer answer and the client timed out, mid-stream, leaving the user staring at a generic error while the server continued generating an answer nobody would ever see.

The fix is two lines:

// Not this — 100s default, will cut you off
var client = new HttpClient();

// This — set the ceiling explicitly, above your worst-case
var client = new HttpClient { Timeout = TimeSpan.FromSeconds(300) };

The number itself matters less than the discipline. If you're calling a local LLM on CPU, measure the worst case on your actual hardware, and set the timeout comfortably above it. And if you're building a UI on top of this, put a progress indicator. Ninety seconds of silence looks like a broken system, even when it's working exactly as designed.

Lesson 3: Ollama only listens on localhost by default.

This one I discovered when I moved from developing on my laptop to deploying on the server, and the .NET application on a different machine couldn't reach Ollama.

Ollama, out of the box, binds to 127.0.0.1:11434. Fine for local development. Useless for any deployment where the LLM host is separate from the application host, or even where the application runs under a service account that doesn't share the loopback context with the interactive user.

The fix is an environment variable:

$env:OLLAMA_HOST = "0.0.0.0:11434"
ollama serve

Which is simple, once you know. The trap is that Ollama's error messages when it's unreachable are generic connection errors, not "hey, I'm only listening on loopback." I spent an afternoon reading firewall rules before I checked the binding.

If you're deploying Ollama as a Windows service — which you probably should — that environment variable needs to be set at service level, not user level. Setting it in a PowerShell prompt won't affect the service. Small detail, real time cost.

Lesson 4: The Python MSI installer fails under corporate GPO. Use the embeddable package.

The ingest pipeline is Python. On a locked-down Windows Server with corporate Group Policy Objects controlling what installers can run, the standard Python MSI would not install. It failed in ways that ranged from silent "operation completed" with nothing on disk, to loud errors about elevation that the actual admin account couldn't resolve either.

The fix, which is not obvious the first time: use the embeddable Python package. It's a ZIP file, not an installer, so it sidesteps most of the GPO surface.

The setup is a little more manual than the installer:

Download python-3.x.x-amd64-embed.zip from python.org.
Extract to a folder — C:\Python311\, wherever.
In that folder, open python3xx._pth and uncomment the import site line. Without this, pip won't work.
Download get-pip.py and run python get-pip.py from that folder.
From there, pip install -r requirements.txt works normally.

Nothing here is hard. It's just not documented as the default path, so if you don't know it exists you spend two days fighting an installer that will never succeed.

Lesson 5: The prompt is where most of the quality lives.

I spent weeks tuning chunking, embedding parameters, and retrieval top-K, and got single-digit percentage improvements each time. Then I rewrote the prompt template and got a step-change in response quality that made all the retrieval tuning look like rounding error.

The two failure modes I kept oscillating between:

Too restrictive. "Answer ONLY from the context. If the context does not contain the answer, say you don't know." The model became allergic to context. It would refuse to answer questions that were partially covered, refuse to make reasonable inferences, and pepper the user with "I don't know" for questions any human reading the same documents could answer.

Too permissive. "Use the context to help you answer the question." The model started hallucinating confidently, filling in gaps in the retrieved chunks with plausible-sounding invention. In a regulated environment, that's not a quality problem. It's a liability.

What ended up working, roughly:

Answer BASED on the provided context.
If the information is partially relevant, use it and be explicit
about what the context does and does not say.
Only if there is absolutely nothing related to the question,
say so clearly.
Do NOT invent data that is not in the context.

The keywords that mattered were "partially relevant" (permission to reason from incomplete context) and "be explicit about what the context does and does not say" (forcing the model to distinguish what it read from what it inferred). Neither is a magic incantation. But together they moved the balance from "refuses to answer" and "makes things up" to "answers when it can, defers when it can't, and tells you which."

What CPU inference actually looks like

The other thing tutorials skip: numbers. Everything above assumes latency you can live with. Here's what I actually measured on the hardware I had:

Hardware	Model	Response time
4 vCPU / 16 GB RAM	Mistral 7B	60–120 seconds
16 vCPU / 32 GB RAM	Mistral 7B	20–45 seconds
4 vCPU / 8 GB RAM	phi3:mini	15–30 seconds
GPU 8 GB+	Mistral 7B	3–8 seconds

The CPU rows are sustained measurements on the servers I actually deploy on. The GPU row is from a single test on borrowed hardware, not sustained production measurements — take that one as a reference point, not a promise.

Two things worth calling out. First, phi3:mini on modest hardware is competitive with Mistral 7B on much better hardware, for latency. If your quality bar allows it, downgrade the model before you upgrade the hardware. Second, the jump from CPU to GPU is roughly 10×. If you can get one 8 GB GPU into your environment, do it — it changes what interactions are possible.

Because CPU latency is what it is, the repo includes a semantic cache in front of the LLM: cosine similarity between the incoming query and cached queries, with a threshold of 0.92. When a user asks something semantically close to a previous query, they get the cached answer in under a second. When they ask something new, they wait for the model. On a moderately busy internal system, cache hit rates got high enough that the average user experience felt reasonable, even though the worst case was still ninety seconds.

One warning I learned by breaking it: clear the cache when you change the LLM. Cached answers are pinned to whoever generated them. When you swap Mistral for a newer model, the cache is now returning answers from a model you're no longer running, and users will notice the personality change before you do.

What I'd tell someone starting today

If you're building on-premise RAG on constrained hardware, the compressed version:

Talk to Qdrant over REST, not gRPC. Fewer surprises on corporate networks.
Set your HTTP timeouts explicitly. The defaults were designed for web traffic, not local LLMs on CPU.
Configure Ollama's binding for your deployment, not your laptop. And set the environment variable at service level if you're running it as a service.
Use Python's embeddable package on locked-down Windows. The MSI is not your friend under GPO.
Tune the prompt before the retrieval. Chunking and top-K matter, but the prompt is where the quality bar actually sits.
Cache aggressively when your LLM is slow, and remember to invalidate when you change models.
Downgrade the model before upgrading the hardware. phi3:mini on 8 GB RAM beats Mistral 7B on a machine you can't afford.

None of this is exotic. It's the part of RAG that gets skipped when the tutorial assumes GPU, cloud, and a Linux dev box. When you don't have any of those, this is the reality you build against.

The next thing on my roadmap is proper embedding evaluation on Spanish clinical text — because "it works" and "it works well in your language on your corpus" are not the same thing, and I haven't measured the gap yet. That's the next article.

This article describes the design and implementation of my personal open-source project rag-onpremise. The measurements are from my own test hardware and my own project, not from any specific institutional deployment. The views expressed here are my own.

Hubert García Gordon works on health information systems in the Costa Rican public sector and teaches at UNED Costa Rica. He maintains rag-onpremise and writes about applied AI in constrained environments.