DEV Community: RecallRun

Make your RAG say 'I don't know': a score threshold, measured

RecallRun — Sun, 25 Oct 2026 11:06:23 +0000

A RAG system always gets something back from the index. Ask it about cooking and it returns the five nearest chunks of API docs, and the LLM then tries to be helpful. So: can a simple score threshold on retrieval tell "the docs cover this" from "they don't"?

Short answer: yes, mostly. On the FastAPI docs, the top-1 cosine score separated the two groups well (AUC 0.986), and a threshold of 0.40 refused 24 of 30 unanswerable questions while answering 29 of 30 answerable ones. The leftovers are near-domain questions, which no threshold fully fixes. Raw BM25 scores did worse, and BM25 almost never returns an empty result.

Setup

Corpus: 152 files of the FastAPI docs, split into 2,808 heading-aware chunks with the heading path prefixed.
Retrievers: MiniLM-L6-v2 cosine similarity, and BM25 over the same chunks.
30 answerable queries: hand-written questions, each matching one docs page (for example "How do I set a cookie on the response?").
30 unanswerable queries: 15 near-domain (pagination, rate limiting, gRPC, Terraform, Prometheus, Selenium...) and 15 unrelated (egg boiling, football, tides...). I checked by keyword search that the near-domain topics are not documented pages, though a few terms are mentioned in passing.
Decision score: the top-1 chunk's score. Refuse when it is below the threshold.

Results

Top-1 cosine score by group:

Group	Median	Min	Max
Answerable (30)	0.619	0.393	0.818
Unanswerable, near-domain (15)	0.385	0.256	0.508
Unanswerable, unrelated (15)	0.218	0.175	0.298

How well each score separates answerable from unanswerable (AUC, 1.0 is perfect, 0.5 is chance):

Signal	AUC
Cosine top-1	0.986
BM25 top-1, stopwords removed	0.927
BM25 top-1, raw	0.910
Cosine gap between top-1 and top-2	0.649

Cosine threshold sweep:

Threshold	Answerable kept	Near-domain refused	Unrelated refused	All unanswerable refused
0.30	100%	13%	100%	57%
0.35	100%	20%	100%	60%
0.40	97% (29/30)	60%	100%	80% (24/30)
0.45	93% (28/30)	73%	100%	87% (26/30)
0.50	90% (27/30)	93%	100%	97% (29/30)
0.55	73%	100%	100%	100%

I also split each group in half, picked the threshold on one half and scored the other. Requiring every answerable calibration query to be kept gave a threshold of 0.393. On the held-out half it kept 15 of 15 answerable queries and refused 11 of 14 unanswerable ones (79%).

What the numbers mean

Unrelated questions are easy. Their best score (0.298) is below the worst answerable score (0.393). Any threshold between 0.30 and 0.39 catches all of them.
Near-domain questions are the real problem. Their scores run up to 0.508, which overlaps the lowest answerable queries. The highest-scoring one was "How do I expose Prometheus metrics from my service?" at 0.508, matched to the OpenTelemetry page. Arguably that page is a partial answer, so this one is a labeling grey zone as much as a miss.
The cost of tightening is visible. Going from 0.40 to 0.50 refuses 5 more unanswerable queries but also loses 2 more answerable ones.
BM25 is a weaker gate. Its raw scores depend on query length and term rarity, so one cutoff does not transfer. It also never returned an empty candidate list for any of the 60 queries, because words like "how" and "do" match something. With stopwords removed, only 5 of the 30 unanswerable queries came back empty. "Check for empty results" alone is not a refusal strategy.
The top-1 vs top-2 gap is nearly useless here (AUC 0.649).

Caveats

60 queries is small. One query moves a percentage by about 3 points.
I wrote both query sets myself, and the unanswerable ones are cleaner than real traffic.
Thresholds are specific to this model and corpus. Recalibrate whenever you change the embedding model or re-chunk.
This tests retrieval only. A model can still hallucinate when the retrieved chunks are on topic but do not contain the answer; that needs an answer-level check such as citation validation.

Copyable code

import numpy as np

def should_refuse(query_vec, chunk_matrix, threshold=0.40):
    """Vectors must be L2-normalised. Returns (refuse, top_score, top_index)."""
    sims = chunk_matrix @ query_vec
    top = int(np.argmax(sims))
    return bool(sims[top] < threshold), float(sims[top]), top

REFUSAL = "I couldn't find this in the documentation."

Pick the threshold from your own labeled set: take the highest value that still answers at least 95-100% of your known-answerable queries, then check what it refuses.

Rule of thumb

Build a set of at least 50 answerable and 50 unanswerable queries from your own logs, including near-domain ones.
Use the top-1 cosine score, not BM25, as the gate, and set the threshold just under your lowest answerable score.
Put the threshold in a test, so a model or chunking change that moves the score scale fails CI.
Expect to refuse most, not all, of the near-domain questions. Treat the rest with answer-level checks.

The script and raw output are on the original post.

I run experiments like this every week at RecallRun. If you want to build RAG systems end to end, the AI Engineering course starts free: https://recallrun.dev/learn/ai-engineering/?utm_source=devto&utm_campaign=rag-idk

Metadata filtering in vector search: post-filter loses results, measured

RecallRun — Thu, 15 Oct 2026 19:19:58 +0000

Multi-tenant RAG needs a filter: “only search documents this user may see.” Vector indexes are built for nearest neighbours, not for “nearest neighbours among these 1,000.” So where does the filter go: after the search, inside it, or before it?

Short answer: post-filtering silently returns fewer results as the filter gets more selective. In my test, a filter that keeps 1% of the data returned 1.9 results out of 10 when I fetched the top 100 and filtered. In-graph filtering fixed recall but got slow. For selective filters, exact search over the filtered subset was the fastest and always correct.

The setup

100,000 vectors, each with a tenant id. I ran two layouts:

random: the tenant is independent of content. Think “documents from many customers about the same product.”
topical: each tenant owns whole topic clusters, and its queries land in its own region.

Selectivity is the fraction of the corpus the filter keeps: about 50%, 10%, 1% and 0.1%. Ground truth is exact search restricted to the tenant’s vectors, so recall@10 here means “how many of the true filtered top 10 did the method return.”

Methods:

post-filter: ask HNSW for the top k’ (10, 100 or 1000), drop results that fail the filter, keep up to 10.
in-graph filter: hnswlib’s filter argument, evaluated while the graph is traversed (hnswlib).
pre-filter exact: select the allowed rows, then do a brute-force dot product. I timed two versions: build the subset per query (mask + gather) and keep a per-tenant matrix prebuilt (partitioned).

def post_filter(index, mask, q, fetch=100, k=10):
    labels, _ = index.knn_query(q, k=fetch)
    return [l for l in labels[0] if mask[l]][:k]

def in_graph(index, mask, q, k=10):
    labels, _ = index.knn_query(q, k=k, filter=lambda i: bool(mask[i]))
    return list(labels[0])

def pre_filter(X_tenant, ids_tenant, q, k=10):  # X_tenant = X[ids_tenant], built once
    s = X_tenant @ q
    top = np.argpartition(-s, k)[:k]
    return list(ids_tenant[top[np.argsort(-s[top])]])

Results: random layout

Recall@10 / average results returned (max 10) / median latency.

Selectivity	post k=10	post k=100	post k=1000 (ef=1000)	in-graph filter	pre-filter (partitioned)
50%	0.553 / 5.5 / 0.21 ms	1.000 / 10 / 0.24 ms	1.000 / 10 / 1.94 ms	1.000 / 10 / 0.44 ms	1.000 / 10 / 1.38 ms
10%	0.181 / 1.8 / 0.20 ms	0.908 / 9.1 / 0.22 ms	1.000 / 10 / 1.83 ms	1.000 / 10 / 2.19 ms	1.000 / 10 / 0.16 ms
1%	0.110 / 1.1 / 0.20 ms	0.191 / 1.9 / 0.22 ms	0.884 / 9.2 / 1.88 ms	1.000 / 10 / 17.03 ms	1.000 / 10 / 0.03 ms
0.1%	0.100 / 1.0 / 0.14 ms	0.105 / 1.0 / 0.16 ms	0.214 / 2.1 / 1.79 ms	1.000 / 10 / 74.22 ms	1.000 / 10 / 0.01 ms

Post-filtering behaves like a coin flip with the selectivity as the odds: fetch 10, keep about selectivity × 10. The query is never told that most of its neighbours are off-limits. Even fetching 1,000 candidates returned only 2.1 results per query at 0.1%.

In-graph filtering never lost recall, but its cost grows as the filter tightens, from 0.44 ms at 50% to 74.22 ms at 0.1%. The traversal has to wander through many forbidden nodes to find allowed ones. At 0.1% it took 74.22 ms, while the exact scan of the 100-vector subset took 0.01 ms.

Results: topical layout

When tenants own their own regions of the space, the problem mostly disappears.

Selectivity	post k=10	in-graph filter	pre-filter (partitioned)
50%	1.000 / 0.20 ms	1.000 / 0.22 ms	1.000 / 1.34 ms
10%	1.000 / 0.20 ms	1.000 / 0.29 ms	1.000 / 0.16 ms
1%	1.000 / 0.12 ms	1.000 / 0.61 ms	1.000 / 0.03 ms
0.1%	1.000 / 0.07 ms	1.000 / 0.16 ms	1.000 / 0.01 ms

Recall is 1.000 for every method. The query’s nearest neighbours are already inside the allowed set, so the filter rejects almost nothing. This is the trap: post-filtering looks fine in a demo where a user asks about their own topic, then fails when a filter is independent of the query. Examples are a date range, a document type, or a tenant whose content overlaps with everyone else’s. Real tenants are somewhere between my two layouts, so test with your own data.

What the numbers mean

Post-filter is only safe when the filter is loose or correlated with the query. Below about 10% selectivity on independent filters it breaks, and it breaks quietly: fewer results, not an error. The 0.1% row returned a single result on average.
Over-fetching helps, but needs a lot. To reach 0.884 recall at 1% selectivity I had to fetch 1,000 and raise ef to 1,000, which cost 1.88 ms versus 0.22 ms.
In-graph filtering is the default to reach for between ~10% and 100%, where it stayed between 0.22 and 2.19 ms (and at 10% the exact partitioned scan was faster, 0.16 ms).
Below ~1%, scan the subset. At 1% the subset has about 1,000 vectors, so a dot product takes 0.03 ms. This is the same reasoning as in Do you need a vector database?: exact search is cheap at small sizes.

Caveats

Synthetic Gaussian clusters, 128 dimensions, one index configuration. Real embeddings will have different structure. Treat the shape of the curves as the result, not the exact milliseconds.
One tenant per setting, 200 queries each. Latency is a median on a shared 2 vCPU machine.
The partitioned pre-filter times exclude building the per-tenant matrices. It needs memory for a second copy of the vectors, or an index per tenant.
hnswlib’s filter is a Python callback here. A native filter in a database engine will have different costs. The qualitative trend (slower as selectivity falls) is reported for hnswlib in the SIEVE paper, which also notes that hnswlib applies the filter during search.
My first run gave a misleading 32 ms for exact search over 50k vectors. The cause was a float64 query array: np.sqrt(D) returns a float64 scalar, and under NumPy 2 this upcast the query, so each dot product converted the whole float32 matrix. I fixed it by casting queries to float32 and reran everything. Check dtypes if your brute-force numbers look too slow.

Rule of thumb

Estimate selectivity per query: allowed rows divided by total rows.
Above ~50%, post-filter with a modest over-fetch is fine.
Between ~10% and 50%, use the index’s in-graph filter.
Below ~1%, or when the filter is a hard permission boundary, search the filtered subset exactly (or give each tenant its own index).
Always log how many results survive the filter. A silent drop from 10 to 1 is the failure to catch. A test that asserts len(results) == k on an independent filter would have caught it here.

For the basics of embeddings and indexes, see the free lesson Embeddings & vector databases.

Originally published on RecallRun. I run experiments like this every week at RecallRun. If you want to build RAG systems end to end, the AI Engineering course starts free: https://recallrun.dev/learn/ai-engineering/?utm_source=devto&utm_campaign=metadata-filter

Do you need a vector database? Brute-force search benchmarked from 10k to 1M vectors

RecallRun — Wed, 07 Oct 2026 11:01:14 +0000

Most RAG tutorials start by installing a vector database. For a lot of projects, that is a dependency you don't need yet. I measured how long plain exact search takes as the corpus grows, and what an HNSW index buys you in return.

Short answer: below about 100,000 chunks, a numpy matrix multiply is fast enough for almost any RAG app. Between 100k and 1M it depends on your traffic. Past that, use an index.

The setup

Every embedding is normalized, so cosine similarity is a dot product. Exact search is one matrix-vector product and a partial sort:

import numpy as np

def top_k(X: np.ndarray, q: np.ndarray, k: int = 10) -> np.ndarray:
    """X: (n, d) normalized float32 matrix. q: (d,) normalized query."""
    scores = X @ q
    idx = np.argpartition(-scores, k)[:k]
    return idx[np.argsort(-scores[idx])]

I timed this against faiss.IndexFlatIP (also exact) for three common embedding sizes: 384 (MiniLM-class models), 768 (BERT-base-class) and 1536 (OpenAI text-embedding-3-small default). The machine is deliberately small: 2 vCPUs, the size of a cheap cloud instance.

Exact search latency

Single-query latency, median of 50 queries:

Dim	Vectors	RAM for vectors	numpy	faiss Flat
384	10,000	15 MB	0.35 ms	0.78 ms
384	100,000	154 MB	3.5 ms	8.3 ms
384	500,000	768 MB	31 ms	66 ms
384	1,000,000	1.5 GB	62 ms	122 ms
768	100,000	307 MB	10.6 ms	22.6 ms
768	500,000	1.5 GB	57 ms	120 ms
768	1,000,000	3.1 GB	114 ms	240 ms
1536	10,000	61 MB	1.2 ms	3.1 ms
1536	100,000	614 MB	20 ms	51 ms
1536	500,000	3.1 GB	107 ms	248 ms

Three things stand out:

Latency scales linearly with n × d. It is a memory-bandwidth problem. Doubling either the corpus or the dimension doubles the time.
Memory is the real limit, not speed. 1M vectors at 1536 dims is 6 GB of float32 before any metadata. I skipped that row: it won't fit comfortably on a 7 GB machine.
Plain numpy beat faiss Flat for single queries here, by about 2×. Faiss is built for batches; one query at a time pays overhead that numpy's direct matrix-vector call doesn't. If you batch queries, test again before assuming this holds.

What an HNSW index buys you

Next, I built hnswlib indexes (M=16, ef_construction=200) and measured query time and recall@10: the fraction of the true top 10 the index actually returns.

I used two synthetic datasets, because recall depends heavily on how your data is shaped:

Uniform: random Gaussian directions. This is the worst case for any approximate index: no structure to exploit.
Clustered: 2,000 tight clusters. Real embeddings sit between these two, usually much closer to clustered, since documents about the same topic land near each other.

Data	Dim	Vectors	Build time	ef=32	ef=64	ef=128
clustered	384	100k	23 s	0.10 ms, recall 0.954	0.17 ms, 0.995	0.34 ms, 1.000
clustered	768	100k	48 s	0.22 ms, 0.878	0.36 ms, 0.985	0.68 ms, 1.000
uniform	384	100k	37 s	0.16 ms, 0.129	0.31 ms, 0.206	0.55 ms, 0.281
uniform	768	100k	69 s	0.34 ms, 0.058	0.63 ms, 0.105	1.07 ms, 0.172
uniform	384	500k	279 s	0.19 ms, 0.034	0.35 ms, 0.057	0.67 ms, 0.099

HNSW queries are 10 to 100 times faster than exact search at these sizes. The cost:

Build time. 500k vectors took over 4.5 minutes on 2 cores. You pay this again on every full re-index, such as after switching embedding models.
Recall is not guaranteed. On structured data, ef=64 gave near-perfect recall. On structureless data, the same settings returned less than a quarter of the true neighbors. Your data will be somewhere in between, so measure recall on your own embeddings before trusting the defaults.

Where the time actually goes in a RAG request

A typical RAG request spends 1 to 5 seconds waiting for the LLM to generate. Next to that:

3.5 ms (100k × 384, exact) is invisible.
60 to 110 ms (1M exact) is noticeable only if you care about p99, or serve many queries per second on the same box.
Exact search gives perfect recall for free, which removes one variable when you debug bad answers.

A rough sizing rule: one PDF page is usually 2 to 3 chunks. 100k chunks is roughly 30,000 to 50,000 pages. Many internal "chat with our docs" projects never get there.

My rule of thumb

Your corpus	What I'd use
under 100k chunks	numpy (or pgvector with no index). Store vectors in a `.npy` file or Postgres.
100k to 1M chunks	exact search is still fine at low QPS. Add HNSW when p95 latency or QPS demands it, and measure recall.
over 1M chunks, or many tenants	a proper ANN index (pgvector HNSW, Qdrant, Milvus, OpenSearch), plus filtering and sharding.

Most importantly, keep the vector store behind one interface so you can swap numpy for pgvector later without touching the rest of the pipeline. That one seam is worth more than picking the "right" database on day one.

Reproduce it

The exact-search benchmark:

import time, numpy as np, faiss
faiss.omp_set_num_threads(2)
rng = np.random.default_rng(0)
norm = lambda x: x / np.linalg.norm(x, axis=1, keepdims=True)
for d in (384, 768, 1536):
    for n in (10_000, 100_000, 500_000, 1_000_000):
        if n * d * 4 > 3.2e9: continue          # skip what won't fit in RAM
        X = norm(rng.standard_normal((n, d), dtype=np.float32))
        Q = norm(rng.standard_normal((50, d), dtype=np.float32))
        t = []
        for q in Q:
            s = time.perf_counter()
            sc = X @ q
            idx = np.argpartition(-sc, 10)[:10]
            t.append(time.perf_counter() - s)
        print(d, n, f"{np.median(t)*1e3:.2f} ms")

For HNSW, build with hnswlib.Index(space="ip", dim=d), init_index(max_elements=n, ef_construction=200, M=16), and compare knn_query results against the exact top 10 to get recall.

Caveat: these are synthetic vectors on one small machine. Latency for exact search doesn't depend on what the vectors mean, so those numbers transfer well. HNSW recall does depend on your data, which is exactly why you should measure it yourself.

Test setup: 2 vCPU Intel Xeon @ 2.1 GHz, 7 GB RAM, Python 3.11, numpy 2.4.4, faiss-cpu 1.15.1, hnswlib. Synthetic normalized float32 vectors. Median of 50 (exact) or 200 (HNSW) single queries.