DEV Community: Klaus Byskov Pedersen

Retrieval confidence can't tell your RAG chatbot when the answer is missing

Klaus Byskov Pedersen — Thu, 08 Oct 2026 07:38:21 +0000

I ran 65 questions against our own knowledge base and checked which ones the retrieved text actually answered. The similarity scores overlapped too much for any threshold to work.

Our own handoff docs used to describe it: if retrieval confidence falls below a threshold, hand the visitor to a human instead of generating a weak answer. The default was 0.4. It's the design most people building a RAG chatbot reach for, and I no longer think it works. This post is about why: first a hunch from production, then a small experiment.

What retrieval confidence is

In a typical RAG pipeline, including ours at Asktopus, the visitor's question is embedded (we use OpenAI's text-embedding-3-small). The vector database (Qdrant) returns the five closest chunks by cosine similarity, and those chunks go into the prompt. The best of those five scores is what usually gets called "retrieval confidence".

It's free, already computed and between 0 and 1, so it looks like the system telling you how sure it is.

The tempting design

if confidence < THRESHOLD:
    hand_off_to_human()      # or reply "I don't know"
else:
    answer_from_context()

It promises two things at once: no made-up answers when the knowledge is thin, and a human for the questions the bot can't handle. All you have to do is pick the number.

What first made me doubt it

In production, the scores of 101 real answers on one site ranged from 0.11 to 0.67, with a median of 0.36. At the 0.4 our docs recommended, more than half of them would have gone to a human. I couldn't tell from the numbers which of those answers were good, and 101 answers on one site is far too few to tune a threshold on. So I set up a test where I could check.

The experiment

The knowledge base is asktopus.com itself: 44 crawled pages in English and Danish, split into 219 chunks of up to 1,500 characters. The pipeline is exactly production's: same embedding model, same Qdrant search, top 5, confidence = the best score.

I wrote 65 questions: 40 I expected the site to answer, 20 I expected it not to (a Zapier integration, ISO 27001, an uptime SLA), and 5 plainly off-topic (the capital of France, tomorrow's weather). Then I set my expectations aside and labelled each question by reading the five chunks the model would receive. Does this text contain the answer? Yes, partial (it answers a narrower version: conversations can be exported, but the format isn't stated), or no.

My expectations were off. Of the 40 questions I expected the site to answer, only 24 got retrieved text that answered them; 6 were partial and 10 got nothing useful. Three of those ten were retrieval misses: the answer was in the knowledge base, just not in the top five. All 25 questions I expected it not to answer got a no.

Results

Label	Questions	Min	Median	Max
Yes	24	0.309	0.499	0.708
Partial	6	0.331	0.379	0.582
No	35	0.102	0.345	0.590

The score isn't noise: answerable questions score higher on average. But the two groups overlap from 0.31 to 0.59, and that band holds 19 of the 24 answerable questions, 23 of the 35 unanswerable ones and all six partials. Clean answers exist only at the edges. The 12 questions below 0.31 were all unanswerable, four of them off-topic. The five above 0.59 were all answerable.

Here's what each threshold would do if you hand off whenever confidence is below it:

Threshold	Answerable, handed off (of 24)	Unanswerable, let through (of 35)
0.30	0	25
0.35	1	17
0.40	2	14
0.45	7	9
0.50	12	5

There's no good row. At 0.40, 14 questions the retrieved text doesn't answer still go through, and 4 of the 6 partial ones are handed off. At 0.50, half of the answerable questions go to a human. The best single cut on this data, found after the fact, is about 0.43, and it still gets 11 of 59 wrong. I found it by looking at labels you won't have in production, and it doesn't travel: at 0.43, more than half of those 101 production answers would have gone to a human.

Two questions show the problem better than the tables.

The highest-scoring "no": "How many employees does Asktopus have?" scored 0.590. The site doesn't say. The top chunk was the features text about inviting colleagues as editors or viewers, which is about the customer's team, not ours. That score beats 19 of the 24 answerable questions.

The lowest-scoring "yes": "How long does it take to set up?" scored 0.309. The best match was a homepage chunk that ends with the FAQ question "How fast can I get started?". The answer to it falls in the next chunk, which wasn't retrieved. The answer came from the installation guide in second place: "up and running on your website in under 5 minutes". 22 of the 30 on-topic unanswerable questions scored higher.

Why this happens

An embedding captures what a text is about. Cosine similarity tells you the question and a chunk are about the same thing, not that the chunk contains the fact the question asks for. "How many employees does Asktopus have?" is about Asktopus and teams, and so is the features page. That's all the score can see.

The clearest pattern in the data: 30 of the unanswerable questions were on-topic. The 14 of them that use the product's own words ("Asktopus", "chatbot", "assistant", "widget", "AI") had a median score of 0.498, the same as the answerable questions. The 16 that don't had a median of 0.307. On a site about AI chatbots, any question about AI chatbots is close to everything.

What's inside the chunks makes it worse in both directions:

Chunks that mix many topics score moderately for many questions. In this crawl, navigation, page headers and footers ended up in the chunks, and the docs navigation lists the title of every docs page. "Can I remove the Asktopus branding from the chat?" scored 0.507 with five chunks that were all page headers and navigation. The line that answers it, "Remove branding" under the Pro plan, was in the knowledge base but wasn't retrieved.
The same mixing dilutes real answers. "Can I upload PDF documents?" scored 0.364 although the top chunk answers it almost word for word. That chunk is a homepage FAQ that also covers setup time, wrong answers, languages and GDPR.
Short questions carry little signal. In a five- or six-word question, one shared word like the product name carries a lot of weight.

Limitations

This is a small test. One small site, one crawl (partly out of date), 65 questions I wrote myself in English, labelled by my own judgement. Different content, languages or chunking would move the numbers. I also measured retrieval, not answers: I didn't test how reliably the model says "I don't know" when the chunks don't contain the answer. But nothing in the mechanism is specific to this site, and the production numbers point the same way.

What we do instead

In Asktopus, the score doesn't make any decision about the conversation.

The model decides whether the answer is there, and says so in a way the code can read. The prompt tells it to answer only from the retrieved context. When the context doesn't contain the answer, it starts its reply with a fixed marker, which we strip from the stream before the visitor sees it. The model reads the chunks; the score only measures how close they are.
That signal, not a score, offers a person. Under an answer the assistant marked as missing, the widget shows a "Talk to a person" button. The same button is always in the header, and explicit requests like "talk to a human" open the same contact form.
Visitors rate answers. A 👍 or 👎 under every answer is a second signal that costs nothing to collect.
Missing answers go on a list to review, never trigger anything else. The dashboard's "Questions to answer" list shows questions whose newest asking the model marked as missing, a visitor rated 👎, or scored very low (below 0.25). The site owner can write an article that answers one, or hide it, and a question drops off by itself once its next asking gets a real answer. In this experiment, the 0.25 cut alone flagged 6 questions, all of them unanswered, and missed the other 29. That's fine as one input to a review list and useless as a gate.

In a first live check on our own site, the marker flagged all three questions the site doesn't cover (one of them in Danish) and neither a pricing question it does cover nor a plain "Hi there!". Five questions prove nothing statistically, but the signal comes from something that reads the text, which is the point.

We also stopped using confidence to pick the model. We used to send low-confidence questions to the bigger one, but a bigger model can't answer from content it wasn't given.

Takeaways if you're building RAG

Don't gate handoffs or "I don't know" on a similarity threshold. Answerable and unanswerable questions overlap across exactly the range where you'd put it.
Let the model judge whether the retrieved text answers the question, and have it say so in a form your code can read, such as a marker you strip from the stream. It reads the text; the score doesn't.
Give people an explicit way to reach a person. Asking is a better signal than any score.
Use low scores as a weak signal for a person to review, where a miss costs nothing.
Label a few dozen of your own questions before you trust any threshold. An afternoon of reading top-5 chunks tells you more than the scores do.
Keep navigation and footers out of your chunks. In our crawl they matched questions they couldn't answer. We now drop lines that repeat across most of a site's pages before chunking: on our site that removed 27% of the text, and the top-5 results that were mostly menus and footers fell from 69 of 325 to 3.

Asktopus, the chatbot this came out of, is at asktopus.com.

I built a harvest tripwire, then almost let an LLM decide who looks legit

Klaus Byskov Pedersen — Mon, 05 Oct 2026 20:32:15 +0000

Free API tiers get farmed. That is not news. What surprised me was how much the farming looked like real evaluation, and how badly a model did at telling the two apart.

This is a Companydata story: Danish company registry data over HTTP, a free monthly quota, and a tripwire that throttles keys that suddenly look up hundreds of different companies in an hour.

If you run a free or freemium API, I want your war stories in the comments. I am still deciding whether to pool free quota by IP after a burn. How did you solve the "looks like a tester, is actually rotation" problem without nuking real evaluators?

The symptom that felt wrong

My harvest tripwire was doing its job: when a free key tore through a wide set of company lookups in a short window, I auto-flagged it and slowed it to one request per minute. Alerts came into Telegram. Several looked like ordinary builders testing an integration. I vouched a few of them by hand.

That felt expensive. Every false positive is a conversion you might have killed with politeness. So the tempting next step showed up fast: put a model on the alert and ask, "does this look like a legit user?"

I had already been evaluating Jev (TypeSafe System One) in another project. Identity-only signals, a handful of typed questions, cheap tokens. Perfect candidate for a quick legitimacy gate. Or so it seemed.

What the data actually showed

Three "testers" in a couple of days were not three testers. Same egress network, same curl client, same habit of burning the free quota to the exact limit, then opening a fresh account. Real people, real company, still 1,500 free calls in 48 hours by one client.

Quota emails were opened. Pricing was known. A free re-signup was cheaper than a small credit pack. Information was not the bottleneck. Friction on the free path was.

That finding mattered more than any model score. The tripwire was not wrong about the pattern. The human vouching step was wrong about independence of accounts.

I ran Jev anyway

I still ran the experiment. Twenty-seven accounts that already had an abuse field: seven I had vouched as legit, twenty from an earlier abusive ring. Identity-only state into Jev.

Jev found every vouched account when I asked for a hard "legit" verdict. It also waved through several of the ring. Probability scores sat in a mushy band for both classes. No useful threshold. When I added behaviour, Jev flipped most of the vouched accounts to abusive, which, given the rotating-quota story, was closer to the truth and also useless as a gate for "please don't annoy real customers."

A dumb rule score on identity (Google signup, non-freemail domain, name shape, and so on) separated the old ring from clean-looking identities, and would have waved the rotating-quota trio straight through. Identity is context for a human on-call, not a gate.

Decision: Jev stays out of the tripwire. Same conclusion I reached with Jev on another product evaluation earlier that month.

What I shipped instead

Two boring, high-leverage changes.

Related accounts on the admin alert. When a key trips, the Telegram message now lists other accounts that share the client address or the exact display name, with usage and state. Admin group only. Nothing about other users ever goes into a user-facing response. The first live run immediately taught me to exclude my own frontend service identity, which had been logging anonymous page views under the visitor's address and therefore headed every related list.

An honest 429. A flagged key used to get the same "Rate limit exceeded" body as an ordinary minute limit. Scripts backed off and nobody wrote in. Flagged keys now send X-RateLimit-Reason: under-review and a body that says I am reviewing a free key that looked up an unusual number of companies, that nothing is blocked forever, and how to buy credits or mail support. Ordinary minute-limit responses are unchanged.

I deliberately did not auto-withhold a new free quota from an address that already exhausted one this month. That is a product call I want to make after living with the better alerts.

What I would tell someone building the same thing

Harvest detection on breadth and velocity works. The failure mode is the human story you tell yourself about who is behind the key.
Account linking for operators beats clever legitimacy scoring for users.
If you throttle someone, say so in the response. Silent identical 429s train both scrapers and real integrators to shrug.
Try the model if you want the data. I named Jev here because people ask. For this problem it did not earn a place in the critical path.

Your turn

If you have shipped something similar (Stripe-style fingerprinting, shared free pools, device attestation, payment before second account, soft CAPTCHA on signup bursts, or something weirder), drop it below. Especially useful:

What signal actually caught rotation without false-flagging agencies and shared offices?
Did you ever put an LLM on abuse triage, and did it earn its keep?
How do you talk to a real evaluator who tripped the wire without sounding like a bank fraud team?

Companydata is live at companydata.dk. The API docs cover the rate-limit header. If you are evaluating and hit the review throttle for real work, mail support and I clear it by hand.