DEV Community

Cover image for My RAG tool refused to answer a question my notes clearly answered

My RAG tool refused to answer a question my notes clearly answered

Salima Iddrisu on October 25, 2026

I built a retrieval-augmented generation tool over my own notes — no LangChain, no LlamaIndex, every stage by hand. Then I asked it a question my ...
Collapse
 
agentsearchhq profile image
AgentSearch •

The mid-word cut is such a sneaky failure because top-k can't fix it, which you showed nicely by raising it to 10. The backwards scan for a paragraph or sentence break is what I'd reach for too. For the second bug, one thing that's helped me is splitting on headings first and only then applying the size limit, so a short section like your four API-key steps stays its own chunk with its heading attached. Did you end up setting a minimum chunk size, or let short sections stand alone?

Collapse
 
salima_iddrisu profile image
Salima Iddrisu •

Neither, and that's the actual answer to the second bug. There's no minimum chunk size and no heading split, it's character windows with a backwards scan for a boundary, so a short section doesn't stand alone, it gets absorbed into whatever window it lands in. That's precisely how the four API-key steps got diluted at 900.

I did end up building a minimum-size mechanism, but only for PDFs: consecutive short pages get merged until they approach the chunk size, and the citation becomes a page range. CSV rows below 12 characters get dropped. The markdown case, the one that actually broke, still has nothing. Splitting on headings first is what I'm doing next, and attaching the heading to the chunk is the part I hadn't considered.

Collapse
 
agentsearchhq profile image
AgentSearch •

That explains it: the 900-char window just swallowed the short section. Merging consecutive short PDF pages is a nice fix. For markdown notes, a cheap middle ground is letting the backwards scan prefer a heading as the cut point, so a short section starts its own chunk with its heading attached, and you can still merge it forward if it's tiny.

Collapse
 
recallrundev profile image
RecallRun •

The 0.3386 vs 0.3655 gap matches what I saw when I ran a threshold on a bigger set (FastAPI docs, 30 answerable and 30 unanswerable questions). Questions that had nothing to do with the docs were easy: all of them scored below the weakest real question. The trouble was "near-domain" questions like yours, where Sunday matches Sun. Those overlapped the real ones, and any cut-off that blocked most of them also dropped a real question or two.

So I'd agree with shipping the floor off at four questions. If you want to try it again later, the thing that made it workable for me was writing 20-30 of each kind first and setting the floor just under the lowest real question, then putting that check in a test so a change of embedding model or chunk size fails loudly instead of silently moving the scale. The prompt-level "you may say you can't find it" still does the heavy lifting either way.

Collapse
 
salima_iddrisu profile image
Salima Iddrisu •

My question set was too kind, which I only saw from your near-domain split. I tested twelve questions, eight the notes answer, four they don't and the absolute floor looked like it separated cleanly, a gap of 0.27. But three of the four unanswerable ones were Peru, a radiator and a football league. Far-domain, the easy case. The only near-domain one scored highest of the four. So that was my questions being generous, not the floor working.

The floor stays off. What I'm taking from your second point is the objective, though: set it just under the lowest real question rather than trying to block the bad ones, and accept it catches less. I had it backwards ,I was looking for a cut-off that silenced the unanswerable question, which is why every value I tried cost a real answer.

The test is the part that generalises beyond the floor. Any number I tuned by hand moves silently when the embedding model or the chunk size changes, and I'd never know. That's the same failure as the 900-character regression, which I also found by accident rather than by a check.

Collapse
 
recallrundev profile image
RecallRun •

Near-domain negatives are cheap to make, which helps with exactly this. Take a question your notes do answer and change one thing the notes don't cover: a different version, a feature they never mention, a neighbouring tool. Those score close to the real ones, so they show you how thin the gap actually is. Far-domain ones (Peru, the radiator) will almost always look fine.

For the test I'd assert two numbers rather than one: the lowest score among the answerable questions stays above the floor plus a small margin, and the gap between that and the highest near-domain negative gets printed every run even if nothing fails. The first catches the regression. The second tells you when the floor has quietly stopped meaning anything after a model or chunk-size change.

Collapse
 
makeyouragent profile image
MakeYourAgent •

Raising top-k never finding the cut sentence is such a good detail, because it's exactly the moment you'd conclude retrieval is fine and go blame the prompt.

Since your notes are short sections with frequent headings, I'd try splitting on the headings first and only falling back to a character size for sections that are actually long. Then 350 stops being a number tuned to your current notes and just becomes "one section", and the key safety question can't get diluted by whatever happens to sit next to it. Putting the heading text at the top of each chunk before embedding helped me a lot too, since questions tend to be worded like headings.

On the unanswerable question, the thing that worked better for me than an absolute floor was looking at the shape of the scores for that one question. A real hit usually stands apart from the rest of its own top 10, while the corner shop question gives you a flat list where the best match barely beats the tenth. Comparing the top score to the median of that same question's top 10 sidesteps the calibration problem, because you're never comparing numbers across questions. Adding plain keyword search next to the embeddings also helps here: "corner shop" and "Sundays" match zero words in your notes, which is a much louder signal than 0.3386 versus 0.3655.

Are you keeping the questions it refused as a test set? Rerunning those every time you change the chunk size would have caught the 900 character regression straight away.

Collapse
 
salima_iddrisu profile image
Salima Iddrisu •

The heading split is the right instinct and I think it fixes the cause rather than the symptom. 350 was always a number fitted to the shape of my current notes, and "one section" survives a corpus I haven't seen yet. Prefixing the heading before embedding is the part I hadn't thought of my questions really are worded like headings, so that should help the key safety question find itself.

I tried your score-shape idea against twelve questions, eight answerable and four not, and it didn't separate them. Top minus median: weakest real question 0.0517, best unanswerable 0.0812. The ratio was worse ,1.12 against 1.83.

Two rows explain it. "What did a relevance floor of 0.40 do" is a real question and comes back almost flat, ratio 1.12, because the answer is in a CSV and all five rows are near-identical in structure, so they all score alike. Meanwhile "who won the league last season" is unanswerable and has the spikiest distribution in the whole set at 1.83. So flatness seems to track how uniform that patch of corpus is, not whether the answer is in it.

Your keyword point held up best of everything I measured. Every answerable question had all of its uncommon words somewhere in the notes; the unanswerable ones ranged from none to 40%. That was a wider gap than the similarity scores gave me.

And no, I wasn't keeping the refused questions. That's the cheapest thing on this list and it would have caught the 900-character regression the same day.

Collapse
 
hannune profile image
Tae Kim •

Hi Salima. Mid-word chunk splits are the sneaky part, they silently eat the one sentence that answers the question and nothing flags it. I've seen the same with overlap windows that cut inside a sentence instead of on its boundary, and cosine thresholds don't hold across differently-worded queries like you found. If you want, send a few failing chunks and I'll check where the cuts land against the sentence structure.

Collapse
 
salima_iddrisu profile image
Salima Iddrisu •

Thank you, I'll take that up. The one that started it was a sentence about token pricing where the 500-character boundary landed mid-word, so retrieval handed over "y 8x more per token than" and the model correctly refused. The overlap was 100, which should have carried it, and didn't.

What I don't understand yet is why the overlap didn't rescue it. By my reading the tail of one chunk is repeated at the head of the next, so a sentence shorter than the overlap should survive whole somewhere. Either my reasoning or my implementation is wrong, and I'd rather know which.

Collapse
 
izgorodin profile image
Edward Izgorodin •

The sentence the tool gave when the passage was cut mid-word is, word for word, the sentence the prompt asks for when the notes have nothing. With the floor off and the prompt doing the work, that refusal is both the correct answer to the corner shop question and the symptom of the two chunking bugs in the post, and the output cannot tell the two apart. It is the silent failure from your takeaways, moved into the one place that looks deliberate.

That makes two counts worth keeping apart: refusals on questions the notes do answer, and answers on questions they do not. The chunk table is already scored by outcome, but the floor sweep is scored in passages, and two passages for Q2 and Q3 at 0.35 do not say whether those two still held the answer.

For the retrieval-level signal, the keyword idea in the thread can also split the refusals. When the tool refuses, look for the question's less common words in the whole notes and in the passages that were sent. Absent from both is the corner shop. Present in the notes but missing from what was sent is a suspected miss, worth logging with those passages, and that is usually where a cut or diluted chunk shows up.

One note on the 0.617 and 0.031, which also matters for the labeled set suggested in the thread. Those were sentences scored against sentences, while retrieval scores a short question against a passage written in a different voice, so the gap a floor had to cut was never the twenty-fold one. The 0.027 is what the right kind of pair gives, so a labeled set for any later cut-off, absolute or relative to the question's own top ten, should be built from question and passage pairs.

Collapse
 
salima_iddrisu profile image
Salima Iddrisu •

You've found the thing I missed, and it's worse than I framed it. The refusal sentence is identical in both cases, notes that genuinely don't cover the question, and notes that do but got cut into fragments. I wrote that silent failures are the ones to design against, then put one in the place that looks most deliberate. I hadn't seen that until you said it.

On the floor sweep, you're right that it's scored in passages rather than outcomes. Two passages at 0.35 for Q2 and Q3 says how much got filtered, not whether the answer survived the filtering. The chunk table has outcomes. That one should too.

The keyword check is what I was fishing for at the end of the post. Take the question's uncommon terms, look for them in the whole corpus and in what was actually sent: absent from both is a real refusal, present in the notes but missing from the passages is a suspected miss. It reuses the baseline I'd written off as useless, as a tripwire rather than a ranker. That's the next thing I'm building.

And the 0.617 against 0.031 point is fair. Sentence against sentence isn't question against passage, so 0.027 was the real number all along.