I built a retrieval-augmented generation tool over my own notes — no LangChain, no LlamaIndex, every stage by hand.
Then I asked it a question my notes definitely answer, and it said:
I can't find the answer to that in your notes.
The answer was there. I could point at the line.
Bug one: the passage arrived mid-word
The question was "why are input and output tokens priced differently?" My notes say:
Output tokens cost about 8x more than input tokens. $2.50 versus $0.30 per million.
Here is what retrieval handed the model:
y 8x more per token than
input**. The `/stats` command prints...
Read the first two characters. That is ...roughly 8x with its opening sliced into the previous chunk.
My 500-character cut had landed mid-word. Half the sentence in chunk N, half in chunk N+1, and neither half complete enough to match anything. I raised top-k to 10 assuming the good chunk ranked lower — it never appeared, because no chunk contained that sentence whole.
Nothing reported a problem. No exception, no warning. The fact was in my notes, unreachable, and the tool looked like it was working.
The fix is to stop treating chunk size as a hard boundary — scan backwards for somewhere sensible to cut:
def _soften(text, start, hard_end, overlap, lookback):
# The cut must stay far enough from start that the next chunk still
# advances past the overlap, or the loop never terminates.
span = hard_end - start
floor = min(start + max(overlap + 1, span - lookback), hard_end)
if floor >= hard_end:
return hard_end
# Paragraph break, then sentence ending, then any whitespace.
for marker in ("\n\n", ". ", "! ", "? ", " "):
found = text.rfind(marker, floor, hard_end)
if found != -1 and found + len(marker) <= hard_end:
return found + len(marker)
return hard_end
I found the infinite-loop risk in that floor line by feeding it a 2,000-character string with no whitespace at all.
Bug two: the same bug, backwards
So I tried bigger chunks. 900 characters. More context is better, surely.
A different question broke.
My notes answer "how did I keep my API key safe?" in four numbered steps under a heading of almost that name. It is eight lines long. At 900 characters those eight lines get absorbed into a chunk dominated by unrelated material, and the embedding becomes an average of all of it. It stopped matching.
| Failure mode | What goes wrong |
|---|---|
| Chunks too small | a fact gets cut in half and survives nowhere |
| Chunks too large | a short section gets averaged into irrelevance |
| size / overlap | pricing question | key safety question |
|---|---|---|
| 500 / 100 | refused — mid-word fragment | refused |
| 900 / 300 | correct partial answer | refused — diluted |
| 350 / 70 | fine | answered |
350 works because my notes are short sections with frequent headings, so ~350 characters is about one section. That is a property of my documents, not a universal constant.
The lesson is not the number. It is that chunk-count statistics cannot tell you this. Only real questions can.
The assumption: a threshold can detect an unanswerable question
Early on, three sentences on one topic scored 0.617 against each other. Three unrelated ones scored 0.031. A twenty-fold gap — so reject anything below a cut-off and you handle unanswerable questions.
Then I tried it for real.
I asked "what time does the corner shop close on Sundays?" Nothing in my notes about shops or opening hours. Best match: 0.3386 — it matched a chunk containing Sun Oct 4, because Sundays and Sun genuinely are related.
My weakest genuinely useful answer scored 0.3655.
A gap of 0.027.
I nearly set the floor at 0.35. Then I swept it across all four real questions instead of just my best example:
| floor | Q1 | Q2 | Q3 | Q4 | unanswerable |
|---|---|---|---|---|---|
| off | 5 | 5 | 5 | 5 | 5 |
| 0.30 | 5 | 3 | 5 | 5 | 2 |
| 0.35 | 5 | 2 | 2 | 4 | 0 |
| 0.40 | 5 | 0 | 0 | 1 | 0 |
At 0.35 the bad question is silenced and two real ones drop to two passages. At 0.40 they return nothing.
The scores are not calibrated across questions. A question worded like my notes scores high on everything; one phrased differently scores low on everything, even when its top hit is right. An absolute threshold compares numbers that were never on the same scale.
What works instead is the prompt — giving the model an acceptable way out:
3. If the passages do not contain the answer, say so plainly - for example:
"I can't find the answer to that in your notes." Do not guess, do not fill
gaps from general knowledge.
Told only "answer from the context", a model handed irrelevant passages still produces something, because refusing is not an option it has. Given explicit permission, refusal becomes the compliant response.
My relevance floor ships switched off.
Three takeaways
Chunk-count statistics are not quality measurements. I spent a day comparing chunk counts and mean lengths, then picked a value that failed on the second real question I asked.
Silent failures are the ones to design against. Every real problem here produced no error — the severed sentence, the diluted section, a mismatched embedding model returning confident nonsense.
Test the weakest case, not the strongest. My threshold looked safe until I swept it across questions that were not my best example.
Code and every measurement behind these decisions, including the ones I got wrong first: github.com/Iddrisusalima/ask-my-docs
If you have built something similar, how do you handle the unanswerable-question problem? The prompt fix works, but it feels like there is a retrieval-level signal I am missing.
Top comments (3)
Raising top-k never finding the cut sentence is such a good detail, because it's exactly the moment you'd conclude retrieval is fine and go blame the prompt.
Since your notes are short sections with frequent headings, I'd try splitting on the headings first and only falling back to a character size for sections that are actually long. Then 350 stops being a number tuned to your current notes and just becomes "one section", and the key safety question can't get diluted by whatever happens to sit next to it. Putting the heading text at the top of each chunk before embedding helped me a lot too, since questions tend to be worded like headings.
On the unanswerable question, the thing that worked better for me than an absolute floor was looking at the shape of the scores for that one question. A real hit usually stands apart from the rest of its own top 10, while the corner shop question gives you a flat list where the best match barely beats the tenth. Comparing the top score to the median of that same question's top 10 sidesteps the calibration problem, because you're never comparing numbers across questions. Adding plain keyword search next to the embeddings also helps here: "corner shop" and "Sundays" match zero words in your notes, which is a much louder signal than 0.3386 versus 0.3655.
Are you keeping the questions it refused as a test set? Rerunning those every time you change the chunk size would have caught the 900 character regression straight away.
The mid-word cut is such a sneaky failure because top-k can't fix it, which you showed nicely by raising it to 10. The backwards scan for a paragraph or sentence break is what I'd reach for too. For the second bug, one thing that's helped me is splitting on headings first and only then applying the size limit, so a short section like your four API-key steps stays its own chunk with its heading attached. Did you end up setting a minimum chunk size, or let short sections stand alone?
Hi Salima. Mid-word chunk splits are the sneaky part, they silently eat the one sentence that answers the question and nothing flags it. I've seen the same with overlap windows that cut inside a sentence instead of on its boundary, and cosine thresholds don't hold across differently-worded queries like you found. If you want, send a few failing chunks and I'll check where the cuts land against the sentence structure.