DEV Community: Salima Iddrisu

My RAG tool refused to answer a question my notes clearly answered

Salima Iddrisu — Sun, 25 Oct 2026 23:22:44 +0000

I built a retrieval-augmented generation tool over my own notes — no LangChain, no LlamaIndex, every stage by hand.

Then I asked it a question my notes definitely answer, and it said:

I can't find the answer to that in your notes.

The answer was there. I could point at the line.

Bug one: the passage arrived mid-word

The question was "why are input and output tokens priced differently?" My notes say:

Output tokens cost about 8x more than input tokens. $2.50 versus $0.30 per million.

Here is what retrieval handed the model:

y 8x more per token than
input**. The `/stats` command prints...

Read the first two characters. That is ...roughly 8x with its opening sliced into the previous chunk.

My 500-character cut had landed mid-word. Half the sentence in chunk N, half in chunk N+1, and neither half complete enough to match anything. I raised top-k to 10 assuming the good chunk ranked lower — it never appeared, because no chunk contained that sentence whole.

Nothing reported a problem. No exception, no warning. The fact was in my notes, unreachable, and the tool looked like it was working.

The fix is to stop treating chunk size as a hard boundary — scan backwards for somewhere sensible to cut:

def _soften(text, start, hard_end, overlap, lookback):
    # The cut must stay far enough from start that the next chunk still
    # advances past the overlap, or the loop never terminates.
    span = hard_end - start
    floor = min(start + max(overlap + 1, span - lookback), hard_end)
    if floor >= hard_end:
        return hard_end

    # Paragraph break, then sentence ending, then any whitespace.
    for marker in ("\n\n", ". ", "! ", "? ", " "):
        found = text.rfind(marker, floor, hard_end)
        if found != -1 and found + len(marker) <= hard_end:
            return found + len(marker)

    return hard_end

I found the infinite-loop risk in that floor line by feeding it a 2,000-character string with no whitespace at all.

Bug two: the same bug, backwards

So I tried bigger chunks. 900 characters. More context is better, surely.

A different question broke.

My notes answer "how did I keep my API key safe?" in four numbered steps under a heading of almost that name. It is eight lines long. At 900 characters those eight lines get absorbed into a chunk dominated by unrelated material, and the embedding becomes an average of all of it. It stopped matching.

Failure mode	What goes wrong
Chunks too small	a fact gets cut in half and survives nowhere
Chunks too large	a short section gets averaged into irrelevance

size / overlap	pricing question	key safety question
500 / 100	refused — mid-word fragment	refused
900 / 300	correct partial answer	refused — diluted
350 / 70	fine	answered

350 works because my notes are short sections with frequent headings, so ~350 characters is about one section. That is a property of my documents, not a universal constant.

The lesson is not the number. It is that chunk-count statistics cannot tell you this. Only real questions can.

The assumption: a threshold can detect an unanswerable question

Early on, three sentences on one topic scored 0.617 against each other. Three unrelated ones scored 0.031. A twenty-fold gap — so reject anything below a cut-off and you handle unanswerable questions.

Then I tried it for real.

I asked "what time does the corner shop close on Sundays?" Nothing in my notes about shops or opening hours. Best match: 0.3386 — it matched a chunk containing Sun Oct 4, because Sundays and Sun genuinely are related.

My weakest genuinely useful answer scored 0.3655.

A gap of 0.027.

I nearly set the floor at 0.35. Then I swept it across all four real questions instead of just my best example:

floor	Q1	Q2	Q3	Q4	unanswerable
off	5	5	5	5	5
0.30	5	3	5	5	2
0.35	5	2	2	4	0
0.40	5	0	0	1	0

At 0.35 the bad question is silenced and two real ones drop to two passages. At 0.40 they return nothing.

The scores are not calibrated across questions. A question worded like my notes scores high on everything; one phrased differently scores low on everything, even when its top hit is right. An absolute threshold compares numbers that were never on the same scale.

What works instead is the prompt — giving the model an acceptable way out:

3. If the passages do not contain the answer, say so plainly - for example:
   "I can't find the answer to that in your notes." Do not guess, do not fill
   gaps from general knowledge.

Told only "answer from the context", a model handed irrelevant passages still produces something, because refusing is not an option it has. Given explicit permission, refusal becomes the compliant response.

My relevance floor ships switched off.

Three takeaways

Chunk-count statistics are not quality measurements. I spent a day comparing chunk counts and mean lengths, then picked a value that failed on the second real question I asked.

Silent failures are the ones to design against. Every real problem here produced no error — the severed sentence, the diluted section, a mismatched embedding model returning confident nonsense.

Test the weakest case, not the strongest. My threshold looked safe until I swept it across questions that were not my best example.

Code and every measurement behind these decisions, including the ones I got wrong first: github.com/Iddrisusalima/ask-my-docs

If you have built something similar, how do you handle the unanswerable-question problem? The prompt fix works, but it feels like there is a retrieval-level signal I am missing.

Memory is just re-sending: what building a CLI chat tool taught me about LLM cost

Salima Iddrisu — Sun, 20 Sep 2026 17:29:44 +0000

I asked a chatbot my name. It told me. Ten seconds later it had no idea who I was.

Nothing about the model changed between those two questions. I deleted a list on my own laptop, and the memory went with it, because that list was the memory.

This post is about what I found building Prompt Lab, a CLI chat tool for Google's Gemini models that prints the tokens and cost of every reply. The code is on GitHub. I'm using the google-genai SDK and gemini-3.5-flash-lite.

Statelessness is the whole design constraint

Language models retain nothing between requests. No session, no history on the server. So "memory" cannot live in the model, and it doesn't. It lives in a list in your process, and it works by re-sending the entire conversation every time.

Which means memory and cost are the same thing. Every turn pays again for every turn before it.

Commands like /reset and /stats are intercepted before the network call, so they cost nothing. Everything else appends to history, then the system instruction plus the entire history goes to the API.

Building the request

The history is a list of types.Content, each carrying a role:

from google.genai import types

history: list[types.Content] = []

def add_turn(role: str, text: str) -> None:
    """role is "user" for my messages, "model" for the bot's replies."""
    history.append(
        types.Content(role=role, parts=[types.Part.from_text(text=text)])
    )

The first gotcha is here. Gemini has no role="system". OpenAI passes the system prompt as a message inside the list; Gemini passes it as configuration alongside the list:

def config() -> types.GenerateContentConfig:
    return types.GenerateContentConfig(system_instruction=system_prompt)

def send_once() -> tuple[str, types.GenerateContentResponseUsageMetadata]:
    response = client.models.generate_content(
        model=MODEL,
        contents=history,        # the whole conversation, every single time
        config=config(),
    )
    return (response.text or "").strip(), response.usage_metadata

That contents=history line is the entire memory mechanism. One turn looks like:

add_turn("user", user_input)      # 1. append what I said
reply, usage = send_once()        # 2. send the WHOLE history
add_turn("model", reply)          # 3. append what it said

Step 3 is the one that's easy to miss. Skip it and the model never sees its own previous answers, so it starts contradicting itself.

Reading the bill

Usage rides along on the response object:

usage = response.usage_metadata
usage.prompt_token_count       # input: my messages + system prompt + history
usage.candidates_token_count   # output: what the model generated
usage.total_token_count        # what you are actually billed on

Input and output are priced separately, so the cost function bills them at different rates. Google publishes per million tokens:

PRICING_PER_1M = {
    "gemini-3.5-flash-lite": (0.30, 2.50),   # (input, output) USD per 1M tokens
}

def cost_of(input_tokens: int, output_tokens: int) -> float:
    return (
        input_tokens / 1_000_000 * INPUT_PER_1M
        + output_tokens / 1_000_000 * OUTPUT_PER_1M
    )

Note the ratio: 2.50 / 0.30 means output costs roughly 8x more per token than input. The /stats command prints the running bill split by direction:

In that session, 74% of the tokens were input, which is history being re-sent. But look at the cost column: 156 output tokens cost $0.000390 while 451 input tokens cost only $0.000135. A quarter of the tokens, three quarters of the price. That gives you two independent levers, not one: trim history to cut input, and ask for brevity to cut output.

The measurement

The same four words, asked seconds apart, either side of a /reset:

You: what is my name
Bot: Your name is Salima!
  [turn 22]  history: 44 msgs  |  tokens  in: 822   out: 6

You: /reset
History cleared. Dropped 44 messages.

You: what is my name
Bot: I don't have access to your personal information, so I don't know your name
     yet! What should I call you?
  [turn 23]  history: 2 msgs   |  tokens  in: 29    out: 27

822 input tokens down to 29. Twenty-eight times cheaper for an identical question. The 822 wasn't my question. It was 44 messages of history being re-sent.

Across another session input went from 41 tokens on turn 1 to 654 by turn 18, while my typed messages stayed the same length and the replies got shorter. Growth is closer to quadratic than linear: turn 10 pays for turns 1 through 9 again.

Two more gotchas

`total_token_count` is not always input + output

I assumed it was addition. Testing gemini-3.6-flash, one request reported 4 input tokens, 2 output tokens, and a total of 62. The missing 56 were internal reasoning tokens, billed but never shown in the reply. Trust total_token_count, not your own arithmetic.

With streaming, usage arrives on the final chunk

Switching to generate_content_stream() doesn't break cost tracking, but the usage data isn't there from the start. Keep the last one you're handed:

for chunk in client.models.generate_content_stream(
    model=MODEL, contents=history, config=config()
):
    if chunk.text:
        print(chunk.text, end="", flush=True)
    if chunk.usage_metadata:
        usage = chunk.usage_metadata   # only the final chunk carries it

Streaming changes perceived speed, not price.

Failing without corrupting state

Two things mattered here. First, status codes need different handling, because retrying a rejected key is pointless while retrying a rate limit is sensible:

if code in (400, 401, 403):
    return "Your API key was rejected. Check GEMINI_API_KEY for typos."
if code == 404:
    return (
        f"The model '{MODEL}' was not found. Your key is fine - a 404 means "
        "auth succeeded and only the model name is wrong."
    )
if code == 429:
    return "Rate limit or quota exceeded. Wait a minute and try again."

That 404 branch exists because I wasted half an hour assuming my key was broken. A 404 means authentication succeeded.

I triggered each of these deliberately rather than coding for them and hoping. Here's an invalid key, reported and survived:

Two details in that output matter. The tool keeps running instead of exiting. And the session totals report zero turns and zero messages in history, which proves the rollback below actually fired.

Second, a failed request must not leave a half-finished conversation:

try:
    reply, usage = send_once()
except errors.APIError as err:
    history.pop()        # roll back the user turn we just appended
    print(explain_api_error(err))
    continue

Without that pop(), history ends on an unanswered user turn, which breaks the shape of the next request.

What I'd fix next

cost_of() takes input and output, which means it misses thinking tokens entirely. On a model that uses them, I'm under-reporting by roughly 10x. My code contradicts my own write-up on that point.

History is also never trimmed. It only grows, so a long enough conversation will stop fitting in the context window. The fix is dropping oldest turns in pairs to keep roles alternating, or summarising the old prefix into one synthetic turn.

Takeaway

Making a stateless model appear to remember means re-sending everything, and that is exactly what you pay for.

If you're building on an LLM, print your token counts once and watch them climb. It changes how you think about conversation length, system prompt size, and when to throw history away.

Code, architecture notes and a day-by-day build log: github.com/Iddrisusalima/prompt-lab