DEV Community

Aleksandr Argunov
Aleksandr Argunov

Posted on Originally published at argbe.tech AI-assisted

A RAG answer can cite every sentence and still be wrong: a script to reproduce it

Every sentence was cited. The answer was still wrong.

Ask a company AI whether you can refund a customer after last month's exception. It finds the contract, quotes the 30-day rule and says no, with a citation. The exception sits in an email it never searched.

Here's that failure as a script you can run. It has no API key, no model and no embeddings. The refund is a made-up example.

I framed the post as “RAG in 2023 vs 2026”: search once versus check and search again. The dates are shorthand. Anthropic's Contextual Retrieval already combined hybrid search and reranking in 2024.

A poster compares searching once with a loop. One search answers no and misses the exception. The six-step loop finds the email on its second round and answers yes, with citations.

The script

Save this as rag_loop.py. It uses only Python's standard library, with SQLite's FTS5 enabled for BM25 search. Python 3.9 and 3.15 print the same output.

# Synthetic data. BM25 and the permission filter are real (SQLite FTS5).
# The check and the answers are keyword rules, so no model or API is needed.
import re
import sqlite3

MAX_ROUNDS = 3

DOCS = [  # id, source, who may read it, text
    ("contract-s4", "contract", "all", "Section 4 Refunds. Customers may request a refund within 30 days of purchase."),
    ("contract-s9", "contract", "all", "Section 9 Renewals. Renewal notice is due 30 days before the term ends."),
    ("ticket-118", "ticket", "all", "Acme asked for a refund on day 41. Waiting on finance. No decision yet."),
    ("email-1012", "email", "finance", "12 October. Finance director approved a one-time refund exception for Acme."),
]

db = sqlite3.connect(":memory:")
db.execute("CREATE VIRTUAL TABLE docs USING fts5(id UNINDEXED, source UNINDEXED, readers UNINDEXED, text)")
db.executemany("INSERT INTO docs VALUES (?, ?, ?, ?)", DOCS)


def search(words, source, role, k=2):
    # The permission filter sits inside the query, so every repeat search gets it too.
    sql = """SELECT id, text, readers FROM docs
             WHERE docs MATCH ? AND source = ? AND readers IN ('all', ?)
             ORDER BY bm25(docs) LIMIT ?"""
    hits = db.execute(sql, (" OR ".join(words), source, role, k)).fetchall()
    assert all(readers in ("all", role) for _, _, readers in hits), "permission leak"
    return hits


def proven(hits, proof):
    # shortcut: a keyword rule stands in for the model. Ask a model here in a real system.
    return next((doc_id for doc_id, text, _ in hits if all(w in text.lower() for w in proof)), None)


def run(title, customer, role, loop=True):
    print(f"\n{title}\n  customer={customer} role={role}")
    needs = {  # sources to try in order, search words, words that count as proof
        "rule": (["contract"], ["refund", "days"], ["refund", "30 days"]),
        "exception": (["ticket", "email", "drive"], ["refund", "exception", customer], ["approved", customer.lower()]),
    }
    queues = {need: list(spec[0]) for need, spec in needs.items()}
    if not loop:
        queues["exception"] = []  # The single search never looks for the exception.
    limit = MAX_ROUNDS if loop else 1
    evidence, seen, used = {}, set(), 0

    for used in range(1, limit + 1):
        for need, (_, words, proof) in needs.items():
            if need in evidence or not queues[need]:
                continue
            source = queues[need].pop(0)
            hits = search(words, source, role)
            seen.update(hit[0] for hit in hits)
            print(f"  round {used}: {need} -> {source}: {[hit[0] for hit in hits] or 'nothing'}")
            if found := proven(hits, proof):
                evidence[need] = found
        if len(evidence) == len(needs):
            break

    missing = [need for need in needs if need not in evidence]
    if not missing:
        reply = f"Yes. {customer} has an approved exception to the 30-day limit. [{evidence['rule']}] [{evidence['exception']}]"
    elif loop:
        reply = f"Partly. The limit is 30 days. [{evidence['rule']}] Could not confirm: {', '.join(missing)}."
    else:
        reply = f"No. Refunds end 30 days after purchase. [{evidence['rule']}]"
    cited = all(doc_id in seen for doc_id in re.findall(r"\[([\w-]+)\]", reply))
    print(f"  answer: {reply}")
    print(f"  cited: {'yes' if cited else 'no'}  sufficient: {'no' if missing else 'yes'}  rounds: {used}/{limit}")


run("A. Search once", "Acme", "finance", loop=False)
run("B. Loop", "Acme", "finance")
run("C. Loop, no permission to read that email", "Acme", "support")
run("D. Loop, no exception exists", "Globex", "finance")
Enter fullscreen mode Exit fullscreen mode

Run it:

python3 rag_loop.py
Enter fullscreen mode Exit fullscreen mode

You get this:

A. Search once
  customer=Acme role=finance
  round 1: rule -> contract: ['contract-s4', 'contract-s9']
  answer: No. Refunds end 30 days after purchase. [contract-s4]
  cited: yes  sufficient: no  rounds: 1/1

B. Loop
  customer=Acme role=finance
  round 1: rule -> contract: ['contract-s4', 'contract-s9']
  round 1: exception -> ticket: ['ticket-118']
  round 2: exception -> email: ['email-1012']
  answer: Yes. Acme has an approved exception to the 30-day limit. [contract-s4] [email-1012]
  cited: yes  sufficient: yes  rounds: 2/3

C. Loop, no permission to read that email
  customer=Acme role=support
  round 1: rule -> contract: ['contract-s4', 'contract-s9']
  round 1: exception -> ticket: ['ticket-118']
  round 2: exception -> email: nothing
  round 3: exception -> drive: nothing
  answer: Partly. The limit is 30 days. [contract-s4] Could not confirm: exception.
  cited: yes  sufficient: no  rounds: 3/3

D. Loop, no exception exists
  customer=Globex role=finance
  round 1: rule -> contract: ['contract-s4', 'contract-s9']
  round 1: exception -> ticket: ['ticket-118']
  round 2: exception -> email: ['email-1012']
  round 3: exception -> drive: nothing
  answer: Partly. The limit is 30 days. [contract-s4] Could not confirm: exception.
  cited: yes  sufficient: no  rounds: 3/3
Enter fullscreen mode Exit fullscreen mode

The search and permission filter execute in SQLite. Source order, sufficiency and answers are fixed rules. This reproduces control flow, not retrieval quality. cited: yes only checks that cited IDs appeared in retrieved results; it doesn't check whether they support the answer.

What each run tests

A: missing source. cited: yes and sufficient: no on the same line. A citation doesn't prove the answer is complete. A reranker won't help, because it scores the documents you give it. It can't add the missing email.

B: bounded repeat. The ticket mentions the refund but holds no decision. So the next round searches email, finds it and stops at round 2 of 3.

C: permission limit. The filter is inside the SQL, so the repeat search gets it too. The email never comes back, and the answer names what's missing. The role is a fixture; in production, derive it from authenticated identity. Enforce access at the search, and again when a source document is opened.

D: no answer exists. Search returns the Acme email for Globex. The check doesn't count it. The for loop ends at the cap, so the question can't spin forever. A real loop also needs time and token limits, enforced in code.

Now break it

Add this line to DOCS, after the approval email:

    ("email-1020", "email", "finance", "20 October. Finance director withdrew the Acme refund exception."),
Enter fullscreen mode Exit fullscreen mode

Run B now prints this:

  round 2: exception -> email: ['email-1020', 'email-1012']
  answer: Yes. Acme has an approved exception to the 30-day limit. [contract-s4] [email-1012]
  cited: yes  sufficient: yes  rounds: 2/3
Enter fullscreen mode Exit fullscreen mode

The withdrawal ranks first. The answer still says yes. Every sentence is cited, and the answer is wrong again.

The check only asks whether an approval exists. A real one also asks whether it applies, whether that person could approve it, and whether it still stands. So test on conflicting versions.

Where this stops

The script exercises keyword search, a deliberately weak sufficiency check, and a round cap. It doesn't test vector retrieval, reranking or exact-ID matching. The full design adds contextual chunks, query planning, hybrid search and reranking. Google's agentic RAG framework treats sufficient context as its own check.

For a real benchmark, keep the synthetic cases and replace the fixed routing, keyword checks and canned answers. Compare vector-only retrieval, hybrid, hybrid plus reranking, and a capped loop. Log how much required evidence reaches the top results, answer correctness, tokens and wall time.

I wouldn't build the whole loop on day one. A simple lookup should skip it. Start with hybrid search and a reranker, measure on real questions, and add planning or a second round only where that test set fails.

The maintained article, with all six steps and the models I'd benchmark for each, is on argbe.tech.

Top comments (0)