DEV Community

Cover image for Clear the Lineup: the passage my RAG retrieved and then threw away in silence
Vinicius Pereira
Vinicius Pereira

Posted on

Clear the Lineup: the passage my RAG retrieved and then threw away in silence

#devchallenge #bugsmash #sentry #googleaichallenge

This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.

Project Overview

rag-quality is a small, self-contained RAG pipeline: it retrieves passages from a corpus (sparse, dense, and hybrid modes), then generates a grounded answer that cites the passages it used and abstains when the context does not support one. It ships with an eval harness that scores retrieval quality.

Recently I commented on a dev.to thread about non-deterministic retrieval, the ways a RAG can fetch the right passage and still answer wrong. In reply, the author added one more check to his list: verify the context you asked for is the context you actually got. That check named the exact hole I had in this repo, one stage past retrieval, in assembly, against a bug I had shipped and not noticed.

Bug Fix or Performance Improvement

The generation step assembles retrieved passages into a context block and sends it to the model. Here is what it did:

def build_prompt(question: str, hits: list[Hit]) -> str:
    blocks = [f"[{h.chunk.id}] {h.chunk.text}" for h in hits]
    context = "\n\n".join(blocks)[: config.MAX_CONTEXT_CHARS]
    return f"Context passages:\n{context}\n\nQuestion: {question}\n\nAnswer:"
Enter fullscreen mode Exit fullscreen mode

MAX_CONTEXT_CHARS is 6000. It joins every retrieved passage, then hard-slices the string to 6000 characters. If the joined context runs longer, the tail is gone. The tail is the lowest-ranked hits, and the lowest-ranked hit is often the one that actually holds the answer: retrieval surfaced it, ranking put it last, the slice deleted it. No exception, no log, no return value that hints anything was cut. The caller gets a shorter prompt and believes it is complete.

What makes this one sting is the asymmetry a few lines down in the same file:

if message.stop_reason == "max_tokens":
    text += "\n[warning: answer truncated at the token limit]"
Enter fullscreen mode Exit fullscreen mode

The code already knows truncation is dangerous. It shouts when the model's output is cut. It stays silent when the model's input is cut, which is the more dangerous of the two, because a truncated input can quietly remove the one passage the answer depended on and the model will confidently answer from what is left.

And the tests stayed green because nothing exercised the failure. The sample corpus is 17 short documents; its joined top-k context fits comfortably under 6000 characters, so the slice never removed anything in the bundled runs. The eval harness measures retrieval quality (hit@k, MRR, recall@k), whether the right passage was retrieved, not whether it survived assembly into the prompt. The bug lives in the gap between those two stages, which is precisely the gap no metric was watching.

Code

The whole change, the fix, the tests, and the optional Sentry and Gemini tooling, is one pull request:

Fix silent context truncation in build_prompt #1

vinimabreu avatar
vinimabreu posted on

build_prompt joined every retrieved passage and hard-sliced the result to MAX_CONTEXT_CHARS, dropping the tail with no exception, log, or return signal. The tail is the lowest-ranked hits, and the lowest-ranked hit can be the one that holds the answer. The same file already warns on output truncation but stayed silent on the input side.

This packs whole passages in rank order up to the budget, never cuts a passage mid-text, and returns dropped_ids so the caller announces what was left out, mirroring the existing output-truncation warning.

Also adds:

  • env-gated Sentry spans across the pipeline (no-op without sentry-sdk and SENTRY_DSN), so an over-budget request shows total_context_chars, dropped_ids, and a context.truncated_chars measurement on the rag.build_prompt span
  • scripts/root_cause.py, an optional Gemini helper that localizes the loss from a trace plus the source
  • five tests that run key-free and offline
View on GitHub

My Improvements

Stop slicing a string. Pack whole passages in rank order until the next one would blow the budget, never cut one mid-text, and return the ids you dropped so the caller can announce them.

-def build_prompt(question: str, hits: list[Hit]) -> str:
-    blocks = [f"[{h.chunk.id}] {h.chunk.text}" for h in hits]
-    context = "\n\n".join(blocks)[: config.MAX_CONTEXT_CHARS]
-    return f"Context passages:\n{context}\n\nQuestion: {question}\n\nAnswer:"
+def build_prompt(question: str, hits: list[Hit]) -> tuple[str, list[str]]:
+    separator = "\n\n"
+    budget = config.MAX_CONTEXT_CHARS
+    blocks = [f"[{hit.chunk.id}] {hit.chunk.text}" for hit in hits]
+
+    kept, dropped_ids, used = [], [], 0
+    for hit, block in zip(hits, blocks):
+        extra = len(block) + (len(separator) if kept else 0)
+        if used + extra <= budget:
+            kept.append(block)
+            used += extra
+        else:
+            dropped_ids.append(hit.chunk.id)
+
+    if dropped_ids:
+        log.warning("context truncated: %d passage(s) dropped: %s",
+                    len(dropped_ids), ", ".join(dropped_ids))
+
+    context = separator.join(kept)
+    prompt = f"Context passages:\n{context}\n\nQuestion: {question}\n\nAnswer:"
+    return prompt, dropped_ids
Enter fullscreen mode Exit fullscreen mode

That diff is the essence of the change. The shipped version also opens the rag.build_prompt span from the Sentry section around this same code, which is how those numbers reach the trace.

Generator.answer now unpacks the tuple and appends a note that mirrors the existing output warning, so the drop is visible in the answer itself:

if dropped_ids:
    note = ", ".join(dropped_ids)
    text += f"\n[warning: {len(dropped_ids)} passage(s) dropped from context: {note}]"
Enter fullscreen mode Exit fullscreen mode

The budget stays in characters here on purpose. The stronger version counts the real token budget of the target model; I noted that in the docstring rather than building it, because the point of this fix is the missing signal, not a better ruler.

Proof. Same input in both runs: four bulky decoy passages plus a short answer chunk that is retrieved but ranked last, so the joined context is 7343 characters against a 6000 budget. Real console output, captured with the repo's own interpreter, no API key.

Before, the answer chunk is gone and nothing says so:

MAX_CONTEXT_CHARS      = 6000
full joined context len= 7343
answer chunk id        = expenses::answer
answer text present in FULL joined context? True
answer text present in BUILT prompt?        False
answer sentence in built prompt?            False
------------------------------------------------------------
RESULT: answer chunk SILENTLY DROPPED. No exception, no warning, no log. Caller gets a prompt missing the answer.
Enter fullscreen mode Exit fullscreen mode

After, packing skips the one passage that will not fit, keeps the small answer chunk behind it, and reports the id it dropped:

context truncated: 1 passage(s) dropped to fit 6000-char budget: expenses::3
MAX_CONTEXT_CHARS      = 6000
full joined context len= 7343
answer chunk id        = expenses::answer
dropped_ids reported   = ['expenses::3']
answer text present in BUILT prompt?        True
answer sentence in built prompt?            True
------------------------------------------------------------
  expenses::0        -> KEPT (verbatim)
  expenses::1        -> KEPT (verbatim)
  expenses::2        -> KEPT (verbatim)
  expenses::3        -> dropped whole
  expenses::answer   -> KEPT (verbatim)
------------------------------------------------------------
RESULT: answer PACKED IN, and the passage that did not fit (['expenses::3']) is reported to the caller. No silent loss.
Enter fullscreen mode Exit fullscreen mode

Four behavioural tests lock the contract: an over-budget answer chunk is either packed in or named in dropped_ids and never silently gone, over-budget context reports its dropped ids, no passage is ever cut mid-text, and within-budget context drops nothing. A fifth pins the tuple return type so a caller cannot regress to the old string signature. All five run with the repo's own interpreter, no key and no network.

The rule I kept: any place that decides what the model sees has to announce what it dropped. A retriever that finds the right passage and an assembler that quietly deletes it produce the same wrong answer, but only one of them leaves a trace. The output truncation in this file already knew that. The input side just needed to learn the same manners.

Best Use of Sentry

The fix for "silent" is instrumentation. I added env-gated Sentry spans across the pipeline (off unless SENTRY_DSN is set and sentry-sdk is installed, so the repo never hard-depends on it). The span that matters here is rag.build_prompt:

  • total_context_chars and max_context_chars
  • n_dropped and dropped_ids
  • a context.truncated_chars measurement
  • a warning-level event whenever n_dropped > 0

Now a wrong answer on a long-context request is not a mystery. The pipeline is instrumented end to end: rag.answer to rag.retrieve (hit_ids) to rag.build_prompt (dropped_ids) to rag.generate (stop_reason). The trace below exercises the assembly step directly, so it shows rag.answer to rag.build_prompt, where the truncation happens and the dropped ids surface.


On the over-budget request, the rag.build_prompt span reads total_context_chars 7343 against a max_context_chars of 6000, a context.truncated_chars measurement of 1343, and the dropped_ids, sitting directly under the rag.answer transaction. The truncation that used to be invisible is now a value on a span.

Best Use of Google AI

To make the localization repeatable I added scripts/root_cause.py: it takes a Sentry trace plus the source files that ran, and asks Gemini to point at the line. The prompt is fixed and paste-ready:

You are debugging a RAG pipeline that returned a wrong answer. Here is a Sentry trace and the relevant source. The retrieval span shows the correct passage was retrieved, but the final answer is wrong. Localize the exact line where the retrieved context is lost, explain the mechanism, and note whether the loss is signalled to the caller. Then propose a minimal fix.

The script is env-gated on GOOGLE_API_KEY or GEMINI_API_KEY and fails with a clear message when no key is set, so it never runs blind.

Run against the pre-fix source with the real trace, on gemini-2.5-flash, it returned this (verbatim):

The Sentry trace clearly indicates that the rag.retrieve span successfully identified expenses::answer as a relevant passage. However, the subsequent rag.build_prompt span shows a total_context_chars of 7343 and a max_context_chars of 6000, leading to 1343 characters being truncated. The symptom confirms that expenses::answer was "absent from the assembled prompt," which directly led to the LLM's abstention (I could not find this in the provided documents.).

1. Exact Line Where Context is Lost. The context is lost in app/generate.py, specifically at this line within the build_prompt function: context = "\n\n".join(blocks)[: config.MAX_CONTEXT_CHARS].

2. Mechanism of Loss: naive string slicing after concatenation. The function joins all blocks into a single long string, then applies a hard character limit by slicing that entire concatenated string. If the total length exceeds config.MAX_CONTEXT_CHARS, the slice cuts characters from the end of the combined string. Passages later in the hits list are more likely to be truncated, and even a highly relevant passage like expenses::answer can be cut off if it falls past the character limit.

3. Signalling of Loss. "No exception, warning, or dropped-id was reported to the caller." The function simply returns the truncated string; there are no checks, logging.warning calls, or exceptions that would inform the calling Generator.answer method. Sentry captured the context.truncated_chars measurement, but that is instrumentation-level reporting, not a signal within the application's business logic.

4. Minimal Fix. Adjust build_prompt to iteratively add passages, respecting config.MAX_CONTEXT_CHARS without splitting individual passages, so higher-ranked complete passages are prioritized. [It then produced a pack-to-fit loop that stops adding blocks once the budget is reached.]

The model landed on the same line I did, read the mechanism straight off the trace measurements, and proposed the same pack-to-fit shape as the fix above. Sentry said which chunk vanished and by how much; Gemini said which line threw it away and why nothing noticed.

Vinicius Pereira
vinimabreu.dev · github.com/vinimabreu

Top comments (0)