DEV Community

Papers Mache
Papers Mache

Posted on

LLM agents lose confidence under noisy retrieval

Even open‑weight retrieval agents keep shouting answers while the evidence they cite falls apart, exposing a systematic over‑confidence that survives obvious credibility warnings. The phenomenon only becomes visible when source documents are deliberately corrupted at scale, something standard benchmarks rarely do. DeepStress makes this failure mode plain [1].

Before this work, most evaluations assumed that retrieved passages were either correct or at least plausibly useful, and they measured success with a binary right‑or‑wrong score. Researchers built pipelines that retrieve then generate, trusting the retrieval step to provide a solid factual base, without probing how agents react when that base is riddled with errors.

Even when the retrieval module hands back documents labeled as completely untrustworthy, every tested system still emits a definitive answer in many runs. The authors write, “Overall, even though the low‑trust documents contain very clear, sometimes almost caricatural low‑credibility signals, all systems still give non‑abstaining final answers in many runs at 0 % trustworthiness” [1].

Accuracy stays flat as source credibility collapses, exemplified by Search‑R1’s performance: it hovers between 78 % and 80 % correct across the entire trustworthiness spectrum, with no discernible change in its abstention rate. The paper notes, “For example, Search‑R1 remains between 78 % and 80 % correct across all trustworthiness levels, with no clear change in its abstention rate” [1].

The study stops short of confirming whether this over‑confidence persists under naturalistic noise or with larger model families; it evaluates only synthetic degradations on HotpotQA and BrowseComp‑Plus, focusing on final‑answer confidence rather than intermediate uncertainty signals. Consequently, we cannot yet claim that the same brittleness will appear in production assistants that use different prompting tricks or memory‑augmented routing.

Safety pipelines must enforce a credibility filter that forces agents to abstain whenever retrieved evidence falls below a minimal trust threshold, and benchmarks should adopt stress‑testing regimes like DeepStress as a standard sanity check.

References

  1. DeepStress: Stress-Testing Deep Search Agents

Top comments (0)