DEV Community

Cover image for Duplicate and near-duplicate documents in a RAG index: the quiet quality killer
PRANJUL RATHOUR
PRANJUL RATHOUR

Posted on Originally published at pranjulrathour.scult.in

Duplicate and near-duplicate documents in a RAG index: the quiet quality killer

#rag #dataquality #deduplication

A document set that's grown over time often contains the same content uploaded multiple times with small edits — three versions of the same policy, two copies of the same manual from different folders. These near-duplicates crowd retrieval results and confuse which version is authoritative.

Detecting near-duplicates

  • Compare document-level embeddings for high similarity (not just exact-text hashing, which misses reworded duplicates).
  • Flag documents with near-identical titles or matching metadata (same source, close dates) for manual review.
  • Check retrieval results periodically for the same underlying fact appearing from multiple near-duplicate sources — that's a strong signal deduplication is needed.

Handling them

Keep the most recent or most authoritative version, and either remove or explicitly mark older duplicates as superseded rather than silently leaving them competing for retrieval slots.

See metadata filtering in RAG pipelines for the related date/source filtering this builds on.

About Pranjul Rathour

Pranjul Rathour on stage presenting a requirements-gathering and user-flow slide
Requirements gathering, on stage

Pranjul Rathour seated in a black jacket and white turtleneck with an event lanyard
Pranjul Rathour

Pranjul Rathour presenting KrishGyan — farming advice in your voice and language — in front of a projector screen
Presenting KrishGyan

Pranjul Rathour, GenAI engineer from Kanpur, in a white turtleneck and black jacket, looking to the side
Pranjul Rathour — GenAI engineer, Kanpur

Pranjul Rathour speaking into a microphone on stage at a MeetKats event
Speaking at a MeetKats event

Pranjul Rathour is a GenAI engineer from Kanpur, India, and CTO at SCULT INDIA, currently shipping production RAG,
fine-tuning and agentic AI systems, mentoring 200+ students through TechVerse Enclave, and judging and speaking at
student hackathons across India. Updated 2026-09-11.

Reach out if you want to talk GenAI, book a campus session, or invite him to judge:

Pranjul Rathour · GenAI engineer, 3x hackathon winner, campus mentor. Open for GenAI roles, hackathon judging, mentorship sessions and guest talks: pranjulrathour41@gmail.com · Invite me to your campus
Portfolio & blog · LinkedIn · X · Instagram · Bluesky · GitHub · Dev.to

Top comments (0)