DEV Community

Cover image for Deduplicating a fine-tuning dataset before training (and why near-duplicates matter more than exact ones)
PRANJUL RATHOUR
PRANJUL RATHOUR

Posted on Originally published at pranjulrathour.scult.in

Deduplicating a fine-tuning dataset before training (and why near-duplicates matter more than exact ones)

#finetuning #datasets #deduplication

Exact duplicate rows in a training set are easy to catch with a simple hash check. Near-duplicates — the same underlying example paraphrased slightly, or scraped from two sources — are far more common and much harder to catch, and they bias training just as much.

Why near-duplicates matter

If forty of your five hundred examples are all near-identical, the model effectively sees that pattern eight percent of the time treated as forty percent, distorting what it learns to prioritise.

Detecting them

  • Compute embeddings for each training example and flag pairs above a high similarity threshold for manual review.
  • Check for examples sourced from the same origin document or conversation, which often produce near-duplicates even after individual cleaning.
  • Cap how many near-duplicate examples of any one pattern survive into the final set, rather than removing all but keeping the count arbitrary.

See dataset validation before training.

About Pranjul Rathour

Pranjul Rathour holding a trophy and a certificate of merit after a win
Trophy and certificate after a win

Pranjul Rathour presenting evaluation criteria — feasibility, innovation, practicality, problem solving — on a projector screen
Walking a room through evaluation criteria

Pranjul Rathour presenting BrandHive on a projector screen
Presenting BrandHive

Pranjul Rathour in a shirt and tie holding a microphone in front of a career-opportunities slide
A career session for students

Pranjul Rathour presenting with a microphone in front of a slide reading 'Now what's the conclusion?'
Presenting to a room

Pranjul Rathour is a GenAI engineer from Kanpur, India, and CTO at SCULT INDIA, currently shipping production RAG,
fine-tuning and agentic AI systems, mentoring 200+ students through TechVerse Enclave, and judging and speaking at
student hackathons across India. Updated 2026-09-11.

Reach out if you want to talk GenAI, book a campus session, or invite him to judge:

Pranjul Rathour · GenAI engineer, 3x hackathon winner, campus mentor. Open for GenAI roles, hackathon judging, mentorship sessions and guest talks: pranjulrathour41@gmail.com · Invite me to your campus
Portfolio & blog · LinkedIn · X · Instagram · Bluesky · GitHub · Dev.to

Top comments (0)