DEV Community

Ayi NEDJIMI
Ayi NEDJIMI

Posted on

Semantic deduplication for large text datasets

#ai #python #tutorial #machinelearning

When you build a dataset for ML training or a RAG knowledge base, exact deduplication is not enough. Copy-paste duplicates are easy to catch with a hash. Paraphrases, reformulations, and semantically equivalent sentences are not. Running standard MinHash or Jaccard on them gives near-zero similarity even when they carry identical information. The result: bloated corpora, biased models, and retrieval systems that return the same fact dressed in different words.

Semantic deduplication fixes this by comparing meaning instead of tokens.

Why Exact and Fuzzy Dedup Fall Short

Exact deduplication works by hashing document content — fast, but it only catches bit-for-bit identical strings. Fuzzy deduplication (MinHash, SimHash, Jaccard on n-grams) extends this to near-verbatim copies. It handles most copy-paste variants well.

Consider these two sentences:

  • "The server returned a 500 error"
  • "An internal server error was encountered"

MinHash Jaccard similarity: roughly 0.08 (no shared tokens). Semantic similarity: close to 1.0. Both describe the same event. Fuzzy dedup keeps both; semantic dedup removes one.

For production RAG pipelines, curated training sets, or any corpus where redundancy directly affects downstream quality, this is a meaningful gap.

The Core Approach: Embeddings + Similarity Threshold

The pipeline has three steps:

  1. Embed all documents into a dense vector space using a sentence embedding model
  2. Find pairs (or clusters) of vectors above a similarity threshold
  3. Keep one representative per cluster, discard the rest

Step 2 is the hard part. Naive pairwise comparison is O(n²) — acceptable at 10k documents, unusable at 1M.

Basic semantic dedup for small datasets 

from sentence_transformers import SentenceTransformer
import numpy as np

def semantic_dedup(texts: list[str], threshold: float = 0.87) -> list[str]:
    model = SentenceTransformer("all-MiniLM-L6-v2")
    embeddings = model.encode(texts, batch_size=64, show_progress_bar=True)

    # L2-normalize for cosine similarity via dot product
    norms = np.linalg.norm(embeddings, axis=1, keepdims=True)
    normalized = (embeddings / norms).astype("float32")

    keep = []
    removed = set()

    for i in range(len(texts)):
        if i in removed:
            continue
        keep.append(i)
        sims = normalized[i] @ normalized[i + 1:].T
        duplicates = np.where(sims >= threshold)[0] + i + 1
        removed.update(duplicates.tolist())

    return [texts[i] for i in keep]


texts = [
    "The server returned a 500 error",
    "An internal server error was encountered",
    "The database connection failed",
    "Failed to connect to the database",
    "User authentication succeeded",
]

result = semantic_dedup(texts, threshold=0.82)
print(f"Before: {len(texts)}, After: {len(result)}")
# Before: 5, After: 3
Enter fullscreen mode Exit fullscreen mode

This handles datasets up to ~50k documents. At 100k, the pairwise similarity matrix starts hitting ~40 GB RAM.

Scaling to Millions of Documents with FAISS

For large-scale deduplication, approximate nearest neighbor (ANN) search replaces exhaustive pairwise comparison. FAISS builds an index over your embeddings and returns the K nearest neighbors for each vector without computing all distances.

The strategy: instead of a similarity matrix, you find clusters of near-duplicates and keep the lowest-index member of each cluster. Union-Find handles cluster merging in O(n α(n)) — essentially linear.

import faiss
import numpy as np
from sentence_transformers import SentenceTransformer
from collections import defaultdict

def semantic_dedup_large(
    texts: list[str],
    threshold: float = 0.87,
    k_neighbors: int = 10,
    batch_size: int = 512,
) -> list[str]:
    model = SentenceTransformer("all-MiniLM-L6-v2")
    embeddings = model.encode(
        texts,
        batch_size=batch_size,
        show_progress_bar=True,
        normalize_embeddings=True,
    ).astype("float32")

    dim = embeddings.shape[1]
    index = faiss.IndexFlatIP(dim)
    index.add(embeddings)

    distances, indices = index.search(embeddings, k_neighbors + 1)

    parent = list(range(len(texts)))

    def find(x: int) -> int:
        while parent[x] != x:
            parent[x] = parent[parent[x]]
            x = parent[x]
        return x

    def union(x: int, y: int) -> None:
        px, py = find(x), find(y)
        if px != py:
            parent[px] = py

    for i in range(len(texts)):
        for j, sim in zip(indices[i], distances[i]):
            if j != i and sim >= threshold:
                union(i, j)

    clusters: dict[int, list[int]] = defaultdict(list)
    for i in range(len(texts)):
        clusters[find(i)].append(i)

    kept = sorted(min(members) for members in clusters.values())
    return [texts[i] for i in kept]
Enter fullscreen mode Exit fullscreen mode

With IndexFlatIP (exact search), this scales comfortably to ~5M documents on a machine with 32 GB RAM using 384-dim embeddings. For 10M+, swap in IndexIVFFlat with a coarse quantizer — you trade ~1–2% recall for a 10–20x speedup.

Choosing the Right Threshold

The threshold controls aggressiveness. There is no universal value; it depends on your domain.

Threshold Effect
0.95+ Near-verbatim only — very conservative
0.87–0.94 Strong semantic overlap removed — good for training data
0.75–0.86 Topically similar content merged — aggressive
< 0.75 Will over-deduplicate most corpora

For RAG knowledge bases, 0.87–0.90 is a reliable starting point. Meta's SemDeDup paper uses a similar range on large-scale web crawls.

Do not pick a threshold blindly. Sample 50 pairs that fall within ±0.03 of your candidate threshold and label them manually as duplicate/not-duplicate. This takes 30 minutes and prevents days of debugging degraded model behavior.

Incremental Deduplication in Practice

In production, you rarely re-embed your entire corpus from scratch. The common pattern is incremental: you have an existing indexed corpus, and you are adding new documents.

Index the existing corpus once. For each new document, query against the index. If the nearest neighbor exceeds your threshold, drop the new document. If not, add it to the index.

This keeps encoding cost proportional to new data, not total corpus size. It also means your FAISS index grows over time — plan for periodic re-indexing if you care about ANN recall staying consistent.

When processing user-generated content or scraped data through these pipelines, review your data handling procedures carefully. A semantic deduplication stage can inadvertently surface sensitive patterns in your corpus. If you are building data pipelines in a security-sensitive context, the security hardening checklists for data pipelines cover the relevant controls.

The Takeaway

Exact and fuzzy deduplication leave real semantic duplicates in your dataset. Semantic dedup via embeddings + ANN search is mature, accessible, and worth adding as a standard pipeline step.

Start with all-MiniLM-L6-v2 and threshold 0.87. Test on a 10k sample, manually inspect ~50 boundary pairs, then scale with FAISS. The quality gains in downstream models — less hallucination, more diverse retrieval results, cleaner training signal — are measurable and worth the extra pipeline step.

I run AYI NEDJIMI Consultants, a cybersecurity consulting firm. We publish free security hardening checklists — PDF and Excel.

Top comments (0)