DEV Community

Perceora profile image Sathvik Dembale Krishnappagowda
Sathvik Dembale Krishnappagowda for Perceora

Posted on Originally published at perceora.ai AI-assisted

Your test score may be lying: near-duplicate images between training and test sets

#machinelearning #computervision #python #datascience

In short

  • If an image in your test set is a near-copy of an image in your training set, the model has already seen it. Your test score then looks better than it really is.
  • A short Python script (about 50 lines) finds most of these copies. It shrinks every image to a tiny grey thumbnail and compares the thumbnails.
  • The check is not perfect, so look at every pair it finds. And split your images by part or by photo session, not file by file.

The problem

A model learns from one group of images, the training set. You then measure it on images it has never seen, the test set. The test score is your best guess of how the model will do on new images.

That guess only works if the test images are really new. If a test image is a resized or re-saved copy of a training image, the model can simply recognise it. The test then partly measures memory, not skill.

This happens even in famous datasets. Barz and Denzler checked CIFAR-10 and CIFAR-100, two widely used image datasets [1, 2]. They found that 3.3 % of the CIFAR-10 test images and 10 % of the CIFAR-100 test images have duplicates in the training set [1]. When they replaced those images with new ones and re-tested popular image classification models, accuracy dropped by 9–14 % relative to the original results [1].

How copies end up in both sets

  • The same part photographed twice, or several shots in a row that look almost the same.
  • Frames from one video, split frame by frame.
  • Copies saved along the way: a smaller version, a JPEG export, a brighter copy, a slightly cropped one.
  • Augmented images saved to disk before the split: a flipped copy lands in test, the original in train.
  • Merged folders: two colleagues' image folders that overlap.

A normal random split can't see any of this. It shuffles file names, not image content.

How the check works

The check has three steps:

  1. Make a tiny thumbnail of every image: 16 × 16 pixels, in grey. Resizing, JPEG compression or a brightness change hardly change this thumbnail.
  2. Compare every test thumbnail with every training thumbnail. The result is a similarity between 0 and 1. A value of 1.00 means the thumbnails look the same.
  3. List every test image whose best match is above a cut-off (the threshold), for example 0.90. Then look at each pair yourself.

The check shrinks each image to a 16 × 16 grey thumbnail. A resized copy keeps the same thumbnail (similarity 1.00); a different part does not (0.46). Computer-generated examples.

For readers who want the detail: before comparing, each thumbnail is shifted so its average is zero and scaled to length 1. The similarity is then the cosine of the angle between the two thumbnails, so overall brightness and contrast don't matter.

The code

You need Python with NumPy [3] and Pillow [4]. Save the code as find_leaks.py and run python find_leaks.py data/train data/test. It prints the pairs it finds and saves them to leaks.csv.

import csv, hashlib, sys
from pathlib import Path
import numpy as np
from PIL import Image

EXTS = {".jpg", ".jpeg", ".png", ".bmp", ".tif", ".tiff", ".webp"}

def list_images(folder):
    return sorted(p for p in Path(folder).rglob("*") if p.suffix.lower() in EXTS)

def signature(path, size=16, mirror=False):
    with Image.open(path) as im:
        im = im.convert("L")
        if mirror:
            im = im.transpose(Image.FLIP_LEFT_RIGHT)
        small = im.resize((size, size), Image.BOX)
    v = np.asarray(small, dtype=np.float32).ravel()
    v -= v.mean()
    n = np.linalg.norm(v)
    return v / n if n > 0 else v

def file_hash(path):
    return hashlib.sha1(Path(path).read_bytes()).hexdigest()

def find_leaks(train_dir, test_dir, threshold=0.90, mirror=True, chunk=2048):
    train, test = list_images(train_dir), list_images(test_dir)
    T = np.stack([signature(p) for p in train])
    best_sim, best_idx = [], []
    for start in range(0, len(test), chunk):
        batch = test[start:start + chunk]
        E = np.stack([signature(p) for p in batch])
        S = E @ T.T                                   # test x train similarities
        if mirror:                                    # catch flipped copies too
            S = np.maximum(S, np.stack([signature(p, mirror=True) for p in batch]) @ T.T)
        best_idx.extend(S.argmax(axis=1))
        best_sim.extend(S.max(axis=1))
    exact = {file_hash(p): p for p in train}
    rows = []
    for p, i, s in zip(test, best_idx, best_sim):
        twin = exact.get(file_hash(p))
        if twin is not None:
            rows.append((p, twin, 1.0, "identical file"))
        elif s >= threshold:
            rows.append((p, train[i], float(s), "near-duplicate"))
    rows.sort(key=lambda r: (-r[2], r[3] != "identical file"))
    return rows

if __name__ == "__main__":
    rows = find_leaks(sys.argv[1], sys.argv[2])
    with open("leaks.csv", "w", newline="") as f:
        csv.writer(f).writerows([("test_image", "closest_train_image", "similarity", "kind")]
                                + [(t, tr, f"{s:.3f}", k) for t, tr, s, k in rows])
    for t, tr, s, kind in rows:
        print(f"{s:.3f}  {Path(t).name}  <->  {Path(tr).name}  ({kind})")
Enter fullscreen mode Exit fullscreen mode

What the parts do:

  • signature makes the tiny thumbnail (step 1).
  • find_leaks compares all thumbnails at once (step 2) and keeps the pairs above the threshold (step 3).
  • file_hash also catches files that are byte-for-byte identical.
  • mirror=True also compares a left-right flipped version, so flipped copies are found too.
  • The test images are compared in batches (chunk), so even large folders fit in memory. For hundreds of thousands of images, a library for fast similarity search such as FAISS [5] does the same job faster.

What it found in an example

To test the check, a short script drew 40 training images and 22 test images of simple metal parts: washers, gears and brackets. All images in this article are computer-generated.

The script hid seven copies of training images in the test folder: an identical file, a JPEG export, a resized copy, a heavily compressed copy, a brighter copy, a slightly cropped copy and a mirrored copy. It also added a new "photo" of a part that is already in the training set.

Each flagged test image next to its closest training image, with the similarity. Computer-generated examples.

At a threshold of 0.90:

  • The check found all seven hidden copies (similarity 0.977 to 1.000). The mirrored copy was found only thanks to the mirror step; without it, it scored 0.771.
  • It also flagged three different parts (0.907 to 0.954). In a 16 × 16 thumbnail, one gear in the middle of a grey table looks a lot like another gear.

The best match in the training set for every test image. Red: hidden copies. Orange: different parts. Blue: the same part photographed again. Computer-generated examples.

Choosing the threshold

There is no perfect number. Two simple rules:

  1. Sort the list and look. Real copies sit near 1.0. Look-alikes come right below them. Open the pairs around your threshold side by side and decide by eye.
  2. Start low, then check by eye. A missed copy quietly makes your score look better. A false alarm costs you a few seconds of looking. A threshold of about 0.90 plus a look at every pair is a good start.

If your images are mostly background, as in the example, look-alikes score higher. A slightly bigger thumbnail (24 or 32 pixels) separates them a little better, but it also handles crops a little worse.

What this check misses

  • The same object photographed again. The new photo of a training part scored only 0.655, so it was not flagged. The thumbnails compare pixels, not objects. A neural network that describes image content (an embedding model) can catch such cases. That is how Barz and Denzler searched CIFAR: they compared images in the feature space of a neural network, then checked every pair by hand [1]. Expect more look-alikes in the list too, so the check by eye stays.
  • Rotations and bigger crops. They change the thumbnail too much. In the example, a copy rotated by 90° scored 0.867, and copies cropped to a quarter or half of the image scored 0.48 to 0.58. All of these are below the threshold.
  • Copies inside one set. Two copies inside the training set don't leak into the test score, but they count twice during training. To find them, compare the training set with itself and ignore each image's match with itself.

Fix the split, not only the pairs

Deleting or moving the flagged pairs fixes today's dataset. It doesn't stop new leaks. The lasting fix is to split by group: all photos of one part, one photo session or one video go into the same set.

Ten parts, three photos each. A random split puts photos of the same part into both sets; splitting by part keeps each part in one set. Computer-generated example.

In this small example, a random split put photos of already-trained parts into all 6 test slots. Splitting by part brought that to 0. The function below does the split. It only needs a rule that turns a file name into its group:

import random

def group_split(paths, group_of, test_share=0.2, seed=0):
    groups = sorted({group_of(p) for p in paths})
    random.Random(seed).shuffle(groups)
    test_groups = set(groups[:max(1, round(len(groups) * test_share))])
    train = [p for p in paths if group_of(p) not in test_groups]
    test = [p for p in paths if group_of(p) in test_groups]
    return train, test

# "part07_shot2.jpg" -> group "part07"
train, test = group_split(paths, lambda p: p.stem.split("_")[0])
Enter fullscreen mode Exit fullscreen mode

If you already use scikit-learn [6], GroupShuffleSplit and GroupKFold do the same job.

Checklist

  • Split by part, session or video, not file by file.
  • Remove identical files first.
  • Run a near-duplicate check between training and test set, and look at every pair it flags.
  • Save augmented images only after the split, or also check for flipped copies.
  • Run the check again whenever you add images.

Sources

  1. B. Barz and J. Denzler, "Do We Train on Test Data? Purging CIFAR of Near-Duplicates," Journal of Imaging, vol. 6, no. 6, article 41, 2020. doi.org/10.3390/jimaging6060041 (open access; preprint: arXiv:1902.00423)
  2. A. Krizhevsky, "Learning Multiple Layers of Features from Tiny Images," technical report, University of Toronto, 2009. cs.toronto.edu/~kriz/learning-features-2009-TR.pdf (describes the CIFAR-10 and CIFAR-100 datasets)
  3. C. R. Harris et al., "Array programming with NumPy," Nature, vol. 585, pp. 357–362, 2020. doi.org/10.1038/s41586-020-2649-2 · numpy.org
  4. Pillow, the Python imaging library. python-pillow.org
  5. J. Johnson, M. Douze and H. Jégou, "Billion-Scale Similarity Search with GPUs," IEEE Transactions on Big Data, vol. 7, no. 3, pp. 535–547, 2021. doi.org/10.1109/TBDATA.2019.2921572 · Code: github.com/facebookresearch/faiss
  6. F. Pedregosa et al., "Scikit-learn: Machine Learning in Python," Journal of Machine Learning Research, vol. 12, pp. 2825–2830, 2011. jmlr.org/papers/v12/pedregosa11a.html · Documentation: GroupShuffleSplit, GroupKFold
  7. The MIT License. opensource.org/license/mit

All links were checked on 8 Nov 2026. The figures and example images were made with our own scripts and are computer-generated.

The scores come from running the code above on computer-generated example images. You may use the code freely under the MIT License [7]; it comes without any warranty. This article was drafted using an AI assistant and reviewed by Sathvik Dembale Krishnappagowda.

Originally published at https://perceora.ai/articles/near-duplicate-leaks/.

Top comments (0)