Building a Video Deduplication Pipeline with Perceptual Hashing in Python

At DailyWatch we ingest tens of thousands of new videos every day from dozens of regional sources. The single most annoying problem in running a free video discovery platform is not storage and it is not bandwidth. It is the same clip showing up eleven times in one feed. A trending music video gets re-uploaded, mirrored, letterboxed, re-encoded at a different bitrate, and stamped with three different channel watermarks. To a byte-for-byte comparison these are eleven distinct files. To a human they are one video, and seeing it eleven times makes the whole feed look broken.

This post walks through the deduplication pipeline we actually run in production: how we turn a video into a compact perceptual fingerprint in Python, how we compare fingerprints without exploding into an O(n²) nightmare, and how we store and query the results from a PHP 8.4 backend sitting on SQLite. The techniques are simple, the code is runnable, and the payoff is a discovery feed that stops repeating itself.

Why MD5 and SHA-256 are exactly the wrong tool

The first instinct of most backend engineers is to hash the file and compare digests. We tried it. It fails immediately, and it fails for a reason that is worth internalizing.

Cryptographic hashes are engineered to be maximally sensitive. Flip a single bit anywhere in the input and the output digest changes completely. That avalanche property is the entire point for integrity verification, but it is the opposite of what deduplication needs. A re-encode from H.264 to H.265 rewrites millions of bytes while the picture a viewer sees is pixel-for-pixel almost identical. A resize from 1080p to 720p changes every byte. Re-muxing the same stream into a different container changes the header. In every one of these cases MD5 confidently reports "different file," and every one of them is the same video to a human being.

We need the inverse property. We want a fingerprint that stays stable under transformations that do not change perceived content, and only diverges when the actual imagery on screen diverges. That is precisely what perceptual hashing gives us.

Perceptual hashing in one paragraph

A perceptual hash reduces an image to a small fixed-size fingerprint (usually 64 bits) that encodes the coarse structure of the image rather than its exact pixels. Two visually similar images produce fingerprints that differ in only a handful of bits. Two unrelated images produce fingerprints that differ in roughly half their bits. Instead of asking "are these hashes equal?" you ask "how many bits differ?" — the Hamming distance. A small distance means "probably the same picture." There are several algorithms (aHash, pHash, dHash, wHash); we use dHash because it is fast, dependency-light, and surprisingly robust against the compression and resizing noise that dominates re-uploaded videos.

Step 1: turn one frame into a 64-bit fingerprint

dHash works by tracking relative brightness gradients. You shrink the frame to a tiny grid, convert to grayscale, and record for each pixel whether it is brighter than its right-hand neighbor. Because it encodes differences between adjacent pixels rather than absolute values, it is naturally immune to changes in overall brightness, contrast, and gamma — all of which shift wildly across re-encodes.

from PIL import Image

def dhash(image: Image.Image, hash_size: int = 8) -> int:
    # Resize to (hash_size+1) x hash_size so each row yields hash_size comparisons.
    small = image.convert("L").resize(
        (hash_size + 1, hash_size), Image.LANCZOS
    )
    pixels = list(small.getdata())

    bits = 0
    for row in range(hash_size):
        for col in range(hash_size):
            left = pixels[row * (hash_size + 1) + col]
            right = pixels[row * (hash_size + 1) + col + 1]
            bits = (bits << 1) | (1 if left > right else 0)
    return bits  # a 64-bit integer when hash_size == 8

That is the entire perceptual core. An 8x8 grid gives 8 comparisons per row across 8 rows, which is exactly 64 bits packed into a single Python integer. No machine learning, no external service, no GPU. Two frames from the same scene will land within a few bits of each other even after a full re-encode.

Step 2: sample representative frames from the video

A video is not one image, and hashing a single frame is fragile: many uploaders splice a 3-second intro card onto an otherwise identical clip, which would throw off any single-frame fingerprint taken from the start. The fix is to sample frames at fixed percentage positions through the runtime rather than at fixed timestamps, so intros and outros of different lengths do not shift the whole sequence.

We lean on ffmpeg, which is already installed on our workers for thumbnail generation. This function seeks to evenly spaced points and decodes one frame at each.

import subprocess
import io
from PIL import Image

def extract_frames(path: str, duration: float, count: int = 8):
    """Grab `count` frames spread evenly across the runtime (skipping the
    very start and end where intros/outros distort things)."""
    frames = []
    for i in range(count):
        # Sample between 5% and 95% of the runtime.
        pct = 0.05 + (0.90 * i / (count - 1))
        ts = duration * pct
        proc = subprocess.run(
            [
                "ffmpeg", "-ss", f"{ts:.2f}", "-i", path,
                "-frames:v", "1", "-f", "image2pipe",
                "-vcodec", "png", "-loglevel", "error", "pipe:1",
            ],
            capture_output=True,
        )
        if proc.returncode == 0 and proc.stdout:
            frames.append(Image.open(io.BytesIO(proc.stdout)))
    return frames

Putting -ss before -i tells ffmpeg to seek at the container level, which is dramatically faster than decoding from the start of the file. For a two-minute clip on a modest worker this whole extraction runs in well under a second.

Step 3: a video-level signature

Now we combine per-frame hashes into a single signature. Keeping the frame hashes as an ordered tuple (rather than XOR-ing them into one number) preserves temporal structure, so a video and its reversed or reordered copy will not accidentally collide. The signature is small — eight 64-bit integers — which matters a lot when you are storing millions of them.

from dataclasses import dataclass

@dataclass
class VideoSignature:
    frame_hashes: tuple[int, ...]

    @classmethod
    def from_video(cls, path: str, duration: float):
        frames = extract_frames(path, duration, count=8)
        return cls(tuple(dhash(f) for f in frames))

def hamming(a: int, b: int) -> int:
    return (a ^ b).bit_count()  # Python 3.10+ popcount

def signature_distance(s1: VideoSignature, s2: VideoSignature) -> int:
    """Total bit difference across aligned frames. Lower = more similar."""
    if len(s1.frame_hashes) != len(s2.frame_hashes):
        return 10_000  # incomparable, treat as far apart
    return sum(
        hamming(x, y)
        for x, y in zip(s1.frame_hashes, s2.frame_hashes)
    )

int.bit_count() (added in Python 3.10) is a hardware popcount and is far faster than counting bits in a loop. Across 8 frames the maximum possible distance is 8 × 64 = 512 bits. In practice, genuine duplicates score under 40, unrelated videos score in the 200s, and the gap between those two populations is wide and clean — which is exactly what makes the threshold easy to pick.

Step 4: storing and querying from the PHP backend

Our ingestion is Python, but the site itself — the discovery feed, the dedup lookups at read time — runs on PHP 8.4 behind LiteSpeed, with SQLite as the datastore and Cloudflare in front. When the Python worker finishes hashing a new video it writes the signature into SQLite, and PHP is responsible for answering "have we seen this before?"

Since SQLite has no native perceptual-distance operator, and PHP 8.4 gives us 64-bit integers and a fast gmp/native XOR, we implement Hamming distance in PHP. The one trap: XOR-ing two 64-bit values can set the sign bit, turning the result negative, and a naive right shift on a negative int sign-extends forever. We clear the sign bit on each shift to avoid an infinite loop.

<?php
declare(strict_types=1);

function hamming64(int $a, int $b): int {
    $x = $a ^ $b;
    $count = 0;
    while ($x !== 0) {
        $count += $x & 1;
        // Logical shift: clear the sign bit so negatives don't loop forever.
        $x = ($x >> 1) & 0x7FFFFFFFFFFFFFFF;
    }
    return $count;
}

function signatureDistance(array $a, array $b): int {
    if (count($a) !== count($b)) {
        return 10_000;
    }
    $total = 0;
    foreach ($a as $i => $h) {
        $total += hamming64($h, $b[$i]);
    }
    return $total;
}

The signatures live in a plain table. We store each frame hash as a signed 64-bit integer column — SQLite's native integer type holds 64 bits, and the bitwise XOR is agnostic to the sign interpretation, so nothing is lost.

CREATE TABLE video_signatures (
    video_id   INTEGER PRIMARY KEY,
    h0 INTEGER, h1 INTEGER, h2 INTEGER, h3 INTEGER,
    h4 INTEGER, h5 INTEGER, h6 INTEGER, h7 INTEGER,
    band0 INTEGER, band1 INTEGER, band2 INTEGER, band3 INTEGER
);
CREATE INDEX idx_band0 ON video_signatures(band0);
CREATE INDEX idx_band1 ON video_signatures(band1);
CREATE INDEX idx_band2 ON video_signatures(band2);
CREATE INDEX idx_band3 ON video_signatures(band3);

Those band columns are the whole trick to making this scale, and they deserve their own section.

Step 5: escaping the O(n) comparison trap

Here is the problem that kills naive perceptual dedup. Hamming distance is not something you can put in a B-tree index and binary-search. If you have a million signatures and a new video arrives, the obvious approach is to compute the distance against all million rows. That is O(n) per insert and O(n²) for a full re-scan, and it collapses the moment your library grows.

The standard escape is the banded LSH (locality-sensitive hashing) approach, and it is beautifully simple. Take the first frame hash — 64 bits — and split it into four 16-bit bands. Two near-duplicate videos differ by only a handful of bits total. By the pigeonhole principle, if two 64-bit hashes differ in at most 3 bits, then across four 16-bit bands at least one band must be bit-for-bit identical. So instead of scanning every row, you only fetch rows that share at least one exact band value with your query. That candidate set is tiny, and you run the expensive full-signature distance only on those.

def bands(h: int, band_bits: int = 16, n_bands: int = 4):
    mask = (1 << band_bits) - 1
    return [(h >> (band_bits * i)) & mask for i in range(n_bands)]

# On insert, precompute and store the four bands of the anchor frame (h0).
def bands_for_storage(sig: VideoSignature):
    b = bands(sig.frame_hashes[0])
    return {"band0": b[0], "band1": b[1], "band2": b[2], "band3": b[3]}

The candidate lookup in PHP becomes a single indexed query — no full-table scan, and it returns on the order of dozens of rows instead of a million:

<?php
function findCandidates(PDO $db, array $bands): array {
    $sql = 'SELECT video_id, h0,h1,h2,h3,h4,h5,h6,h7
              FROM video_signatures
             WHERE band0 = :b0 OR band1 = :b1
                OR band2 = :b2 OR band3 = :b3
             LIMIT 500';
    $stmt = $db->prepare($sql);
    $stmt->execute([
        ':b0' => $bands[0], ':b1' => $bands[1],
        ':b2' => $bands[2], ':b3' => $bands[3],
    ]);
    return $stmt->fetchAll(PDO::FETCH_ASSOC);
}

function isDuplicate(PDO $db, array $querySig, array $bands, int $threshold = 40): ?int {
    foreach (findCandidates($db, $bands) as $row) {
        $candidate = [
            (int)$row['h0'], (int)$row['h1'], (int)$row['h2'], (int)$row['h3'],
            (int)$row['h4'], (int)$row['h5'], (int)$row['h6'], (int)$row['h7'],
        ];
        if (signatureDistance($querySig, $candidate) <= $threshold) {
            return (int)$row['video_id'];  // the original we already have
        }
    }
    return null;  // genuinely new
}

This turns a linear scan into an indexed candidate fetch followed by a distance check on a handful of rows. On our library it keeps dedup lookups in the low single-digit milliseconds even as the table crosses seven figures, which means we can run it inline during ingestion without a queue backlog.

Tuning the threshold and the failure modes

The single most important number in this whole system is the distance threshold, and you cannot pick it from theory — you have to measure it against your own data. Here is what we learned:

• Set it too low and you miss re-encodes with heavy watermarks or aggressive re-compression. Duplicates leak through and your feed still repeats.
• Set it too high and you start collapsing distinct videos that merely share a visual style — two different gameplay clips of the same game, two news segments with the same studio background. Those false positives are worse than duplicates because you are hiding real content.
• The sweet spot is empirical. We sampled a few thousand known-duplicate and known-distinct pairs, plotted the distance distributions, and found a wide valley between them. For our 8-frame signatures that valley sits around 40 out of a possible 512, so that is our threshold with comfortable margin on both sides.

A few operational notes from running this for real:

• Letterboxing breaks dHash because the black bars dominate the gradient grid. We detect and crop uniform borders with a quick ffmpeg cropdetect pass before hashing. This one fix recovered a large chunk of missed duplicates from mobile re-uploads.
• Solid-color or near-static intro cards produce degenerate all-zero hashes that collide with everything. We skip frames whose hash has fewer than 10 set bits, treating them as uninformative.
• Store the signature, not the frames. Sixty-four bytes per video (eight 64-bit hashes plus four band columns) is nothing. We never keep the extracted PNGs; they exist only in memory during ingestion.
• Run it inline, cache the verdict. Because the lookup is a few milliseconds, we do it during ingestion and write a duplicate_of pointer. The read path never recomputes anything, and Cloudflare caches the resulting feed pages happily.

Conclusion

Perceptual hashing is one of those techniques that feels like it should require a machine learning model and a GPU cluster, and instead comes down to shrinking an image to an 8x8 grid and counting brightness gradients. The three pieces that make it production-grade are picking the right per-frame algorithm (dHash for compression robustness), sampling frames by percentage so intros do not shift the sequence, and banding the hashes so lookups stay indexed instead of collapsing into an O(n²) scan.

We run exactly this pipeline behind DailyWatch: Python workers hash incoming videos with ffmpeg and Pillow, write compact signatures into SQLite, and a PHP 8.4 backend answers "seen this before?" in a couple of milliseconds during ingestion. The feed stopped repeating itself, our storage barely moved, and the whole thing is a few hundred lines of very boring, very fast code. If you are shipping any kind of user-facing video or image feed, build the dedup layer before the duplicates build themselves a reputation in your product.