DEV Community

ChRainG
ChRainG

Posted on

I Built a Photo Management SaaS with FastAPI, Qdrant, CLIP and Automatic Duplicate Detection

#python #saas #ai #webdev

Managing a few hundred photos is easy.

Managing 20,000, 50,000 or 100,000 photos is a different problem.

At some point you stop remembering where a photo is stored, whether you already have another copy of it, or why there are 15 almost identical images from the same moment.

That problem is what led me to build Renvumi.

Renvumi is a web service for analyzing and organizing large photo libraries.

It is available here:

https://renvumi.ru/

The interface is now available in both English and Russian.

The original problem

The first version was supposed to do one thing:

find duplicate photos

Exact duplicates are relatively easy.

If two files have the same content, a cryptographic hash can identify them.

But real photo libraries are much messier.

You often have:

  • resized copies
  • recompressed JPEGs
  • edited versions
  • burst shots
  • nearly identical frames
  • the same photo stored in several albums
  • files with identical names but completely different contents

So exact hashing solved only a small part of the problem.

Exact duplicates

For exact duplicates, Renvumi uses file hashes.

This is useful when several backup drives or old folders have been merged together.

If two files have identical content, they can be grouped immediately without expensive AI processing.

Near duplicates

Near duplicates are more interesting.

Two files can look almost identical but have different binary contents.

For this case I use perceptual hashing.

This works well for things like:

  • resized images
  • recompressed images
  • small edits
  • slightly modified copies

But perceptual hashes are not enough for broader semantic similarity.

Similar image search with CLIP

For more flexible similarity search, Renvumi uses CLIP embeddings.

Each analyzed image gets a vector representation.

Those vectors are stored in Qdrant.

This makes it possible to search for visually related photos even when they are not near duplicates.

For example, you can select one photo and search for similar photos across the entire library.

That can work for:

  • the same car photographed at different times
  • the same building
  • similar landscapes
  • related scenes
  • different photos from the same event

Renvumi uses absolute CLIP cosine similarity for this type of search.

Perceptual hashes are kept for near-duplicate detection.

I intentionally separated these two concepts because they solve different problems.

Text search

CLIP also makes text-to-image search possible.

Instead of remembering filenames, you can search for something like:

dog on the beach
Enter fullscreen mode Exit fullscreen mode

or:

red car near a house
Enter fullscreen mode Exit fullscreen mode

The system converts the query into the same embedding space and searches the existing image vectors.

This is one of the features that makes a large photo archive feel less like a filesystem and more like a searchable database.

Global search across albums

Originally, every operation was scoped to one album.

That quickly became a limitation.

Imagine this:

Phone Backup/
    IMG_1001.JPG

Old HDD/
    IMG_1001.JPG

Vacation/
    copy_IMG_1001.JPG
Enter fullscreen mode Exit fullscreen mode

The same image may exist in several places.

So Renvumi now has a Global mode.

It aggregates results from all analyzed albums.

From there users can work with:

  • exact duplicates
  • near duplicates
  • similar photos
  • unique photos
  • people
  • best shots
  • cleanup candidates
  • filename conflicts

The original album name is still shown for every result.

Same filename, different file

This turned out to be a surprisingly useful feature.

Cameras and phones reuse filenames over time.

After several years, a library may contain many files called:

IMG_0001.JPG
Enter fullscreen mode Exit fullscreen mode

But they may all be different photos.

Renvumi detects this separately.

The rule is simple:

same filename + different SHA-256 = filename conflict
Enter fullscreen mode Exit fullscreen mode

If both filename and content are identical, it is treated as an ordinary exact duplicate instead.

Choosing the best shots in a series

Another common problem is burst photography.

You may have 10 or 20 images taken within a few seconds.

They are all different files, but from a user's perspective they form one series.

Renvumi groups related shots and helps identify stronger candidates.

It can use signals such as:

  • sharpness
  • face detection
  • closed-eye detection
  • visual similarity

But I deliberately do not let the software automatically decide what the user must delete.

A technically imperfect photo can still be the best photo emotionally.

So the system narrows the set.

The user makes the final decision.

Face detection and people grouping

For people detection I use OpenCV components including YuNet and face recognition logic.

The pipeline also contains additional checks to reduce obvious false positives.

The goal is to group images of the same person across an album and then aggregate those results globally.

This part turned out to require much more tuning than I initially expected.

Face detection itself is easy.

Reliable grouping across thousands of real photos is not.

Automatic analysis of new uploads

One workflow decision changed recently.

Previously the process was:

upload photos
click Analyze
wait
Enter fullscreen mode Exit fullscreen mode

That felt unnecessary.

A photo analysis service without analysis is not very useful.

Now the workflow is:

upload photos
analysis starts automatically
Enter fullscreen mode Exit fullscreen mode

But there is an important optimization.

Only new, not-yet-analyzed photos are processed.

For example:

Day 1:
Upload 5,000 photos
Analyze 5,000 photos

Day 7:
Upload 100 photos
Analyze only those 100 photos
Enter fullscreen mode Exit fullscreen mode

The existing 5,000 images are not analyzed again.

This saves GPU time and user quota.

If another upload happens while analysis is already running, the new photos remain pending and are picked up by the next analysis job.

Background processing

Image analysis is expensive enough that I did not want HTTP requests doing the heavy work.

Renvumi uses Celery with Redis for background jobs.

Different types of work are separated.

For example:

  • GPU analysis
  • import operations
  • maintenance operations
  • scheduled jobs

The GPU worker currently runs with concurrency set to one.

That prevents multiple large model workloads from fighting for GPU memory.

Interactive work also gets higher queue priority than large background analysis jobs.

Avoiding O(n²) comparisons

One of the less glamorous problems appears when a photo library becomes large.

A naive pairwise comparison of all images grows very quickly.

For 10,000 photos:

10,000 × 9,999 / 2
Enter fullscreen mode Exit fullscreen mode

is almost 50 million pairs.

That is not something I wanted to do every time a user opened a result page.

So candidate generation happens first.

For near-duplicate processing I use perceptual-hash-based indexing to narrow the candidate set before applying more expensive comparisons.

This was one of the biggest performance improvements for the global archive mode.

The backend stack

The current backend includes:

FastAPI
PostgreSQL 16
Redis
Celery
Qdrant
OpenCV
CLIP
LibreTranslate
Docker Compose
Caddy
Enter fullscreen mode Exit fullscreen mode

The entire service runs in Docker.

Caddy handles HTTPS.

PostgreSQL stores application data.

Qdrant stores image embeddings.

Redis handles Celery, caching and temporary runtime state.

Why LibreTranslate?

Renvumi supports text-to-image search.

But CLIP works more predictably for my use case when queries are normalized into English.

So I run a private LibreTranslate container inside the Docker network.

For example:

жираф
Enter fullscreen mode Exit fullscreen mode

can be translated internally to:

giraffe
Enter fullscreen mode Exit fullscreen mode

before going to the search pipeline.

No external translation API is required.

The same internal translation system is also now used as part of the English localization workflow.

English and Russian interface

Renvumi originally started as a Russian-language product.

I recently added full language switching:

🇷🇺 Russian
🇬🇧 English
Enter fullscreen mode Exit fullscreen mode

The selected language applies to:

  • main UI
  • account pages
  • Help
  • API documentation
  • examples
  • UI messages
  • PWA metadata

The language is remembered in a cookie.

On the first visit, the service can use the browser language as the default.

Storage and safety

Photo management software has one dangerous feature:

it can delete photos.

That means recovery matters.

Renvumi uses a trash workflow rather than immediately destroying files.

There are also backup and restore mechanisms.

I have spent more time than expected on things like:

  • interrupted jobs
  • database pool exhaustion
  • backup consistency
  • recovery
  • storage encryption
  • failed analysis
  • user quotas
  • long-running imports

These parts are much less exciting than AI models, but they are what make the service usable in practice.

API

Renvumi also has an external API for paid plans.

The API documentation includes examples for:

cURL
Python
PowerShell
Enter fullscreen mode Exit fullscreen mode

I wanted the API docs to describe not just the endpoint name, but also:

  • what the function does
  • parameters
  • returned data
  • request example
  • response example

The documentation is available in both English and Russian.

The infrastructure is currently self-hosted

One unusual part of the project is that Renvumi currently runs on my own server.

The machine has:

  • 72 CPU threads
  • 256 GB RAM
  • NVIDIA GPU

This gives me much more control over GPU workloads than using an expensive managed inference API for every image.

It also makes performance tuning interesting because I can see exactly where the bottlenecks are.

What I learned building it

One thing became clear fairly quickly:

AI was not the hardest part.

Getting an embedding from CLIP is easy.

Building a real service around it is much harder.

The difficult parts were things like:

  • making large jobs resumable
  • not analyzing the same photo twice
  • keeping interactive searches responsive
  • limiting database connections
  • cleaning up thousands of images safely
  • keeping backup operations isolated
  • handling users with large libraries
  • making analysis results understandable

The model is only one component.

The product is everything around it.

Current status

Renvumi is live:

https://renvumi.ru/

The free tier currently includes:

1,000 analyses
1 GB storage
Enter fullscreen mode Exit fullscreen mode

My main goal now is not to keep adding features.

I want to see how the system behaves on real photo libraries.

Especially:

10,000 photos
50,000 photos
100,000+ photos
Enter fullscreen mode Exit fullscreen mode

Test datasets are useful, but real archives always contain weird cases you did not expect.

I would like feedback

If you have a large photo library, I would be interested in hearing how you manage it today.

Do you use:

  • Lightroom?
  • Google Photos?
  • Apple Photos?
  • local folders?
  • a NAS?
  • custom scripts?
  • something else?

And what would stop you from using a service like Renvumi?

For example:

  • privacy
  • upload speed
  • cloud storage
  • price
  • wanting a desktop version
  • not trusting automatic similarity detection

Those answers are probably more valuable to me right now than another feature request.

Renvumi:

https://renvumi.ru/

Top comments (0)