Introduction
Managing data across Appwrite projects can become difficult when duplicate files and documents take up storage and make data harder to manage.
This is why I built Appwrite AI Duplicates Detector (AADD), a full-stack web application that detects, visualizes and manages duplicates across Appwrite databases and storage using AI-powered algorithms and perceptual hashing.
The goal was to make the entire process simple, from connecting an Appwrite project and scanning data to understanding results and managing duplicates through filtering, sorting and bulk actions.
I also introduced AI Garden, a gamified data-health system that lets users track their cleaning progress through a dynamic garden and interact with an AI Gardener.
My Role: UI/UX Design, Frontend Development, Backend Development
Tech Stack: Next.js, React, TypeScript, Tailwind CSS, shadcn/ui, Framer Motion, Flask, Appwrite, Google Gemini API
Live App: https://appwrite-ai-duplicates-detector-aadd.appwrite.network/
The Problem
Managing duplicates manually across Appwrite projects can be time-consuming, especially when data is spread across multiple databases and storage buckets.
Different types of files also require different ways of identifying duplicates. Simply checking file names or exact matches is not enough to detect files that may have been renamed, compressed or slightly modified.
AADD was built to simplify this process by bringing duplicate detection, result visualization and duplicate management into one place.
Existing Solutions & Gap
Before building AADD, I looked at the problem from two sides.
Appwrite provides the tools to manage databases, collections, documents and storage, including querying, indexing and file management. However, duplicate detection itself is not presented as a dedicated workflow within these core features.
There are also dedicated duplicate-file tools such as dupeGuru, Duplicate Cleaner, AllDup and Czkawka. These focus mainly on finding duplicate files across local storage using methods such as hashing, content comparison and similar-image detection.
The gap I wanted to explore was bringing this idea specifically to Appwrite projects.
AADD combines duplicate detection for both Appwrite Storage and Databases with similarity-based analysis, visual results, filtering, bulk management and direct cleanup of detected duplicates.
This became the foundation for the product I designed and built.
My Approach
Before starting the development, I planned the user flows and page architecture to understand how the different parts of AADD would fit together.
The main focus was to keep the duplicate detection process simple for the user, while still providing enough information to understand and manage the results.
I structured the experience around three main stages:
Connect: Connect an Appwrite project and choose what needs to be scanned.
Detect: Run the scan and present the detected duplicates with relevant information such as similarity scores and visualizations.
Manage: Review, filter and sort the results, then select and manage duplicates through bulk actions.
Alongside this core workflow, I also designed supporting experiences such as scheduled scans, activity tracking, profile management and the AI Garden.
Information Architecture
AADD is structured around managing connected Appwrite projects, scanning their databases and storage buckets, viewing duplicate results and managing detected duplicates.
The main sections of the application include:
Dashboard
- Connected Projects
- Quick Stats
- Automated Scan Reminders
- Recent Activities
Project Overview
- Storage
- Database
- Collections
- Scan
Scan Results
- Detected Duplicates
- Similarity Scores
- Search
- Filtering
- Sorting
- Circle Packing Visualizations
- View in Console / Open in Appwrite
- Bulk Operations
AI Garden
- Data Health Visualization
- Cleaning Statistics
- AI Gardener
Profile Management
- Update Name and Email
- Profile Picture
- Delete Account
Activity Log
- Project Connections
- Scan Operations
- Deletion Activities
- Configuration Changes
This structure brings project management, duplicate detection, visualization and cleanup into a single application.
Connecting an Appwrite Project
To start using AADD, users first need to connect the Appwrite project they want to scan.
The Connect Project page requires three details:
- Project ID
- API Endpoint
- API Key
Once connected, the project becomes available on the dashboard, where users can view their connected projects and initiate scans.
For security, the API keys provided by users are encrypted using Fernet symmetric encryption before being stored in the AADD Appwrite Database.
Scanning & Selecting Data
After connecting an Appwrite project, users can choose what they want AADD to scan.
For Storage, users can scan all available buckets.
For Database, users can enter a Database ID, load its collections, and choose whether to scan specific collections or the entire database.
Once the scan target is selected, clicking the Scan button automatically triggers the scan and takes the user to the results page.
A loading state is displayed while the scan is in progress, giving users feedback while the system processes their data.
Viewing & Understanding Results
Once the scan is complete, AADD displays the detected duplicates along with their similarity scores.
The results page also provides different ways to explore the detected duplicates. Users can search for specific results, filter and sort them based on similarity, date or file size and view the duplicate distribution through different visualizations.
Each result also provides options to view the corresponding file or document in the Appwrite Console.
Managing Duplicates
After reviewing the scan results, users can select individual duplicates or use Select All and Deselect All for bulk operations.
The Delete Selected option provides two ways to manage the selected duplicates:
- Delete from source: Removes the actual files or documents from the connected Appwrite project.
- Remove from list: Removes the duplicates only from AADD's tracking without deleting them from the source project.
Users can then confirm the selected action before it is carried out.
AI Garden & Gamification
To make the data-cleaning experience more engaging, AADD includes the AI Garden, a gamified data-health system.
The page provides a visual representation of data health through dynamic plant visualizations and tracks cleaning progress through statistics.
It also includes an AI Gardener, powered by Google Gemini, which provides tips and encouragement based on the user's progress.
From Design to Development
After planning the user flows and page architecture, I moved into development and translated the planned experience into the actual product.
The frontend was built using Next.js, React, TypeScript, Tailwind CSS and shadcn/ui, with Framer Motion used for animations and interactions.
The backend was developed using Flask, handling API requests, Appwrite operations and the duplicate detection logic.
The final system connects the frontend, backend, Appwrite services and AI/ML processing into one workflow, allowing users to connect their projects, scan data, view results and manage duplicates from a single application.
Outcome & Learnings
AADD brought duplicate detection, visualization and management into a single application for Appwrite projects.
The system achieved 85-95% similarity accuracy and reduced the manual effort required to review duplicates by approximately 70%.
The project was also selected as a Top 5 Winner in Appwrite X Hacktoberfest 2025.
Building AADD helped me understand how important it is to plan the user flows and page structure before development, especially when a product has multiple features and workflows.
It also gave me hands-on experience in taking an idea from rough sketches to a working full-stack product while continuously refining the interface and functionality during development.
Thanks for reading!😊🤍
Top comments (0)