What I Built
Crawaler is an open-source, AI-powered price comparison platform for Pakistani e-commerce.
It searches products across Daraz and Telemart, groups identical products together, and displays prices side by side so users can instantly find the best deal without opening multiple tabs.
The platform also supports AI image search. Users can upload a product photo, and Google Gemini identifies the product and automatically performs the search across supported stores.
Key Features
- Real-time product search across multiple stores
- AI-powered image search using Google Gemini
- Side-by-side price comparison
- Smart filtering and sorting
- User authentication with OTP-based password recovery
- Admin dashboard for user and crawler management
- Fully responsive design
🔗 Live Link: https://crawaler-front-v2.vercel.app
Screenshots
Tech Stack
| Layer | Technologies |
|---|---|
| Frontend | React, Vite, TypeScript, Tailwind CSS, shadcn/ui |
| Backend | Node.js, Express, Puppeteer, Cheerio |
| Database | MongoDB, Mongoose |
| AI | Google Gemini |
| Authentication | JWT, bcrypt, Nodemailer |
Who It Helps
Online buyers often spend a lot of time checking different websites to compare prices. Crawaler solves this problem by bringing everything into a single platform.
For Buyers
- Save money by instantly finding the best available price.
- Save time by avoiding manual comparisons across multiple websites.
- Search using images instead of product names.
- Compare identical products with confidence.
The Comeback Story
This project started as a side project that remained unfinished for months.
When I returned to it, the Daraz integration was already working, but a comparison platform needs multiple sources to provide real value. Finding and integrating a reliable second platform became the biggest challenge.
Attempt 1: OLX
I first experimented with OLX and managed to extract some data, but maintaining a stable integration proved difficult. Eventually, I decided to move on.
Attempt 2: Temu
Next, I explored Temu. Their APIs and requests were heavily protected and obfuscated, making reliable integration extremely challenging. After several attempts, I abandoned this approach.
Attempt 3: Telemart
Finally, I turned to Telemart. While it exposed API, access tokens expired frequently, causing the scraper to fail during execution.
To solve this, I implemented automatic token refresh handling, allowing Crawaler to detect expired tokens and recover seamlessly. This became the reliable second data source the platform needed.
The Final Piece: AI Image Search
With comparison functionality complete, I added the feature I had wanted from the beginning: AI-powered image search.
Buyers can upload a product image, Google Gemini identifies the product, and Crawaler automatically searches and compares prices across supported platforms.
My Experience with GitHub Copilot
GitHub Copilot played an important role in helping me finish the project.
It assisted with debugging scraper issues, improving reliability, generating boilerplate code, refining React components, and accelerating development when I was stuck on implementation details.
Copilot helped me focus more on solving complex engineering challenges while spending less time on repetitive tasks. It was especially useful while refining scraping logic, handling token refresh workflows, and improving overall code quality.
Final Thoughts
Crawaler transformed from an unfinished side project into a fully functional platform that helps buyers save both time and money.
The GitHub Finish-Up-A-Thon Challenge provided the motivation I needed to finally complete something that had been sitting unfinished for months. I'm excited to continue improving the platform, add more store integrations, and welcome contributions from the open-source community.
Top comments (1)
The AI image search is a smart addition that actually improves user experience, not just a gimmick.