A picture is worth a thousand words - Image Search with Deep Learning

I'm scheduled for graduation in this October, but hey DEV + Github offers me to graduate early. Such compelling offers!

Final Project

For this short final project, we —in my department, the final project can be done by team, by the way— avoid "information system" topics that unquestionably comes with a myriad of sprints. But don't get me wrong, I strongly admit, I was greatly helped by information systems.

After going through a long discussion, our choice falls on * drums roll *
<h1 style="color:red; text-transform: uppercase; text-decoration: underline overline;"> Deep Learning </h1>

Well, the next step is to determine the background of the problem that we will solve with our solution. In our country —Indonesia, e-commerce is growing rapidly with the clothing category is being one of the best-selling items.

One of the most important services in e-commerce is a search service. E-commerce users will search for products through a search service. Searching is usually done by text-based keywords. However, the ability of each person to describe the product in the text is different. Hence, that often causes the user not to get appropriate search results.

Seeing that opportunity, we chose the topics Content-Based Image Retrieval with deep learning for clothing commerce. Therefore, we open up new ways for users to search for clothes from various e-commerce sites using images as input. Isn't proverb says, "A picture is worth a thousand words" ?
So just because "World is visual, now search is too!" —if you are Bing enthusiasts, I'm sure you have familiar with that tagline.

How we built it (the stack, issues, something new discovered along the way)

As we know, Convolutional Neural Network has become a reliable Deep Learning method for computer vision.

Object Detection

We applied Mask Region-based Convolutional Neural Network (Mask R-CNN) for object detection. Mask R-CNN can detect more than one object in one image, each object is marked with a class label (object name), bounding box, and object mask.

Feature Extraction

We use VGG16 architecture to extract images feature, because VGG16 is more accurate to extracts object information and thus will improve search performance.

Dataset

DeepFashion2 is a comprehensive fashion dataset. It contains 491K diverse images of 13 popular clothing categories from both commercial shopping stores and consumers. It totally has 801K clothing items, where each item in an image is labeled with scale, occlusion, zoom-in, viewpoint, category, style, bounding box, dense landmarks, and per-pixel mask. There are also 873K Commercial-Consumer clothes pairs. The dataset is split into a training set (391K images), a validation set (34k images), and a test set (67k images). It developed by Yuying, et al from The Chinese University of Hongkong.

Image Similarity

We use Cosine Similarity as a method to calculate image similarity

Data Test

Ecommerce that we used as a test are 3 of the top 4 e-commerce sites in Indonesia, there are Zalora, Mapemall, and Berrybenka. We obtain an e-commerce product by crawling. The tools are Scrapy and Selenium.

Link to Code

The project is not already done yet. We still keep up our good work on it!
To catch sight of, feel free to visit our repositories:

KoTA208-2017 / crawling-ecommerce

Crawling data from e-commerce site using scrapy and selenium

crawling-ecommerce

Crawling data from several e-commerce site using scrapy and selenium webdriver

View on GitHub

KoTA208-2017 / client-side-image-search

Client side application of image search for women's clothing products in e commerce

client-side-image-search

Client side application of image search for women's clothing products in e commerce

View on GitHub

KoTA208-2017 / server-side-image-search

Server side application of image search for women's clothing products in e commerce

CBIR with Deep Learning

Instalation

1. Clone this repository
2. Install dependencies

   pip install -r requirements.txt

3. Run setup from the repository root directory

   python setup.py install

View on GitHub

Additional Thoughts / Feelings / Stories

For all final year students who are struggling to finish their final assignment in the midst of a pandemic, this is certainly not easy. Perform seminars online, equating the frequency of enthusiasm with the team remotely, ugh~ truly challenging. But, as Ryan Stone said in Gravity Movie "There are only two possible outcomes. Either I make it down there in one piece and I have one hell of a story to tell! Or I burn up in the next ten minutes. Either way whichever way, no harm no foul".

Come on me! Stop scrolling your Instagram and close your Youtube. The world awaits you for a bigger project.

Thanks DEV + GitHub for this cool event!