E-Books Web Crawling Application with Recommendation System

My Final Project

Finally finished my last semester of my degree, this article is about my final year project (FYP). It is an mobile application that allows users to search for the web crawled e-books from a few websites. All the e-books are free to download. There are also recommendation systems to recommend users some e-books.

How I built it ?

There are 4 main components for this FYP as shown below:

Web Crawler

3 e-books websites are being scraped by using the following Python technologies:

1. Scrapy
2. Scrapyd
3. Pymongo

---

Mobile Application

The mobile application is being built with Flutter. Here are just a few UI designs.

^this is the Library Screen^

^this is the Explore Screen^

^this is the Story Screen^

^this is the Profile Screen^

---

Back-end for Mobile Application

The back-end is used to handle the actions of the mobile application such as login, register, search for books and etc.
It is built with Node.js + MongoDB.

---

Recommendation Systems

The recommendation systems are being built with the following Python technologies:

1. flask
2. pandas
3. numpy
4. gensim
5. sklearn

There are something one might need to take care for is the cold start problem of the recommendation system. In order to solve it, the users will be prompted to select e-books and genres that they like.

There are 2 recommendation systems, the first one is a content-based recommender after users have selected their favourite e-books and genres.
The 2nd one is being used for collaborative filtering (CF) when users have rated a few e-books. The 2nd one is based on k-NN algorithm.

---

Website

I've also created a website with React to promote this application and for users to download the apk file.
Note: The website may take quite a while to load due to inactivity.
New Updates: The backend server has been shut down to save money, so the application is not working anymore.

---

Additional Thoughts / Feelings / Stories

In short, I am really happy that I am able to complete my FYP. Hopefully that I could get a really good grade for it.

Thank you for reading !!

Comments

[Crawlbase]

Wow! Congrats on wrapping up your final year project! Your e-books web crawling app with a recommendation system sounds awesome. Using Scrapy, Scrapyd, and Pymongo shows serious tech skills. If you're up for it, Crawlbase could add some extra oomph to your project with its proxy rotation feature.