Hey guys ;)
Today I’m going to tell you about some progress in books analysis and my plans for the weekend. Let’s start.
So from my previous post, I’ve been working with the author translation problem. What is it? Okay, so for extending books data I should use some API, and for that I need an ISBN that will give a certain book with 99% accuracy, or use a Title + Author combination. When a book is from a Ukrainian publisher, it will find the book with 80-90% accuracy. But what happens if the book is from a foreign publisher? Well, in fact, the accuracy will definitely be less.
Another problem was the amount of unique books that I parsed from 16 categories. That amount surprised even me: 117,000 unique books and 63,000 unique foreign authors.
So before I found out the exact amount of unique books, I wrote a script based on the deep-translator library (Google Translator). My first few requests were done without a rate limit, so my IP got blocked. After that, I added time.sleep(2). Even after adding a dictionary for saving unique authors and reducing the number of requests, the execution time was too long - like 8 hours or more.
After that, I found an interesting library called argostranslate. The main difference is that it’s an offline library, and it works entirely on laptop resources. The process of translating took only 1h 30 min to 2 hours.
(Though someone forgot about saving! For Parquet files, Python needs the pyarrow library. So I installed pyarrow and will start the translation process on Monday.)
Okay, now about some plans. Of course, for book analysis, the next step after translation is working with an API. Now I’m thinking about a few variants:
- Google Books API: With an ordinary free account, I have only 1,000 requests per day. If I connect a payment account, I’ll get 50,000 requests.
- Open Library API: Free, but this API doesn’t have much info about Ukrainian publishers.
So the best option could be Google Books API. But that’s for next week!
Now it's the weekend, so I can rest, play games, and watch something interesting. Speaking of watching, first I’m going to watch some Django tutorials (because in the future I’ll definitely need it) and maybe something else.
Have a great weekend!
#python, #dataanalysis, #learning, #buildinpublic
Top comments (2)
Nice Read 👍🏻
Thank you 😊