"Why hit API request limits for 111,000 books when a single log10 formula and direct web scraping can solve the problem?"
Hey guys ;)
Looks like my focus is completely locked on data analysis right now. I can’t help it-this project is just too fun.
Over the past two or three posts, I kept promising that I would enrich my book dataset using the Google Books API and OpenLibrary. Well... forget about the APIs.
After wrestling with rate limits, poor metadata quality, and HTML structures all week, I dropped the APIs entirely and built my own data enrichment pipeline. Here is how it happened, why I deleted 1,451 scraped books today, and what’s coming next.
The Problem with Books APIs
When I started planning the data enrichment phase for my 111,000 unique scraped books, I ran into two massive brick walls:
- Google Books API Limits: The free tier caps you at 1,000 requests per day. At that rate, enriching 111k books would take over three months (I know about payment account and increasing of limits).
- OpenLibrary Limitations: It handles Ukrainian literature poorly without exact ISBN matches. Title and Author search combinations gave way too many false positives.
I quickly realized I didn't actually need all 111,000 books. Thousands of them had zero reviews, no ratings, and no active search interest. Processing them was just adding noise.
Filtering the Catalog and Popularity Formula
To isolate the most relevant titles, I designed a custom metric to score and rank the entire catalog:
popularity_rating = log10(amount_of_reviews + 1) * rating
Using the logarithmic scale prevents titles with hundreds of reviews from completely breaking the ranking curve while still giving heavy weight to actual reader engagement.
This formula immediately isolated ~3,000 highly active titles. I expanded the threshold to capture the top 6,000 books for the final analytical sample.
Parsing Metadata Directly
To analyze publishing trends over time, I needed the exact publication year for a specific edition, not just the general first release year of a work.
To get that accuracy from APIs, I would need to parse the book's individual URL anyway to extract its ISBN. That's when the realization hit me: If I'm already scraping the specific book pages for ISBNs, why not just scrape the missing metadata directly from Yakaboo?
So from Monday until yesterday, I wrote a dedicated secondary parser to process my top 6,000 URL list, extract extended properties, and map them into structured category buckets.
Here is my best books selection code
Here is scraper fragment
And how I deal with categories for final dataset preprocessing
The HTML Trap and deleting 1,451 Books
It wouldn't be a real project without a good mistake.
While running the extended parser, I noticed that Yakaboo's HTML structure reuses generic CSS classes across different metadata blocks while hiding or omitting unique element IDs. Because of a misplaced selector logic, my parser was extracting Author names and saving them into the Publisher field. 😅
After 1,451 books were parsed, I caught the bug, wiped the corrupted JSON file, and fixed the DOM selector.
Today, I’m re-running the script on the top 6,000 clean URLs. The most important lesson in Data Engineering: never give up, even when you have to wipe your output and start over!
Summary & What's Next
- GitHub: All changes are finally committed and pushed! I'm much more cautious here than in my Chess project, pushing only when the pipeline logic is 100% verified.
- Current Task: Re-running the extended parser for the top 6k books without the Author/Publisher swap bug.
- Next Goal: Data cleaning, category normalization, and generating our first analytical charts to answer if people are reading fewer books over time.
Have you ever built a custom metric to downsample a massive dataset? What’s your worst HTML scraping horror story? Let’s chat in the comments! 👇
#python #datascience #webscraping #pandas #showdev
Top comments (0)