Funny to bump into your post and to realise that I went to the exact same steps a few months ago!
I didn't write a blog post (though I should have, because it really helps) but after some time playing with Axios and Cheerio and having to face more complex use cases, I eventually decided to create my own library: github.com/mawrkus/jason-the-miner
It's modular, has simple yet powerful schema definitions (including following/paginating at any level) and is extensible.
My experience developing Jason was (and still is) fun, challenging and full of surprises... Starting to scrape is really easy but can get complicated really fast ("What? This is not the content I see with my browser!" "Ach! they blocked my IP, I need a Ninja HTTP client!"), which makes this kind of project, a perfect way to learn Node.js.
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
Funny to bump into your post and to realise that I went to the exact same steps a few months ago!
I didn't write a blog post (though I should have, because it really helps) but after some time playing with Axios and Cheerio and having to face more complex use cases, I eventually decided to create my own library: github.com/mawrkus/jason-the-miner
It's modular, has simple yet powerful schema definitions (including following/paginating at any level) and is extensible.
My experience developing Jason was (and still is) fun, challenging and full of surprises... Starting to scrape is really easy but can get complicated really fast ("What? This is not the content I see with my browser!" "Ach! they blocked my IP, I need a Ninja HTTP client!"), which makes this kind of project, a perfect way to learn Node.js.