DEV Community

Scraping in production Series' Articles

Back to Andrii Votiakov's Series
Google Trends will 429 you into the ground — here's how I got past it
votiakov profile

Google Trends will 429 you into the ground — here's how I got past it

#webscraping #python #javascript #seo
3
Comments 3
5 min read
Stop parsing HTML to get product data — the JSON is already in the page
votiakov profile

Stop parsing HTML to get product data — the JSON is already in the page

#webscraping #ecommerce #javascript #api
2
Comments 2
4 min read
Reading Google Play and App Store reviews straight from their JSON, no browser
votiakov profile

Reading Google Play and App Store reviews straight from their JSON, no browser

#webscraping #javascript #api #node
Comments 2
5 min read
Most company job boards are just a public JSON API you can GET
votiakov profile

Most company job boards are just a public JSON API you can GET

#webscraping #api #javascript #jobs
1
Comments
5 min read
Stop writing a parser per site. Run five and let confidence decide.
Cover image for Stop writing a parser per site. Run five and let confidence decide.
votiakov profile

Stop writing a parser per site. Run five and let confidence decide.

#webscraping #javascript #webdev
8
Comments
3 min read
Your scraper probably doesn't need a browser. It needs the right request.
Cover image for Your scraper probably doesn't need a browser. It needs the right request.
votiakov profile

Your scraper probably doesn't need a browser. It needs the right request.

#webscraping #javascript #api
Comments 1
3 min read
Anti-bot without melting your budget: the proxy waterfall.
Cover image for Anti-bot without melting your budget: the proxy waterfall.
votiakov profile

Anti-bot without melting your budget: the proxy waterfall.

#webscraping #devops #python
1
Comments
4 min read
Scraping millions of pages a day: what actually breaks
Cover image for Scraping millions of pages a day: what actually breaks
votiakov profile

Scraping millions of pages a day: what actually breaks

#webscraping #distributedsystems #devops #python
3
Comments 6
4 min read
My pipeline parsed every page twice, and I blamed the parser
Cover image for My pipeline parsed every page twice, and I blamed the parser

Summer Bug Smash: Smash Stories 🐛🛹
votiakov profile

My pipeline parsed every page twice, and I blamed the parser

#devchallenge #bugsmash #webscraping #performance
Comments
5 min read
Half the web already hands you clean product data. You just have to ask.
Cover image for Half the web already hands you clean product data. You just have to ask.
votiakov profile

Half the web already hands you clean product data. You just have to ask.

#webscraping #seo #javascript #webdev
Comments
4 min read