DEV Community

Discussion on: Introduction to web scraping with Node.js

Collapse
 
akb20 profile image
Aaron

Awesome tutorial. Thanks!
Is there an easy way to automate this so when you publish new posts the changes will be reflected in your app?

Collapse
 
aurelkurtula profile image
aurel kurtula

Yes, you could set a timer when you want the scrapping to happen. For example if I wanted to scrape my articled I have it run Thursday and Sunday. If I was scraping twitter I'd have it run every 15 minutes (running it every second would be costly).

If I remember correctly this is how ifttt tasks work. In fact I know because I tried it years back, they don't update when the new content is published but every how ever many minutes/hours.