DEV Community

Discussion on: Scraping HTML with PHP Node and Puppeteer

Collapse
sticklight profile image
sticklight

Great article, found it really helpful :)
I was wondering though - how could I use what you've shown here with a website in which the data is generated with infinite-scrolling; you have any idea?

Collapse
alanmbarr profile image
Alan Barr Author • Edited

The easier thing to do would be to first check if you can use their api first if it is exposed in some way even if you have to login with authentication and use the api to page.

If you for some reason you cannot because the data is rendered from multiple apis you would need to potentially call more JavaScript calls on the page. Basically simulating scrolling as a user would. Maybe there is some element that is at the bottom that triggers the next page load you would need to scroll to that spot and force the next load. Repeat.

Collapse
sticklight profile image
sticklight

Thanks for the quick response!
Yeah, that could work - I found which element triggers the loading but how can I possibly simulate scrolling?

Thread Thread
alanmbarr profile image
Alan Barr Author

Let's take this conversation off here and reach out to me