DEV Community

Discussion on: Scraping 127,000 Lines with Python

Collapse
 
fetchworkglenn profile image
Info Comment hidden by post author - thread only visible in this permalink
Glenn

By automatically, do you mean you're trying to figure out how to have it go endlessly until you stop getting results?

If yes, maybe a simple stars check that sees if there are any 'stars dividers' on the page that was returned. If there aren't any, then simply exit out of the loop.

Some comments have been hidden by the post's author - find out more