DEV Community

suren40
suren40

Posted on

Python Infinite Scrolling problem.

I do web scraping on my daily basis. I was looking for the way to deal with the infinite scrolling using Selenium Python. One of the method I used while dealing with infinite scrolling data from chrome web store was to loop around the screen size.

        number = 0
       number2 = 2500
    for i in range(105):
        script_scroll = "window.scrollTo("+str(number)+","+str(number2)+")"
        print(script_scroll)
        contents = browser.find_elements_by_xpath("//div[@role='row']")
        urls_list = ['urls']

        for content in contents:
            link = content.find_element_by_xpath(".//a")
            link_url = link.get_attribute('href')
            urls_list.append(link_url)
        time.sleep(4)
        browser.execute_script(script_scroll)
        number = number2
        number2 = number2 + number2

Top comments (0)