Selenium infinite scrolling to bottom using python

We have 3 kinds of infinite scrolling to bottom while using selenium.

1. Simply scrolling – Stop until there is no more content.
2. Meet scrolling – Stop until we meet a speciy element.
3. Partial scrolling – Scrolling content in specify html container, stop until there are no more content.

All these 3 scenses will use Javascript. Let’s assume we have a driver created like this:

driver = webdriver.Chrome()

Note: here we all use smooth to scroll the windows to make it more like a humman action.

1 Simply scrolling

def scroll_to_bottom(driver):
    old_height = driver.execute_script("return document.body.scrollHeight;")

    while True:
        driver.execute_script('document.body.scrollTo({top: document.body.scrollHeight, behavior: "smooth"})')

        # sleep randomly
        time.sleep(random.randint(1, 3))

        new_height = driver.execute_script("return document.body.scrollHeight;")
        if new_height == old_height:
            break

        old_height = new_height

Here, we use new_height == old_height to decide if we meet the bottom, which means the content doesn’t expand anymore.

2 Meet scrolling

def scroll_to_bottom(driver):
    # start to scroll until DOM <div id="content"> is displayed.
    # Will throw an exception after 20 seconds
    wait = WebDriverWait(driver, 20)
    wait.until(EC.visibility_of_element_located((By.XPATH, '//div[@id="content"]')))

    # start to scroll
    while True:
        driver.execute_script('window.scrollTo({top: document.body.scrollHeight, behavior: "smooth"})')

        # sleep randomly
        time.sleep(random.randint(1, 3))

        try:
            # stop scrolling until we meet the DOM <span class="bottom">
            driver.find_element(By.XPATH, '//span[@class="bottom""]')
            break
        except NoSuchElementException as e:
            # Keep scrolling
            continue

Partial scrolling

Here we only scroll the sidebar container, not the whole page.

def scroll_to_bottom(driver):
    sidebar_element = 'document.querySelector(".sidebar")'
    old_height = driver.execute_script(f"return {sidebar_element}.scrollHeight;")

    while True:
        driver.execute_script(f'{sidebar_element}.scrollTo({top: {sidebar_element}.scrollHeight, behavior: "smooth"})')

        # sleep randomly
        time.sleep(random.randint(1, 3))

        new_height = driver.execute_script(f"return {sidebar_element}.scrollHeight;")
        if new_height == old_height:
            break

        old_height = new_height