使用Selenium Python实现网页的无限滚动。

huangapple go评论107阅读模式
英文:

Infinite scrolling of webpage using Selenium Python

问题

我想滚动直到所有元素都加载完毕。我的代码在前面7到8页运行良好,使用以下代码。

  1. driver.get('https://www.bigbasket.com/ps/?q=rice')
  2. last_height = driver.execute_script("return document.body.scrollHeight")
  3. while True:
  4. # 向下滚动到底部
  5. driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
  6. # 等待页面加载
  7. WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, '//div[@class="show-more"]/button')))
  8. driver.find_element_by_xpath('//div[@class="show-more"]/button').click()
  9. # 计算新的滚动高度并与上次滚动高度进行比较
  10. new_height = driver.execute_script("return document.body.scrollHeight")
  11. if new_height == last_height:
  12. break
  13. last_height = new_height

尽管不是所有元素都加载完毕,但最后的高度与新的高度相等。我需要滚动直到所有元素都加载完毕。

英文:

I want to scroll till all the element is loaded. My code works well for starting till 7 8 pages using this code.

  1. driver.get('https://www.bigbasket.com/ps/?q=rice')
  2. last_height = driver.execute_script("return document.body.scrollHeight")
  3. while True:
  4. # Scroll down to bottom
  5. driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
  6. # Wait to load page
  7. WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, '//div[@class="show-more"]/button')))
  8. driver.find_element_by_xpath('//div[@class="show-more"]/button').click()
  9. # Calculate new scroll height and compare with last scroll height
  10. new_height = driver.execute_script("return document.body.scrollHeight")
  11. if new_height == last_height:
  12. break
  13. last_height = new_height

The last height became equal to new height even though not all the element is loaded. I need to scroll till all element is loaded.

答案1

得分: 0

  1. 要[滚动](https://stackoverflow.com/a/54689339/7429447)到页面底部并不断点击<kbd>显示更多</kbd>按钮,您可以使用以下[_定位策略_](https://stackoverflow.com/questions/48054321/of-the-many-findelements-by-functions-in-selenium-when-would-you-use-one-over):
  2. - 代码块:
  3. driver.get("https://www.bigbasket.com/ps/?q=rice")
  4. while True:
  5. try:
  6. driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
  7. print("浏览器滚动")
  8. driver.execute_script("arguments[0].click();", WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "//button[text()='显示更多']"))))
  9. print("点击显示更多按钮")
  10. time.sleep(3)
  11. except WebDriverException:
  12. print("不再滚动")
  13. break
  14. - 控制台输出:
  15. 浏览器滚动
  16. 点击显示更多按钮
  17. 浏览器滚动
  18. 点击显示更多按钮
  19. 浏览器滚动
  20. 点击显示更多按钮
  21. ...
  22. ...

请注意,上述内容中的链接将保持原样,不进行翻译。

英文:

To scroll down till the bottom of the page and keep on clicking the <kbd>Show More</kbd> button you can use the following locator strategy:

  • Code block:

    1. driver.get(&quot;https://www.bigbasket.com/ps/?q=rice&quot;)
    2. while True:
    3. try:
    4. driver.execute_script(&quot;window.scrollTo(0, document.body.scrollHeight);&quot;)
    5. print(&quot;Browser scrolled&quot;)
    6. driver.execute_script(&quot;arguments[0].click();&quot;, WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, &quot;//button[text()=&#39;Show More&#39;]&quot;))))
    7. print(&quot;Show More button clicked&quot;)
    8. time.sleep(3)
    9. except WebDriverException:
    10. print(&quot;No more scrolling&quot;)
    11. break
  • Console Output:

    1. Browser scrolled
    2. Show More button clicked
    3. Browser scrolled
    4. Show More button clicked
    5. Browser scrolled
    6. Show More button clicked
    7. ...
    8. ...

huangapple
  • 本文由 发表于 2023年6月15日 00:32:25
  • 转载请务必保留本文链接:https://go.coder-hub.com/76475750.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定