Selenium通过Selenium保持循环导航（Python）

huangapple
go评论86阅读模式
英文:

Selenium navigation through selenium keep looping (python)

问题

我刚刚开始使用Selenium来从网页上抓取表格。因此，我使用Selenium实现了网页的导航。但是，当我运行代码时，结果一直在循环。我很确定我的代码写错了。我应该如何修复代码，以使Selenium导航正常工作？

  1. import requests
  2. import csv
  3. from bs4 import BeautifulSoup as bs
  4. from selenium import webdriver
  5. import time  # 添加导入时间模块
  7. browser = webdriver.Chrome()
  8. browser.get('https://dir.businessworld.com.my/15/posts/16-Computers-The-Internet')
  10. soup = bs(browser.page_source, 'html.parser')  # 指定解析器为'html.parser'
  12. filename = "C:/Users/User/Desktop/test.csv"
  13. csv_writer = csv.writer(open(filename, 'w', newline=''))  # 添加newline=''以防止行之间的额外空行
  15. pages_remaining = True
  17. while pages_remaining:
  18.     for tr in soup.find_all("tr"):
  19.         data = []
  20.         for th in tr.find_all("th"):
  21.             data.append(th.text.strip())
  22.         if data:
  23.             print("Inserting headers: {}".format(','.join(data)))
  24.             csv_writer.writerow(data)
  25.             continue
  27.         for td in tr.find_all("td"):
  28.             if td.a:
  29.                 data.append(td.a.text.strip())
  30.             else:
  31.                 data.append(td.text.strip())
  32.         if data:
  33.             print("Inserting data: {}".format(','.join(data)))
  34.             csv_writer.writerow(data)
  36.     try:
  37.         next_link = browser.find_element_by_xpath('//*[@id="content"]/div[3]/table/tbody/tr/td[2]/table/tbody/tr/td[6]/a')
  38.         next_link.click()
  39.         time.sleep(30)
  40.     except NoSuchElementException:
  41.         pages_remaining = False

请注意，我对代码进行了一些修改，包括指定了HTML解析器，修复了CSV写入的newline问题，并添加了时间模块以等待页面加载。希望这可以帮助您解决问题。

英文:

I'm just started using selenium to scrape the table from webpage. So, I implemented the navigation of webpage using selenium. But, the the result keep looping when I run the code. Pretty sure that I wrote the code wrong. What should I fix the code so the navigation selenium works?

  1. import requests
  2.     import csv
  3.     from bs4 import BeautifulSoup as bs
  4.     from selenium import webdriver
  6. browser=webdriver.Chrome()
  7. browser.get('https://dir.businessworld.com.my/15/posts/16-Computers-The-Internet')
  9. # url = requests.get("https://dir.businessworld.com.my/15/posts/16-Computers-The-Internet/")
  10. soup=bs(browser.page_source)
  12. filename = "C:/Users/User/Desktop/test.csv"
  13. csv_writer = csv.writer(open(filename, 'w'))
  15. pages_remaining = True
  16.  
  17. while pages_remaining:
  18.     for tr in soup.find_all("tr"):
  19.         data = []
  20.         # for headers ( entered only once - the first time - )
  21.         for th in tr.find_all("th"):
  22.             data.append(th.text)
  23.         if data:
  24.             print("Inserting headers : {}".format(','.join(data)))
  25.             csv_writer.writerow(data)
  26.             continue
  28.         for td in tr.find_all("td"):
  29.             if td.a:
  30.                 data.append(td.a.text.strip())
  31.             else:
  32.                 data.append(td.text.strip())
  33.         if data:
  34.             print("Inserting data: {}".format(','.join(data)))
  35.             csv_writer.writerow(data)
  37. try:
  38.     #Checks if there are more pages with links
  39.     next_link = driver.find_element_by_xpath('//*[@id="content"]/div[3]/table/tbody/tr/td[2]/table/tbody/tr/td[6]/a ]')
  40.     next_link.click()
  41.     time.sleep(30)
  42. except NoSuchElementException:
  43.     rows_remaining = False

答案1

得分: 1

检查页面上是否存在“下一页”按钮，然后单击，否则退出循环。

  1. if len(browser.find_elements_by_xpath("//a[contains(.,'下一页')]")) > 0:
  2.     browser.find_element_by_xpath("//a[contains(.,'下一页')]").click()
  3. else:
  4.     break

不需要使用 time.sleep()，而是使用 WebDriverWait。

  1. WebDriverWait(browser, 10).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "table.postlisting")))
英文:

Check if there any next button present on the page then click else exit from while loop.

  1. if len(browser.find_elements_by_xpath("//a[contains(.,'Next')]"))>0:
  2.       browser.find_element_by_xpath("//a[contains(.,'Next')]").click()
  3. else:
  4.       break

No need to use time.sleep() instead use WebDriverWait()

Code:

  1. import csv
  2. from bs4 import BeautifulSoup as bs
  3. from selenium import webdriver
  4. from selenium.webdriver.support.ui import WebDriverWait
  5. from selenium.webdriver.support import expected_conditions as EC
  6. from selenium.webdriver.common.by import By
  8. browser=webdriver.Chrome()
  9. browser.get('https://dir.businessworld.com.my/15/posts/16-Computers-The-Internet')
  10. WebDriverWait(browser, 10).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "table.postlisting")))
  11. soup=bs(browser.page_source)
  13. filename = "C:/Users/User/Desktop/test.csv"
  14. csv_writer = csv.writer(open(filename, 'w'))
  16. pages_remaining = True
  18. while pages_remaining:
  19.     WebDriverWait(browser,10).until(EC.visibility_of_element_located((By.CSS_SELECTOR,"table.postlisting")))
  20.     for tr in soup.find_all("tr"):
  21.         data = []
  22.         # for headers ( entered only once - the first time - )
  23.         for th in tr.find_all("th"):
  24.             data.append(th.text)
  25.         if data:
  26.             print("Inserting headers : {}".format(','.join(data)))
  27.             csv_writer.writerow(data)
  28.             continue
  30.         for td in tr.find_all("td"):
  31.             if td.a:
  32.                 data.append(td.a.text.strip())
  33.             else:
  34.                 data.append(td.text.strip())
  35.         if data:
  36.             print("Inserting data: {}".format(','.join(data)))
  37.             csv_writer.writerow(data)
  40.     if len(browser.find_elements_by_xpath("//a[contains(.,'Next')]"))>0:
  41.         browser.find_element_by_xpath("//a[contains(.,'Next')]").click()
  42.     else:
  43.         break

huangapple
  • 本文由 发表于 2020年1月6日 17:59:34
  • 转载请务必保留本文链接：https://go.coder-hub.com/59610011.html
  • pagination
  • selenium
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定