用Selenium网页爬取页面上的所有项目。

huangapple
go评论100阅读模式
英文:

web scrape all items in a page with selenium

问题

我正在尝试获取网页中的所有项目，例如，我想获取第一个

"Hillebrand Boudewynsz. van der Aa (1661 - 1717)"

然后获取页面中的其他49个项目。

我的代码如下，我尝试使用Selenium通过xpath或CSS获取项目，但我不确定正确的路径是什么。

欢迎使用两种选项。

这是代码中所需的句子：

  1. #Finding element
  2. object<-remDr$findElement(using="xpath","/html/body/div[2]/div/ul/li[1]/a")
  3. #---------------------------------------------------------------------

和网站：

https://www.vondel.humanities.uva.nl/ecartico/persons/index.php?subtask=browse

  1. rm(list=ls())
  2. library(tidyverse)
  3. library(robotstxt)
  4. library(rvest)
  5. library(RSelenium)
  6. library(tidyverse)
  7. library(netstat)
  8. library(wdman)
  9. selenium()
  11. # see path
  12. selenium_object<-selenium(retcommand = T,check = F)
  14. #binman::list_versions("chromedriver")
  16. #start the server
  17. remote_driver<-rsDriver(
  18.   
  19.   browser = "chrome",
  20.   
  21.   chromever = "113.0.5672.63",
  22.   verbose = F,
  23.   port = free_port()
  24. )
  26. # create a client object
  27. remDr<-remote_driver$client
  29. #open a browser
  30. remDr$open()
  32. # maximaize window size
  33. remDr$maxWindowSize()
  35. #navigate website
  36. remDr$navigate("https://www.vondel.humanities.uva.nl/ecartico/persons/index.php?subtask=browse")
  38. #Finding element
  39. object<-remDr$findElement(using="xpath","/html/body/div[2]/div/ul/li[1]/a")
  40. #---------------------------------------------------------------------

希望这对你有所帮助。

英文:

I am trying to bring all items in the webpage-for instance I would like to bring the first

"Hillebrand Boudewynsz. van der Aa (1661 - 1717)"
and then all the other 49 in the page
MY code is below, I am trying to use selenium and bring the items through xpath or CSS
but I am not sure for the right path
both options will be welcome
this is the required sentence from the code
#Finding element
object<-remDr$findElement(using="xpath","/html/body/div[2]/div/ul/li[1]/a")
#---------------------------------------------------------------------
and the website
https://www.vondel.humanities.uva.nl/ecartico/persons/index.php?subtask=browse

  1. rm(list=ls())
  2. library(tidyverse)
  3. #install.packages(&quot;robotstxt&quot;)
  4. library( robotstxt)
  5. #install.packages(&quot;RSelenium&quot;)
  6. library(rvest)
  7. library(RSelenium)
  8. library(tidyverse)
  9. #install.packages(&quot;netstat&quot;)
  10. library(netstat)
  11. library(wdman)
  12. selenium()
  14. # see path
  15. selenium_object&lt;-selenium(retcommand = T,check = F)
  17. #binman::list_versions(&quot;chromedriver&quot;)
  19. #start the server
  20. remote_driver&lt;-rsDriver(
  21.   
  22.   browser = &quot;chrome&quot;,
  23.   
  24.   chromever = &quot;113.0.5672.63&quot;,
  25.   verbose = F,
  26.   port = free_port()
  27. )
  29. # create a client object
  30. remDr&lt;-remote_driver$client
  32. #open a browser
  33. remDr$open()
  35. # maximaize window size
  36. remDr$maxWindowSize()
  38. #navigate website
  39. remDr$navigate(&quot;https://www.vondel.humanities.uva.nl/ecartico/persons/index.php?subtask=browse&quot;)
  41. #Finding element
  42. object&lt;-remDr$findElement(using=&quot;xpath&quot;,&quot;/html/body/div[2]/div/ul/li[1]/a&quot;)
  43. #---------------------------------------------------------------------

答案1

得分: 0

这应该适用于您！

  1. import csv
  2. from selenium import webdriver
  3. from selenium.webdriver.chrome.options import Options
  4. from selenium.webdriver.common.by import By
  5. from selenium.webdriver.support.ui import WebDriverWait
  6. from selenium.webdriver.support import expected_conditions as EC
  7. from bs4 import BeautifulSoup
  9. url = "https://www.vondel.humanities.uva.nl/ecartico/persons/index.php?subtask=browse"
  10. all_data = []
  12. # Selenium 配置
  13. chrome_options = Options()
  14. chrome_options.add_argument("--headless")  # 以无头模式运行
  15. driver = webdriver.Chrome(options=chrome_options)
  16. wait = WebDriverWait(driver, 10)
  18. try:
  19.     # 访问页面
  20.     driver.get(url)
  22.     # 等待产品加载
  23.     wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, "#setwidth > ul > li > a")))
  25.     # 获取页面的 HTML 内容
  26.     html = driver.page_source
  27. finally:
  28.     driver.quit()  # 即使出现异常也要关闭浏览器
  30. # 使用 BeautifulSoup 提取必要的数据
  31. soup = BeautifulSoup(html, 'html.parser')
  32. products = soup.select('#setwidth > ul > li > a')
  34. for title in products:
  35.     title_text = title.get_text(strip=True) if title else ""
  36.     all_data.append([title_text])
  38. # 将数据写入 CSV 文件
  39. with open("vondel.csv", "w", newline="", encoding="utf-8") as csv_file:
  40.     writer = csv.writer(csv_file)
  41.     writer.writerow(['Title'])
  42.     writer.writerows(all_data)

希望这对您有所帮助！

英文:

That should work for you!

<!-- begin snippet: js hide: false console: true babel: false -->

<!-- language: lang-html -->

  1. import csv
  2. from selenium import webdriver
  3. from selenium.webdriver.chrome.options import Options
  4. from selenium.webdriver.common.by import By
  5. from selenium.webdriver.support.ui import WebDriverWait
  6. from selenium.webdriver.support import expected_conditions as EC
  7. from bs4 import BeautifulSoup
  9. url = &quot;https://www.vondel.humanities.uva.nl/ecartico/persons/index.php?subtask=browse&quot;
  10. all_data = []
  12. # Selenium Configuration
  13. chrome_options = Options()
  14. chrome_options.add_argument(&quot;--headless&quot;)  # Running in headless mode
  15. driver = webdriver.Chrome(options=chrome_options)
  16. wait = WebDriverWait(driver, 10)
  18. try:
  19.     # Access to page
  20.     driver.get(url)
  22.     # Wait for products to load
  23.     wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, &quot;#setwidth &gt; ul &gt; li &gt; a&quot;)))
  25.     # Get the HTML content of the page
  26.     html = driver.page_source
  27. finally:
  28.     driver.quit()  # Close the browser even if an exception occurs
  30. # Extracting the necessary data using BeautifulSoup
  31. soup = BeautifulSoup(html, &#39;html.parser&#39;)
  32. products = soup.select(&#39;#setwidth &gt; ul &gt; li &gt; a&#39;)
  34. for title in products:
  35.     title_text = title.get_text(strip=True) if title else &quot;&quot;
  36.     all_data.append([title_text])
  38. # Write data to a CSV file
  39. with open(&quot;vondel.csv&quot;, &quot;w&quot;, newline=&quot;&quot;, encoding=&quot;utf-8&quot;) as csv_file:
  40.     writer = csv.writer(csv_file)
  41.     writer.writerow([&#39;Title&#39;])
  42.     writer.writerows(all_data)

<!-- end snippet -->

答案2

得分: 0

RSelenium / Selenium应该是在网站抓取时的最后选择。这个网站可以在R中使用rvest轻松抓取。以下是如何从其26个页面中抓取所有链接，并保存在一个tibble /数据框中的方法。

  1. library(tidyverse)
  2. library(rvest)
  4. scraper <- function(page) {
  5.   cat("Scraping page", page, "\n")
  6.   str_c("https://www.vondel.humanities.uva.nl/ecartico/persons/index.php?subtask=browse&field=surname&strtchar=A&page=", page) %>%  
  7.     read_html() %>% 
  8.     html_elements("ul") %>% 
  9.     pluck(2) %>% 
  10.     html_elements("a") %>% 
  11.     map_dfr(., ~ tibble(
  12.       name = .x %>%
  13.         html_text2(), 
  14.       link = .x %>%
  15.         html_attr("href") %>%
  16.         str_replace("..", "https://www.vondel.humanities.uva.nl/ecartico")
  17.     )) %>%
  18.     mutate(page = page)
  19. }
  21. df <- map_dfr(1:26, scraper)

一个 tibble: 1,269 × 3

name link page

1 Hillebrand Boudewynsz. van der Aa (1661 - 1717) https://www.vondel.humanities.uva.nl/ecartico/persons/414 1
2 Boudewijn Pietersz van der Aa (? - ?) https://www.vondel.humanities.uva.nl/ecartico/persons/10566 1
3 Pieter Boudewijnsz. van der Aa (1659 - 1733) https://www.vondel.humanities.uva.nl/ecartico/persons/10567 1
4 Boudewyn van der Aa (1672 - ca. 1714) https://www.vondel.humanities.uva.nl/ecartico/persons/10568 1
5 Machtelt van der Aa (? - ?) https://www.vondel.humanities.uva.nl/ecartico/persons/27132 1
6 Claas van der Aa I (? - ?) https://www.vondel.humanities.uva.nl/ecartico/persons/33780 1
7 Claas van der Aa II (? - ?) https://www.vondel.humanities.uva.nl/ecartico/persons/33781 1
8 Willem van der Aa (? - ?) https://www.vondel.humanities.uva.nl/ecartico/persons/33782 1
9 Johanna van der Aa (? - ?) https://www.vondel.humanities.uva.nl/ecartico/persons/59894 1
10 Hans von Aachen (1552 - 1615) https://www.vondel.humanities.uva.nl/ecartico/persons/9203 1

ℹ 1,259 more rows

ℹ Use print(n = ...) to see more rows

  4. <details>
  5. <summary>英文:</summary>
  7. RSelenium / Selenium should be your last resort when scraping websites. This site can easily be scraped with `rvest` in R. This is how you can scrape **all** links from its 26 pages, saved in a tibble / data frame. 
  9.     library(tidyverse)
  10.     library(rvest)
  11.     
  12.     scraper &lt;- function(page) {
  13.       cat(&quot;Scraping page&quot;, page, &quot;\n&quot;)
  14.       str_c(&quot;https://www.vondel.humanities.uva.nl/ecartico/persons/index.php?subtask=browse&amp;field=surname&amp;strtchar=A&amp;page=&quot;, page) %&gt;%  
  15.         read_html() %&gt;% 
  16.         html_elements(&quot;ul&quot;) %&gt;% 
  17.         pluck(2) %&gt;% 
  18.         html_elements(&quot;a&quot;) %&gt;% 
  19.         map_dfr(., ~ tibble(
  20.           name = .x %&gt;% 
  21.             html_text2(), 
  22.           link = .x %&gt;% 
  23.             html_attr(&quot;href&quot;) %&gt;% 
  24.             str_replace(&quot;..&quot;, &quot;https://www.vondel.humanities.uva.nl/ecartico&quot;)
  25.         )) %&gt;% 
  26.         mutate(page = page)
  27.     }
  28.     
  29.     df &lt;- map_dfr(1:26, scraper)
  31.     # A tibble: 1,269 &#215; 3
  32.        name                                            link                                                         page
  33.        &lt;chr&gt;                                           &lt;chr&gt;                                                       &lt;int&gt;
  34.      1 Hillebrand Boudewynsz. van der Aa (1661 - 1717) https://www.vondel.humanities.uva.nl/ecartico/persons/414       1
  35.      2 Boudewijn Pietersz van der Aa (? - ?)           https://www.vondel.humanities.uva.nl/ecartico/persons/10566     1
  36.      3 Pieter Boudewijnsz. van der Aa (1659 - 1733)    https://www.vondel.humanities.uva.nl/ecartico/persons/10567     1
  37.      4 Boudewyn van der Aa (1672 - ca. 1714)           https://www.vondel.humanities.uva.nl/ecartico/persons/10568     1
  38.      5 Machtelt van der Aa (? - ?)                     https://www.vondel.humanities.uva.nl/ecartico/persons/27132     1
  39.      6 Claas van der Aa I (? - ?)                      https://www.vondel.humanities.uva.nl/ecartico/persons/33780     1
  40.      7 Claas van der Aa II (? - ?)                     https://www.vondel.humanities.uva.nl/ecartico/persons/33781     1
  41.      8 Willem van der Aa (? - ?)                       https://www.vondel.humanities.uva.nl/ecartico/persons/33782     1
  42.      9 Johanna van der Aa (? - ?)                      https://www.vondel.humanities.uva.nl/ecartico/persons/59894     1
  43.     10 Hans von Aachen (1552 - 1615)                   https://www.vondel.humanities.uva.nl/ecartico/persons/9203      1
  44.     # ℹ 1,259 more rows
  45.     # ℹ Use `print(n = ...)` to see more rows
  47. </details>

huangapple
  • 本文由 发表于 2023年6月5日 10:11:30
  • 转载请务必保留本文链接：https://go.coder-hub.com/76403162.html
  • r
  • selenium-webdriver
  • web-scraping
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定