rvest：调整网络抓取功能，以便在遇到无数据时继续。

huangapple
go评论105阅读模式
英文:

rvest: Adapt webscraping function to continue if it comes across no data

问题

以下是翻译好的部分：

这个函数可以很好地用于从此数据集存储库中提取不同数据集的Begin:和End:日期。然而，并非每个数据集的元数据都包含Begin:和End:，这会导致函数停止并抛出错误消息。是否可以修改函数以跳过下一个URL（package_id）而不是停止？

  1. library(tidyverse)
  2. library(rvest)
  3. library(janitor)
  5. # 用于从不同元数据URL提取开始和结束日期的函数
  7. scraper <- function(package_id) {
  8.   cat("正在提取数据", package_id, "\n")
  9.   data <- str_c("https://portal.edirepository.org/nis/metadataviewer?packageid=",
  10.                 package_id) %>%
  11.     read_html() %>%
  12.     html_elements(".subgroup.onehundred_percent") %>%
  13.     pluck(1) %>%
  14.     html_elements(".roweven") %>%
  15.     html_text2() 
  16.   
  17.   tibble(begin = pluck(data, 1), 
  18.          end = pluck(data, 1))
  19. }
  21. # 获取package_id
  23. good_page <-
  24.   "http://portal.edirepository.org:80/nis/simpleSearch?defType=edismax&amp;q=*:*&amp;fq=-scope:ecotrends&amp;fq=-scope:lter-landsat*&amp;fq=scope:(knb-lter-kbs)&amp;fl=id,packageid,title,author,organization,pubdate,coordinates&amp;debug=false&amp;start=0&amp;rows=3" %>%
  25.   read_html()
  27. bad_page <-
  28.   "http://portal.edirepository.org:80/nis/simpleSearch?defType=edismax&amp;q=*:*&amp;fq=-scope:ecotrends&amp;fq=-scope:lter-landsat*&amp;fq=scope:(knb-lter-kbs)&amp;fl=id,packageid,title,author,organization,pubdate,coordinates&amp;debug=false&amp;start=40&amp;rows=4" %>%
  29.   read_html()
  31. # 正常工作，返回发布日期、package_id、开始日期和结束日期
  33. dat1 <- good_page %>%
  34.   html_table() %>%
  35.   pluck(4) %>%
  36.   clean_names() %>%
  37.   mutate(across(title, ~ str_squish(str_remove_all(., "\n")))) %>%
  38.   mutate(date = map(package_id, scraper)) %>%
  39.   unnest(date) %>%
  40.   select(!c(title,creators))
  41. dat1
  43. # 出错，因为 knb-lter-kbs.141.1 不包含开始或结束日期
  44. # 上面的函数是否可以修改以跳过下一个package_id而不是停止？
  46. dat2 <- bad_page %>%
  47.   html_table() %>%
  48.   pluck(4) %>%
  49.   clean_names() %>%
  50.   mutate(across(title, ~ str_squish(str_remove_all(., "\n")))) %>%
  51.   mutate(date = map(package_id, scraper)) %>%
  52.   unnest(date) %>%
  53.   select(!c(title,creators))
英文:

The following function works well for scraping the Begin: and End: dates for different datasets from this dataset repository. However, not every dataset's metadata includes Begin: and End: which causes the function to stop and throw an error message. Can the function be adapted to skip to the next URL (package_id) rather than stop?

  1. library(tidyverse)
  2. library(rvest)
  3. library(janitor)
  5. # Function for scraping begin and end dates from different metadata URLs
  7. scraper &lt;- function(package_id) {
  8.   cat(&quot;Scraping&quot;, package_id, &quot;\n&quot;)
  9.   data &lt;- str_c(&quot;https://portal.edirepository.org/nis/metadataviewer?packageid=&quot;,
  10.                 package_id) %&gt;%
  11.     read_html() %&gt;%
  12.     html_elements(&quot;.subgroup.onehundred_percent&quot;) %&gt;%
  13.     pluck(1) %&gt;%
  14.     html_elements(&quot;.roweven&quot;) %&gt;%
  15.     html_text2() 
  16.   
  17.   tibble(begin = pluck(data, 1), 
  18.          end = pluck(data, 1))
  19. }
  21. # Grabs package_id
  23. good_page &lt;-
  24.   &quot;http://portal.edirepository.org:80/nis/simpleSearch?defType=edismax&amp;q=*:*&amp;fq=-scope:ecotrends&amp;fq=-scope:lter-landsat*&amp;fq=scope:(knb-lter-kbs)&amp;fl=id,packageid,title,author,organization,pubdate,coordinates&amp;debug=false&amp;start=0&amp;rows=3&quot; %&gt;%
  25.   read_html()
  27. bad_page &lt;-
  28.   &quot;http://portal.edirepository.org:80/nis/simpleSearch?defType=edismax&amp;q=*:*&amp;fq=-scope:ecotrends&amp;fq=-scope:lter-landsat*&amp;fq=scope:(knb-lter-kbs)&amp;fl=id,packageid,title,author,organization,pubdate,coordinates&amp;debug=false&amp;start=40&amp;rows=4&quot; %&gt;%
  29.   read_html()
  31. # Works as it should, returns publication_date, package_id, begin, and end
  33. dat1 &lt;- good_page %&gt;%
  34.   html_table() %&gt;%
  35.   pluck(4) %&gt;%
  36.   clean_names() %&gt;%
  37.   mutate(across(title, ~ str_squish(str_remove_all(., &quot;\\n&quot;)))) %&gt;%
  38.   mutate(date = map(package_id, scraper)) %&gt;% 
  39.   unnest(date) %&gt;%
  40.   select(!c(title,creators))
  41. dat1
  43. # Breaks because knb-lter-kbs.141.1 does not include begin or end dates
  44. # Can the function above be amended to skip to the next package_id rather than stop?
  46. dat2 &lt;- bad_page %&gt;%
  47.   html_table() %&gt;%
  48.   pluck(4) %&gt;%
  49.   clean_names() %&gt;%
  50.   mutate(across(title, ~ str_squish(str_remove_all(., &quot;\\n&quot;)))) %&gt;%
  51.   mutate(date = map(package_id, scraper)) %&gt;% 
  52.   unnest(date) %&gt;%
  53.   select(!c(title,creators))

答案1

得分: 2

使用purrr::possibly()函数操作符，您可以定义一个版本的函数，当出现错误时返回NA值：

  1. library(tidyverse)
  2. library(rvest)
  3. library(janitor)
  5. safe_scraper <- possibly(
  6.   scraper, 
  7.   otherwise = tibble(begin = NA_character_, end = NA_character_)
  8. )
  10. bad_page %>%
  11.   html_table() %>%
  12.   pluck(4) %>%
  13.   clean_names() %>%
  14.   mutate(
  15.     title = str_squish(str_remove_all(title, "\\n")),
  16.     date = map(package_id, safe_scraper)
  17.   ) %>% 
  18.   unnest(date) %>%
  19.   select(!c(title, creators))
  1. # A tibble: 4 × 4
  2.   publication_date package_id         begin      end           
  3.              <int> <chr>              <chr>      <chr>         
  4. 1             2018 knb-lter-kbs.139.5 2008-01-01 2008-01-01    
  5. 2             2018 knb-lter-kbs.140.5 NA         NA            
  6. 3             2016 knb-lter-kbs.141.1 Information Manager (LTER Network Office) [ email ] Information Manager (LTER Network Office) [ …
  7. 4             2018 knb-lter-kbs.162.79 1950-01-01 1950-01-01

从第3行的判断，似乎您的工作流程可能存在其他问题。

英文:

Using the purrr::possibly() function operator, you can define a version of your function that returns NA values when an error occurs:

  1. library(tidyverse)
  2. library(rvest)
  3. library(janitor)
  5. safe_scraper &lt;- possibly(
  6.   scraper, 
  7.   otherwise = tibble(begin = NA_character_, end = NA_character_)
  8. )
  10. bad_page %&gt;%
  11.   html_table() %&gt;%
  12.   pluck(4) %&gt;%
  13.   clean_names() %&gt;%
  14.   mutate(
  15.     title = str_squish(str_remove_all(title, &quot;\\n&quot;)),
  16.     date = map(package_id, safe_scraper)
  17.   ) %&gt;% 
  18.   unnest(date) %&gt;%
  19.   select(!c(title, creators))
  1. # A tibble: 4 &#215; 4
  2.   publication_date package_id          begin                                               end                                          
  3.              &lt;int&gt; &lt;chr&gt;               &lt;chr&gt;                                               &lt;chr&gt;                                        
  4. 1             2018 knb-lter-kbs.139.5  2008-01-01                                          2008-01-01                                   
  5. 2             2018 knb-lter-kbs.140.5  NA                                                  NA                                           
  6. 3             2016 knb-lter-kbs.141.1  Information Manager (LTER Network Office) [ email ] Information Manager (LTER Network Office) [ …
  7. 4             2018 knb-lter-kbs.162.79 1950-01-01                                          1950-01-01

Judging by row 3, though, looks like there may be another issue in your workflow.

答案2

得分: 1

从HTML方法更改为XML方法以获得更一致的数据提取。

  1. library(httr2)
  2. library(tidyverse)
  3. library(janitor)
  5. bad_page <-
  6.   "http://portal.edirepository.org:80/nis/simpleSearch?defType=edismax&amp;q=*:*&amp;fq=-scope:ecotrends&amp;fq=-scope:lter-landsat*&amp;fq=scope:(knb-lter-kbs)&amp;fl=id,packageid,title,author,organization,pubdate,coordinates&amp;debug=false&amp;start=40&amp;rows=4" %>%
  7.   read_html()
  9. scraper <- function(package_id) {
  10.   cat("Scraping", package_id, "\n")
  11.   data <- str_c(
  12.     "https://portal.edirepository.org/nis/metadataviewer?packageid=",
  13.     package_id, 
  14.     "&amp;contentType=application/xml"
  15.   ) %>%
  16.     request() %>%
  17.     req_perform() %>%
  18.     resp_body_xml() %>%
  19.     xml_find_all(".//calendarDate") %>%
  20.     xml_text()
  21.   
  22.   tibble(begin = pluck(data, 1),
  23.          end =   pluck(data, 1))
  24. }
  26. data <- bad_page %>%
  27.   html_table() %>%
  28.   pluck(4) %>%
  29.   clean_names() %>%
  30.   mutate(across(title, ~ str_squish(str_remove_all(., "\\n")))) %>%
  31.   mutate(date = map(package_id, possibly(
  32.     scraper, otherwise = tibble(begin = NA, end = NA)
  33.   ))) %>%
  34.   unnest(date, keep_empty = TRUE) %>%
  35.   select(!c(title, creators))

结果：

  1. # A tibble: 4 × 4
  2.   publication_date package_id          begin      end       
  3.              <int> <chr>               <chr>      <chr>     
  4. 1             2018 knb-lter-kbs.139.5  2008-01-01 2008-01-01
  5. 2             2018 knb-lter-kbs.140.5  NA         NA        
  6. 3             2016 knb-lter-kbs.141.1  NA         NA        
  7. 4             2018 knb-lter-kbs.162.79 1950-01-01 1950-01-01
英文:

Changed from html approach to XML for more consistent data extraction

  1. library(httr2)
  2. library(tidyverse)
  3. library(janitor)
  5. bad_page &lt;-
  6.   &quot;http://portal.edirepository.org:80/nis/simpleSearch?defType=edismax&amp;q=*:*&amp;fq=-scope:ecotrends&amp;fq=-scope:lter-landsat*&amp;fq=scope:(knb-lter-kbs)&amp;fl=id,packageid,title,author,organization,pubdate,coordinates&amp;debug=false&amp;start=40&amp;rows=4&quot; %&gt;%
  7.   read_html()
  9. scraper &lt;- function(package_id) {
  10.   cat(&quot;Scraping&quot;, package_id, &quot;\n&quot;)
  11.   data &lt;- str_c(
  12.     &quot;https://portal.edirepository.org/nis/metadataviewer?packageid=&quot;,
  13.     package_id, 
  14.     &quot;&amp;contentType=application/xml&quot;
  15.   ) %&gt;%
  16.     request() %&gt;%
  17.     req_perform() %&gt;%
  18.     resp_body_xml() %&gt;%
  19.     xml_find_all(&quot;.//calendarDate&quot;) %&gt;%
  20.     xml_text()
  21.   
  22.   tibble(begin = pluck(data, 1),
  23.          end =   pluck(data, 1))
  24. }
  26. data &lt;- bad_page %&gt;%
  27.   html_table() %&gt;%
  28.   pluck(4) %&gt;%
  29.   clean_names() %&gt;%
  30.   mutate(across(title, ~ str_squish(str_remove_all(., &quot;\\n&quot;)))) %&gt;%
  31.   mutate(date = map(package_id, possibly(
  32.     scraper, otherwise = tibble(begin = NA, end = NA)
  33.   ))) %&gt;%
  34.   unnest(date, keep_empty = TRUE) %&gt;%
  35.   select(!c(title, creators))

The result:

  1. # A tibble: 4 &#215; 4
  2.   publication_date package_id          begin      end       
  3.              &lt;int&gt; &lt;chr&gt;               &lt;chr&gt;      &lt;chr&gt;     
  4. 1             2018 knb-lter-kbs.139.5  2008-01-01 2008-01-01
  5. 2             2018 knb-lter-kbs.140.5  NA         NA        
  6. 3             2016 knb-lter-kbs.141.1  NA         NA        
  7. 4             2018 knb-lter-kbs.162.79 1950-01-01 1950-01-01

huangapple
  • 本文由 发表于 2023年6月16日 01:45:26
  • 转载请务必保留本文链接：https://go.coder-hub.com/76484264.html
  • function
  • r
  • rvest
  • tidyr
  • web-scraping
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定