英文:
Scraping with {rvest} yields "character (empty)"?
问题
我已经在爬取一个文件,但现在有一个新的网址 - 我尝试更改了网址和 CSS 选择器 - 但我的 link
对象不是一个搜索路径,而是 "character (empty)" - 问题似乎是什么?
网站:https://arbetsformedlingen.se/statistik/statistik-om-varsel
我想获取位于 Antal varsel och berörda personer 'box' 中的文件 "Tillfällig statistik per län och bransch, januari-april 2023"。
R 代码:
library(tidyverse)
library(stringr)
library(rio) #import-function
library(rvest) #read_html()-function
# 链接到目标网站
url <- "https://arbetsformedlingen.se/statistik/statistik-om-varsel"
## 解析 HTML 内容
doc <- read_html(url)
## 查找要抓取的数据
### 选择 CSS 定位器
link <- html_elements(doc, css = '#cardContainer > app-downloads:nth-child(3) > div > div:nth-child(3) > div > digi-link-internal > digi-link > a') %>%
html_attr("href")
# 创建文件下载的 URL
url2 <- "https://arbetsformedlingen.se"
full_link <- sprintf("%s%s", url2, link)
# 获取并本地保存文件
td = tempdir() # 创建临时文件夹
varsel_fil <- tempfile(tmpdir=td, fileext = ".xlsx")
download.file(full_link, destfile = varsel_fil, mode = "wb")
# 读取文件到一个数据框中
df_imported <- import(varsel_fil, which=1) #which - 选择 'sheet'-nr
之前 html_elements
函数中的 css 参数是 #svid12_142311c317a09e842af1a94 > div.sv-text-portlet-content > p:nth-child(20) > strong > a
-> 所以开头有些不同 - 不过我不明白它意味着什么...
感谢任何帮助!
英文:
I've been scraping a file, but now there's a new URL - I just tried to chg. the URL and CSS-selector - but my link
-object don't result in a searchpath but "character (empty)" - what's seems to be the problem?
Site: https://arbetsformedlingen.se/statistik/statistik-om-varsel
I want to grab the file "Tillfällig statistik per län och bransch, januari-april 2023" in the 'box' Antal varsel och berörda personer.
R-code:
library(tidyverse)
library(stringr)
library(rio) #import-function
library(rvest) #read_html()-function
# Link to target site
url <- "https://arbetsformedlingen.se/statistik/statistik-om-varsel"
## Parsa HTML-innehållet
doc <- read_html(url)
## Hitta data som du vill skrapa
### Select CSS locator
link <- html_elements(doc, css = '#cardContainer > app-downloads:nth-child(3) > div > div:nth-child(3) > div > digi-link-internal > digi-link > a') %>%
html_attr("href")
# Create URL for file download
url2 <- "https://arbetsformedlingen.se"
full_link <- sprintf("%s%s", url2, link)
# Get and save file locally
td = tempdir() # skapa temporär mapp
varsel_fil <- tempfile(tmpdir=td, fileext = ".xlsx")
download.file(full_link, destfile = varsel_fil, mode = "wb")
# Read file into a df
df_imported <- import(varsel_fil, which=1) #which - välj 'sheet'-nr
Previously the css-argument in the html_elements
-function was #svid12_142311c317a09e842af1a94 > div.sv-text-portlet-content > p:nth-child(20) > strong > a
-> So the beginning is quite different - I don't understand what it implies though..
Thanks for any assistance!
答案1
得分: 2
以下是您要翻译的内容:
"That page is now mostly rendered by JavaScript and most of that content is not included in the page source, you can check by disabling JS for the site in your browser. List of files in the described box is pulled from https://arbetsformedlingen.se/rest/analysportalen/va/sitevision
.
A quick way to find this API endpoint would be through the network tab of browser's developer tools -- after launching dev tools, refresh the page to capture all requests and search for some phrase that can't be found from the source of the main page, i.e. 'januari-april', looks something like this. Once the API endpoint with file list is identified, we can extract the file URL and proceed with the download:
library(dplyr)
url_ <- "https://arbetsformedlingen.se/rest/analysportalen/va/sitevision"
xlsx_response <- jsonlite::fromJSON(url_, simplifyVector = FALSE) %>%
# there are 3 files listed, naively picking the last one;
# may or may not work in a long run
dplyr::last() %>%
# we could also use purrr::keep() **before** last() to keep only
# the record with matching name, like
# purrr::keep(~ .x$name == "Länktext till varsel tillfällig statistik")
purrr::pluck("properties", "link") %>%
# combine first part of url_ and extracted link to get full URL for the file
# last parameter, ".", is where the output of previous pipe ends up
# next expression evaluates as:
# str_replace("https://arbetsformedlingen.se/rest/analysportalen/va/sitevision", "(?<=\\w)/.*", "/download/18.793fa1821869801540c14b3/1683719045927/web-varsel-bransch-lan-2023.xlsx")
stringr::str_replace(url_, "(?<=\\w)/.*", .) %>%
httr2::request() %>%
# we can store the file by setting the path
httr2::req_perform(path = file.path(tempdir(), basename(.$url)))
# httr2 response:
xlsx_response
#> <httr2_response>
#> GET
#> https://arbetsformedlingen.se/download/18.793fa1821869801540c14b3/1683719045927/web-varsel-bransch-lan-2023.xlsx
#> Status: 200 OK
#> Content-Type: application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
#> Body: On disk 'body'
# open file location:
# browseURL(xlsx_response$body[1])
Downloaded file:
readxl::read_xlsx(xlsx_response$body[1], 1, "A4:Y26") %>% glimpse()
#> New names:
#> • `` -> `...1`
#> • `` -> `...2`
#> • `` -> `...25`
#> Rows: 22
#> Columns: 25
#> $ ...1 <chr> "SNI-kod", "A", "B", "C", "D", "E", "F", "G", "H", "I", "J", "K"…
#> $ ...2 <chr> "Näringsgren", "Jordbruk, skogsbruk och fiske", "Utvinning av mi…
#> $ AB <chr> "Stockholms län", "5", NA, "56", NA, NA, "256", "384", "141", "3…
#> $ C <chr> "Uppsala län", NA, NA, NA, NA, NA, "18", NA, NA, NA, NA, NA, NA,…
#> $ D <chr> "Södermanlands län", NA, NA, NA, NA, "8", NA, NA, NA, NA, NA, NA…
#> $ E <chr> "Östergötlands län", NA, NA, "20", NA, NA, "61", NA, "17", NA, N…
#> $ F <chr> "Jönköpings län", NA, "5", "246", NA, NA, NA, "21", NA, NA, NA, …
#> $ G <chr> "Kronobergs län", NA, NA, "201", NA, NA, "19", NA, NA, NA, NA, N…
#> $ H <chr> "Kalmar län", NA, NA, NA, NA, NA, NA, NA, "15", NA, NA, NA, NA, …
#> $ I <chr> "Gotlands län", NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, …
#> $ K <chr> "Blekinge län", NA, NA, "9", NA, NA, NA, "18", NA, NA, NA, NA, N…
#> $ M <chr> "Skåne län", NA, NA, "30", NA, NA, "57", "63", NA, NA, "46", NA,…
#> $ N <chr> "Hallands län", NA, NA, "24", NA, NA, "15", "6", NA, NA, NA, NA,…
#> $ O <chr> "Västra Götalands län", NA, NA, "62", NA, NA, "120", "27", "96",…
#> $ S <chr> "Värmlands län", NA, NA, "20", NA, NA, NA, NA, NA, "10", NA, NA,…
#> $ T <chr> "Örebro län", NA, NA, "47", NA, NA, NA, "8", NA, "18", NA, NA, "…
#> $ U <chr> "Västmanlands län", NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, …
#> $ W <chr> "Dalarnas län", NA, NA, NA, NA, NA, "15", NA, NA, NA, NA, NA, NA…
#> $ X <chr> "Gävleborgs län", NA, NA, "32", NA, NA, NA, NA, NA, "50", NA, NA…
#> $ Y <chr> "Västernorrlands län", NA, NA, "10", NA, NA, NA, NA, NA, NA, NA,…
#> $ Z <chr> "Jämtlands län", NA, NA, "14", NA, NA, NA, NA, NA
<details>
<summary>英文:</summary>
That page is now mostly rendered by JavaScript and most of that content is not included in the page source, you can check by disabling JS for the site in your browser. List of files in the described box is pulled from `https://arbetsformedlingen.se/rest/analysportalen/va/sitevision`.
A quick way to find this API endpoint would be through the network tab of browser's developer tools -- after launching dev tools, refresh the page to capture all requests and search for some phrase that can't be found from the source of the main page, i.e. "januari-april", looks something [like this][1]. Once the API endpoint with file list is identified, we can extract the file URL and proceed with the download:
``` r
library(dplyr)
url_ <- "https://arbetsformedlingen.se/rest/analysportalen/va/sitevision"
xlsx_response <- jsonlite::fromJSON(url_, simplifyVector = FALSE) %>%
# there are 3 files listed, naively picking the last one;
# may or may not work in a long run
dplyr::last() %>%
# we could also use purrr::keep() **before** last() to keep only
# the record with matching name, like
# purrr::keep(~ .x$name == "Länktext till varsel tillfällig statistik")
purrr::pluck("properties", "link") %>%
# combine first part of url_ and extracted link to get full URL for the file
# last parameter, ".", is where the output of previous pipe ends up
# next expression evaluates as:
# str_replace("https://arbetsformedlingen.se/rest/analysportalen/va/sitevision", "(?<=\\w)/.*", "/download/18.793fa1821869801540c14b3/1683719045927/web-varsel-bransch-lan-2023.xlsx")
stringr::str_replace(url_, "(?<=\\w)/.*", .) %>%
httr2::request() %>%
# we can store the file by setting the path
httr2::req_perform(path = file.path(tempdir(), basename(.$url)))
# httr2 response:
xlsx_response
#> <httr2_response>
#> GET
#> https://arbetsformedlingen.se/download/18.793fa1821869801540c14b3/1683719045927/web-varsel-bransch-lan-2023.xlsx
#> Status: 200 OK
#> Content-Type: application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
#> Body: On disk 'body'
# open file location:
# browseURL(xlsx_response$body[1])
Downloaded file:
readxl::read_xlsx(xlsx_response$body[1], 1, "A4:Y26") %>% glimpse()
#> New names:
#> • `` -> `...1`
#> • `` -> `...2`
#> • `` -> `...25`
#> Rows: 22
#> Columns: 25
#> $ ...1 <chr> "SNI-kod", "A", "B", "C", "D", "E", "F", "G", "H", "I", "J", "K"…
#> $ ...2 <chr> "Näringsgren", "Jordbruk, skogsbruk och fiske", "Utvinning av mi…
#> $ AB <chr> "Stockholms län", "5", NA, "56", NA, NA, "256", "384", "141", "3…
#> $ C <chr> "Uppsala län", NA, NA, NA, NA, NA, "18", NA, NA, NA, NA, NA, NA,…
#> $ D <chr> "Södermanlands län", NA, NA, NA, NA, "8", NA, NA, NA, NA, NA, NA…
#> $ E <chr> "Östergötlands län", NA, NA, "20", NA, NA, "61", NA, "17", NA, N…
#> $ F <chr> "Jönköpings län", NA, "5", "246", NA, NA, NA, "21", NA, NA, NA, …
#> $ G <chr> "Kronobergs län", NA, NA, "201", NA, NA, "19", NA, NA, NA, NA, N…
#> $ H <chr> "Kalmar län", NA, NA, NA, NA, NA, NA, NA, "15", NA, NA, NA, NA, …
#> $ I <chr> "Gotlands län", NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, …
#> $ K <chr> "Blekinge län", NA, NA, "9", NA, NA, NA, "18", NA, NA, NA, NA, N…
#> $ M <chr> "Skåne län", NA, NA, "30", NA, NA, "57", "63", NA, NA, "46", NA,…
#> $ N <chr> "Hallands län", NA, NA, "24", NA, NA, "15", "6", NA, NA, NA, NA,…
#> $ O <chr> "Västra Götalands län", NA, NA, "62", NA, NA, "120", "27", "96",…
#> $ S <chr> "Värmlands län", NA, NA, "20", NA, NA, NA, NA, NA, "10", NA, NA,…
#> $ T <chr> "Örebro län", NA, NA, "47", NA, NA, NA, "8", NA, "18", NA, NA, "…
#> $ U <chr> "Västmanlands län", NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, …
#> $ W <chr> "Dalarnas län", NA, NA, NA, NA, NA, "15", NA, NA, NA, NA, NA, NA…
#> $ X <chr> "Gävleborgs län", NA, NA, "32", NA, NA, NA, NA, NA, "50", NA, NA…
#> $ Y <chr> "Västernorrlands län", NA, NA, "10", NA, NA, NA, NA, NA, NA, NA,…
#> $ Z <chr> "Jämtlands län", NA, NA, "14", NA, NA, NA, NA, NA, NA, "6", NA, …
#> $ AC <chr> "Västerbottens län", NA, NA, "50", NA, NA, "9", NA, "16", NA, NA…
#> $ BD <chr> "Norrbottens län", NA, NA, "6", NA, NA, "15", "6", NA, NA, NA, N…
#> $ `-` <chr> "Uppgift saknas", NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA…
#> $ ...25 <chr> "Riket", "5", "5", "827", NA, "8", "585", "533", "285", "125", "…
<sup>Created on 2023-05-20 with reprex v2.0.2</sup>
More base-like approach would perhaps be:
download.file(paste0("https://arbetsformedlingen.se",
jsonlite::read_json("https://arbetsformedlingen.se/rest/analysportalen/va/sitevision")[[3]]$properties$link),
file.path(tempdir(),"out.xlsx"), mode = "wb")
答案2
得分: 0
感谢 @margusl,以下是翻译好的代码部分:
# 创建临时目录,将文件下载到本地
td = tempdir() ## 创建临时目录
varsel_fil <- tempfile(tmpdir=td, fileext = ".xlsx")
## 下载文件
download.file(paste0("https://arbetsformedlingen.se",
jsonlite::read_json("https://arbetsformedlingen.se/rest/analysportalen/va/sitevision")[[3]]$properties$link),
destfile = varsel_fil, mode = "wb")
# 将文件导入为数据框
df <- import(varsel_fil, which=1) # which - sheet nbr
这段代码选择了 JSON 响应中的第三个列表,首先创建一个对象并检查 JSON 响应是一个很好的主意。第一个建议(带有更多代码)稍微更加严格,根据条件选择列表。
英文:
So thanks to @margusl this code would suffice to do what my original code did:
# Create temp dir, download file locally
td = tempdir() ## created temp map
varsel_fil <- tempfile(tmpdir=td, fileext = ".xlsx")
## Download file
download.file(paste0("https://arbetsformedlingen.se",
jsonlite::read_json("https://arbetsformedlingen.se/rest/analysportalen/va/sitevision")[[3]]$properties$link),
destfile = varsel_fil, mode = "wb")
#Importera file as df
df <- import(varsel_fil, which=1) #which - sheet nbr
This code select third list from json-response, it's a good idea to first create an object and inspect the json-response. The first suggestion (with more code) is a bit more rigid and select list based on criteria.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论