英文:
R: Reading a File in Chunks from a Website
问题
I am working with the R programming language.
Here is some R code that reads a CSV file from an internet website, saves it to the temporary directory, and then imports it into R:
# Set the URL of the CSV file
url <- "https://people.sc.fsu.edu/~jburkardt/data/csv/hw_200.csv"
# Set the path for the temporary file
temp_file_path <- tempfile(fileext = ".csv")
# Download the file from the URL and save it to the temporary file
download.file(url, destfile = temp_file_path, mode = "wb")
a <- read.csv(temp_file_path)
My Problem: For larger files, sometimes after downloading the file into the temporary directory, when I try to import the file into R - I get an error stating that "Can not allocate vector of size ..."
My Question: I was wondering - in such situations, is it possible to import the file into "chunks" (e.g. 10 MB chunks) in an effort to avoid this error? Can someone please show me how to do this if it is possible?
Thanks!
英文:
I am working with the R programming language.
Here is some R code that reads a CSV file from an internet website, saves it to the temporary directory, and then imports it into R:
# Set the URL of the CSV file
url <- "https://people.sc.fsu.edu/~jburkardt/data/csv/hw_200.csv"
# Set the path for the temporary file
temp_file_path <- tempfile(fileext = ".csv")
# Download the file from the URL and save it to the temporary file
download.file(url, destfile = temp_file_path, mode = "wb")
a <- read.csv(temp_file_path)
My Problem: For larger files, sometimes after downloading the file into the temporary directory, when I try to import the file into R - I get an error stating that "Can not allocate vector of size ..."
.
My Question: I was wondering - in such situations, is it possible to import the file into "chunks" (e.g. 10 MB chunks) in an effort to avoid this error? Can someone please show me how to do this if it is possible?
Thanks!
答案1
得分: 1
你可以考虑类似这样的方式:
library(data.table)
url <- "https://people.sc.fsu.edu/~jburkardt/data/csv/hw_200.csv"
temp_file_path <- tempfile(fileext = ".csv")
download.file(url, destfile = temp_file_path, mode = "wb")
df <- fread(file = temp_file_path, skip = 50, nrows = 50)
使用上述代码,你可以读取第50到99行的数据。
英文:
You can consider something like this :
library(data.table)
url <- "https://people.sc.fsu.edu/~jburkardt/data/csv/hw_200.csv"
temp_file_path <- tempfile(fileext = ".csv")
download.file(url, destfile = temp_file_path, mode = "wb")
df <- fread(file = temp_file_path, skip = 50, nrows = 50)
With the code above, you read the rows 50 to 99
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论