Reading a File in Chunks from a Website

huangapple go评论72阅读模式
英文:

R: Reading a File in Chunks from a Website

问题

I am working with the R programming language.

Here is some R code that reads a CSV file from an internet website, saves it to the temporary directory, and then imports it into R:

# Set the URL of the CSV file
url <- "https://people.sc.fsu.edu/~jburkardt/data/csv/hw_200.csv"

# Set the path for the temporary file
temp_file_path <- tempfile(fileext = ".csv")

# Download the file from the URL and save it to the temporary file
download.file(url, destfile = temp_file_path, mode = "wb")

a <- read.csv(temp_file_path)

My Problem: For larger files, sometimes after downloading the file into the temporary directory, when I try to import the file into R - I get an error stating that "Can not allocate vector of size ..."

My Question: I was wondering - in such situations, is it possible to import the file into "chunks" (e.g. 10 MB chunks) in an effort to avoid this error? Can someone please show me how to do this if it is possible?

Thanks!

英文:

I am working with the R programming language.

Here is some R code that reads a CSV file from an internet website, saves it to the temporary directory, and then imports it into R:

# Set the URL of the CSV file
url &lt;- &quot;https://people.sc.fsu.edu/~jburkardt/data/csv/hw_200.csv&quot;

# Set the path for the temporary file
temp_file_path &lt;- tempfile(fileext = &quot;.csv&quot;)

# Download the file from the URL and save it to the temporary file
download.file(url, destfile = temp_file_path, mode = &quot;wb&quot;)

a &lt;- read.csv(temp_file_path)

My Problem: For larger files, sometimes after downloading the file into the temporary directory, when I try to import the file into R - I get an error stating that &quot;Can not allocate vector of size ...&quot; .

My Question: I was wondering - in such situations, is it possible to import the file into "chunks" (e.g. 10 MB chunks) in an effort to avoid this error? Can someone please show me how to do this if it is possible?

Thanks!

答案1

得分: 1

你可以考虑类似这样的方式:

library(data.table)
url <- "https://people.sc.fsu.edu/~jburkardt/data/csv/hw_200.csv"
temp_file_path <- tempfile(fileext = ".csv")
download.file(url, destfile = temp_file_path, mode = "wb")
df <- fread(file = temp_file_path, skip = 50, nrows = 50)

使用上述代码,你可以读取第50到99行的数据。

英文:

You can consider something like this :

library(data.table)
url &lt;- &quot;https://people.sc.fsu.edu/~jburkardt/data/csv/hw_200.csv&quot;
temp_file_path &lt;- tempfile(fileext = &quot;.csv&quot;)
download.file(url, destfile = temp_file_path, mode = &quot;wb&quot;)
df &lt;- fread(file = temp_file_path, skip = 50, nrows = 50)

With the code above, you read the rows 50 to 99

huangapple
  • 本文由 发表于 2023年5月14日 14:14:39
  • 转载请务必保留本文链接:https://go.coder-hub.com/76246089.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定