R read.csv error: Operation not permittedError in file(file, "rt")

I have an error with read.csv() in R that I absolutely can not solve.

What I did: I downloaded a .zip file from https://www.finra.org/finra-data/browse-catalog/equity-short-interest/data (I set dates from 2019-01-01 to 2019-12-31 and a filter for Market = NYSE; afterwards I clicked "Export"). I opened the .zip file on my Macbook and now had the .csv file in my downloads folder. I renamed the file to "equityshortinterest_2019.csv".

Now I did

Short_Interest <- read.csv("~/Downloads/equityshortinterest_2019.csv")

and got this error

Warning: cannot open file '/Users/macbook/Downloads/equityshortinterest_2019.csv': Operation not permittedError in file(file, "rt") : cannot open the connection

Other .csv files in my downloads folder can be opened this way! I checked the permissions for the file and I have read and write permissions. I also tried other ways to open the csv but all gave me the same error.

Does anyone know how to solve this? Thanks in advance!

The issue you are seeing is unlikely related to R (do you have the file open in some other program, for example?), but for more robust workflows you might want to keep external datasets in your RStudio project folder. And if you are not using projects, you probably should, you'd get reliable relative paths.

And/or automate as much as you can, i.e. instead of manually extracting zip archives and renaming resulting files, use something like this instead:

# In Windows, "~" refers to "username/Documents" by default
archive <- "~/../Downloads/equityshortinterest.zip" 
file.exists(archive)
#> [1] TRUE

# unzip(..., list = TRUE) returns dataframe with zip content, 
# [1,1] is Name of the first file
df_ <- unz(archive, unzip(archive, list = TRUE)[1,1]) |> read.csv()

Or just use readr::read_csv() on the archive itself; assuming there's a single CSV in zip, it will handle extraction for you:

readr::read_csv(archive)
#> Rows: 73942 Columns: 11
#> ── Column specification ────────────────────────────────────────────────────────
#> Delimiter: ","
#> chr  (4): Issue Name, Symbol, Market, Revision Flag
#> dbl  (6): Current Short, Previous Short, Chg, % Change from Previous, Avg Da...
#> date (1): Settlement Date
#> 
#> ℹ Use `spec()` to retrieve the full column specification for this data.
#> ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
#> # A tibble: 73,942 × 11
#>    `Settlement Date` `Issue Name` Symbol Market `Current Short` `Previous Short`
#>    <date>            <chr>        <chr>  <chr>            <dbl>            <dbl>
#>  1 2019-12-31        3D Systems … DDD    NYSE          23804548         23954237
#>  2 2019-12-31        3M Company   MMM    NYSE           9949341          9957296
#>  3 2019-12-31        500.com Lim… WBAI   NYSE           2509504          2563847
# ...
#> # ℹ 73,932 more rows
#> # ℹ 5 more variables: Chg <dbl>, `% Change from Previous` <dbl>,
#> #   `Avg Daily Vol` <dbl>, `Days to Cover` <dbl>, `Revision Flag` <chr>

For download automation httr / httr2 would suffice:

library(httr2)

# extract cURL request from browser's dev tools and pass it to 
# httr2::curl_translate() for quick and dirty httr2 prototype

dest <- tempfile(fileext = ".zip")
request("https://services-dynarep.ddwa.finra.org/public/reporting/v2/data/export/group/OTCMarket/name/ConsolidatedShortInterest") %>% 
  req_headers(`content-type` = "application/json") %>% 
  req_body_raw('{"aggregationFilter":null,"arrayFields":[],"compareFilters":[{"fieldName":"marketClassCode","fieldValue":"NYSE","compareType":"EQUAL"}],"dateRangeFilters":[{"fieldName":"settlementDate","startDate":"2019-01-01","endDate":"2019-01-15"}],"delimiter":null,"domainFilters":[],"fileName":"equityshortinterest.csv","fileType":"CSV","groupFields":[],"limit":5000,"mainFields":[{"id":"settlementDate","displayName":"Settlement Date","include":true,"primary":false},{"id":"issueName","displayName":"Issue Name","include":true,"primary":false},{"id":"symbolCode","displayName":"Symbol","include":true,"primary":false},{"id":"marketClassCode","displayName":"Market","include":true,"primary":false},{"id":"currentShortPositionQuantity","displayName":"Current Short","include":true,"primary":false},{"id":"previousShortPositionQuantity","displayName":"Previous Short","include":true,"primary":false},{"id":"changePreviousNumber","displayName":"Chg","include":true,"primary":false},{"id":"changePercent","displayName":"% Change from Previous","include":true,"primary":false},{"id":"averageDailyVolumeQuantity","displayName":"Avg Daily Vol","include":true,"primary":false},{"id":"daysToCoverQuantity","displayName":"Days to Cover","include":true,"primary":false},{"id":"revisionFlag","displayName":"Revision Flag","include":true,"primary":false}],"offset":0,"orFilters":[],"quoteValues":false,"sortFields":["-settlementDate","+issueName"],"singleFile":false,"zipped":true}') %>%
  req_perform(path = dest)
readr::read_csv(dest)

^{Created on 2023-05-07 with reprex v2.0.2}