Why do I get an error reading an RDS file when the file exists and the path is correct?

For this question, it's really hard for me to make a reproducible example because the error happens on the high performance cluster. I am not able to reproduce the error locally.

I have this code:

summaries <- list()

for (a in c("A", "B")) {
  for (b in c("C", "D", "E", "F")) {
    for (c in c("01", "02", "03", "04", "05")) {
      summaries[[a]][[b]][[c]] <- readRDS(paste0("/home/user5556/Proj1/Simulation/", a, "/", b, "/", c, "/summary.Rds"))
    }
  }
}

saveRDS(summaries, "summaries.Rds")

Running this will return:

Error in readRDS(paste0("/home/user5556/Proj1/Simulation/", a, "/",  :
  error reading from connection

However, now observe what happens if in the previous block of code we replace readRDS with file.exists() and print it.

for (a in c("A", "B")) {
  for (b in c("C", "D", "E", "F")) {
    for (c in c("01", "02", "03", "04", "05")) {
      print(file.exists(paste0("/home/user5556/Proj1/Simulation/", a, "/", b, "/", c, "/summary.Rds")))
    }
  }
}

[1] TRUE
[1] TRUE
[1] TRUE
[1] TRUE
[1] TRUE
[1] TRUE
[1] TRUE
[1] TRUE
[1] TRUE
[1] TRUE
[1] TRUE
[1] TRUE
[1] TRUE
[1] TRUE
[1] TRUE
[1] TRUE
[1] TRUE
[1] TRUE
[1] TRUE
[1] TRUE
[1] TRUE
[1] TRUE
[1] TRUE
[1] TRUE
[1] TRUE
[1] TRUE
[1] TRUE
[1] TRUE
[1] TRUE
[1] TRUE
[1] TRUE
[1] TRUE
[1] TRUE
[1] TRUE
[1] TRUE
[1] TRUE
[1] TRUE
[1] TRUE
[1] TRUE
[1] TRUE

When manually typing out the file path, the file is successfully read:

readRDS("/home/user5556/Proj1/Simulation/A/C/01/summary.Rds")

This reads the RDS file as expected. So what exactly is happening here? It can't be the case that the file path is incorrectly specified, because then the file.exists() would return false. It's also not possible that I don't have the permissions to view the file, because then I wouldn't be able to read it in when typing out the path manually. So what's wrong?

Since we can't run the code, this is just a guess, but I'd guess one of the files is corrupted somehow. You tested the first one successfully, but the error doesn't say which one was problematic.

To debug it add debug output to the loop. For example,

summaries <- list()

for (a in c("A", "B")) {
  for (b in c("C", "D", "E", "F")) {
    for (c in c("01", "02", "03", "04", "05")) {
      filename <- paste0("/home/user5556/Proj1/Simulation/", a, "/", b, "/", c, "/summary.Rds")
      cat("Reading ", filename)
      summaries[[a]][[b]][[c]] <- readRDS(filename)
      cat(" successfully!\n")
    }
  }
}

saveRDS(summaries, "summaries.Rds")

When the error occurs, you have seen the "Reading" message, but not the "successfully" part, so you will know which file caused the problem.

If it turns out to be the first file, then I was wrong. But I'm guessing it won't. You can try readRDS(filename) outside of the loop, and see if the error repeats. I'm guessing it will.

How to fix it isn't clear, but you could just test for that particular filename and skip it, or redo the simulation, or fix file permissions as @Limey suggested.