英文:
Java.nio.file.NoSuchFileException but files exits
问题
我正在尝试从一个文件夹中读取文件。该文件夹中有超过1000个文件,我尝试使用ScheduledThreadPoolExecutor.scheduleWithFixedDelay()
每10秒读取100个文件。
Files.list(Paths.get(path)).sorted().limit(limit).forEach(file -> { //limit = 100
try {
String content = new String(Files.readAllBytes(file), "UTF-8");
//执行业务逻辑
doneFile.add(fileName);
} catch (Exception e) {
log.error(e);
}
});
if (doneFile.size() > 0) {
for (String fileName : doneFile) {
try {
Files.delete(Paths.get(path + fileName));
} catch (IOException e) {
log.error(e);
}
}
}
大约有60-80个文件成功读取,其他文件抛出java.nio.file.NoSuchFileException
。删除函数也抛出一些java.nio.file.NoSuchFileException
。
但最终,尽管有异常,所有文件都在一些线程后被读取和删除。
这种情况下的异常原因是什么,我该如何修复它呢?
非常感谢!
附言:对我的糟糕英语感到抱歉。
英文:
I'm trying to read files from a folder. There are >1000 files in that folder and i trying read 100 files every 10s by using ScheduledThreadPoolExecutor.scheduleWithFixedDelay()
Files.list(Paths.get(path)).sorted().limit(limit).forEach(file ->{ //limit = 100
try{
String content = new String(Files.readAllBytes(file), "UTF-8");
//do business logic
doneFile.add(fileName);
}catch (Exception e){
log.error(e);
}
});
if(doneFile.size() > 0){
for(String fileName: doneFile){
try {
Files.delete(Paths.get(path + fileName));
} catch (IOException e) {
log.error(e);
}
}
}
About 60-80 files were read successfully and the others throw java.nio.file.NoSuchFileException
.
Delete function also throws some java.nio.file.NoSuchFileException
.
But finally, all files were read and deleted after some threads in spite of exceptions
What cause exception in this case and how can i fix it?
Many thanks!
Ps. Sr about my bad English
答案1
得分: 2
我评论道:
> 听起来好像有其他东西在添加和删除文件。有可能您有两个定时任务在运行吗?有可能是外部进程在执行这个操作吗?
你回答道:
> 我检查了日志文件,发现ScheduledThreadPoolExecutor.scheduleWithFixedDelay()
创建了10个线程,而不是预期的4个循环所期望的8个线程。
所以...听起来您有两个(或更多)线程在遍历同一个目录,处理和删除文件而没有任何协调。再加上Files.list
会缓冲部分或全部正在列出的目录,就会出现以下问题:
-
一个线程在另一个文件的流中删除一个文件,然后第二个线程尝试打开它。这会导致后者在尝试打开文件时出现
NoSuchFileException
。 -
两个线程同时处理同一个文件,其中一个在第二个线程仍在处理它时将其删除。这会导致后者在尝试删除该文件时出现
NoSuchFileException
。
可能的解决方案:
-
编写代码以忽略这些异常。(但仍然可能会出现两个线程同时处理同一个文件的情况,这似乎是不好的。)
-
重构代码,以便在任何时候只有一个线程在扫描目录。
如果您在目录内部处理时需要并行性,则让目录处理线程为每个文件提交一个任务,提交到具有有限线程池的执行器中。
英文:
I commented:
> It sounds like something else is adding and removing files. Is it possible that you have two scheduled tasks running? Is it possible that an external process is doing this?
You replied:
> I checked log file and got that ScheduledThreadPoolExecutor.scheduleWithFixedDelay()
created 10 threads instead of 8 as expected for 4 loops
So ... it sounds like you have two (or more) threads iterating through the same directory, processing and deleting files without any coordination. Combine that with the fact that Files.list
is going to buffer part or all of the directory that it is listing, and you will get problems like:
-
One thread deletes a file that is in another file's stream before the second thread tries to open it. This leads to a
NoSuchFileException
when the latter tries to open the file. -
Two threads process the same file at the same time, and one deletes it while the second thread is still processing it. This leads to a
NoSuchFileException
when the latter tries to delete the file.
Possible solutions:
-
Write the code to ignore these exceptions. (But you could still have two threads processing the same file at the same time, which seems like a bad thing.)
-
Restructure the code so that there is only one thread scanning the directory at any one time.
If you need parallelism when processing within the directory, have the directory processor thread submit a task for each file to an executor with a bounded thread pool.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论