Java.nio.file.NoSuchFileException但文件存在。

huangapple go评论74阅读模式
英文:

Java.nio.file.NoSuchFileException but files exits

问题

我正在尝试从一个文件夹中读取文件。该文件夹中有超过1000个文件,我尝试使用ScheduledThreadPoolExecutor.scheduleWithFixedDelay()每10秒读取100个文件。

Files.list(Paths.get(path)).sorted().limit(limit).forEach(file -> { //limit = 100
    try { 
        String content = new String(Files.readAllBytes(file), "UTF-8");
        //执行业务逻辑
        doneFile.add(fileName);
    } catch (Exception e) {
        log.error(e);
    }
});

if (doneFile.size() > 0) {
    for (String fileName : doneFile) {
        try {
            Files.delete(Paths.get(path + fileName));
        } catch (IOException e) {
            log.error(e);
        }
    }
}

大约有60-80个文件成功读取,其他文件抛出java.nio.file.NoSuchFileException。删除函数也抛出一些java.nio.file.NoSuchFileException

但最终,尽管有异常,所有文件都在一些线程后被读取和删除。

这种情况下的异常原因是什么,我该如何修复它呢?
非常感谢!
附言:对我的糟糕英语感到抱歉。

英文:

I'm trying to read files from a folder. There are >1000 files in that folder and i trying read 100 files every 10s by using ScheduledThreadPoolExecutor.scheduleWithFixedDelay()

Files.list(Paths.get(path)).sorted().limit(limit).forEach(file ->{ //limit = 100
                    try{ 
                        String content = new String(Files.readAllBytes(file), "UTF-8");
                        //do business logic
                        doneFile.add(fileName);
                    }catch (Exception e){
                        log.error(e);
                    }
                });

if(doneFile.size() > 0){
                for(String fileName: doneFile){
                    try {
                        Files.delete(Paths.get(path + fileName));
                    } catch (IOException e) {
                        log.error(e);
                    }
                }
            }

About 60-80 files were read successfully and the others throw java.nio.file.NoSuchFileException.
Delete function also throws some java.nio.file.NoSuchFileException.

But finally, all files were read and deleted after some threads in spite of exceptions

What cause exception in this case and how can i fix it?
Many thanks!
Ps. Sr about my bad English

答案1

得分: 2

我评论道:

> 听起来好像有其他东西在添加和删除文件。有可能您有两个定时任务在运行吗?有可能是外部进程在执行这个操作吗?

你回答道:

> 我检查了日志文件,发现ScheduledThreadPoolExecutor.scheduleWithFixedDelay()创建了10个线程,而不是预期的4个循环所期望的8个线程。

所以...听起来您有两个(或更多)线程在遍历同一个目录,处理和删除文件而没有任何协调。再加上Files.list会缓冲部分或全部正在列出的目录,就会出现以下问题:

  • 一个线程在另一个文件的流中删除一个文件,然后第二个线程尝试打开它。这会导致后者在尝试打开文件时出现NoSuchFileException

  • 两个线程同时处理同一个文件,其中一个在第二个线程仍在处理它时将其删除。这会导致后者在尝试删除该文件时出现NoSuchFileException

可能的解决方案:

  1. 编写代码以忽略这些异常。(但仍然可能会出现两个线程同时处理同一个文件的情况,这似乎是不好的。)

  2. 重构代码,以便在任何时候只有一个线程在扫描目录。

如果您在目录内部处理时需要并行性,则让目录处理线程为每个文件提交一个任务,提交到具有有限线程池的执行器中。

英文:

I commented:

> It sounds like something else is adding and removing files. Is it possible that you have two scheduled tasks running? Is it possible that an external process is doing this?

You replied:

> I checked log file and got that ScheduledThreadPoolExecutor.scheduleWithFixedDelay() created 10 threads instead of 8 as expected for 4 loops

So ... it sounds like you have two (or more) threads iterating through the same directory, processing and deleting files without any coordination. Combine that with the fact that Files.list is going to buffer part or all of the directory that it is listing, and you will get problems like:

  • One thread deletes a file that is in another file's stream before the second thread tries to open it. This leads to a NoSuchFileException when the latter tries to open the file.

  • Two threads process the same file at the same time, and one deletes it while the second thread is still processing it. This leads to a NoSuchFileException when the latter tries to delete the file.

Possible solutions:

  1. Write the code to ignore these exceptions. (But you could still have two threads processing the same file at the same time, which seems like a bad thing.)

  2. Restructure the code so that there is only one thread scanning the directory at any one time.

If you need parallelism when processing within the directory, have the directory processor thread submit a task for each file to an executor with a bounded thread pool.

huangapple
  • 本文由 发表于 2020年10月5日 15:50:16
  • 转载请务必保留本文链接:https://go.coder-hub.com/64204463.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定