英文:
SFTP Integration - SftpInboundFileSynchronizer - How not to download same file again
问题
在我的当前应用程序中，使用 Spring Batch 作业，我触发进程将远程 SFTP 文件传输到本地目录，处理该文件，然后在处理后删除文件。
@Bean("ftpMessageSource")
@EndpointId("streamInboundAdapter")
@InboundChannelAdapter(channel = "sftpChannel", poller = @Poller(fixedDelay = "5000"), autoStartup = "false")
public MessageSource<File> sftpMessageSource() {
SftpInboundFileSynchronizingMessageSource source =
new SftpInboundFileSynchronizingMessageSource(sftpInboundFileSynchronizer());
source.setLocalDirectory(new File("sftp-inbound"));
source.setAutoCreateLocalDirectory(true);
source.setLocalFilter(new FileSystemPersistentAcceptOnceFileListFilter(metadataStore(),""));
source.setMaxFetchSize(10);
return source;
}
@Bean(name="fileStore")
public PropertiesPersistingMetadataStore metadataStore() {
PropertiesPersistingMetadataStore metadataStore = new PropertiesPersistingMetadataStore();
metadataStore.setBaseDirectory("filestore");
metadataStore.setFileName("filestore.properties");
metadataStore.afterPropertiesSet();
return metadataStore;
}
在处理每个文件时，我将文件名的条目添加到 fileStore.properties 文件中。
metadataStore.put(file.getName(),file.getName());
我遇到的一个问题是，在下次处理时（无需重新启动服务器并再次启动相同的进程），进程会再次获取相同的一组要处理的文件。
我不想将已处理的文件通过 SFTP 传输，您能否指出我缺少哪个配置以避免重新下载相同的文件。
英文:
In my current application, using spring-batch job, I trigger the process to SFTP remote files to local directory, process it and delete files post processing.
@Bean("ftpMessageSource")
@EndpointId("streamInboundAdapter")
@InboundChannelAdapter(channel = "sftpChannel", poller = @Poller(fixedDelay = "5000"), autoStartup = "false")
public MessageSource<File> sftpMessageSource() {
SftpInboundFileSynchronizingMessageSource source =
new SftpInboundFileSynchronizingMessageSource(sftpInboundFileSynchronizer());
source.setLocalDirectory(new File("sftp-inbound"));
source.setAutoCreateLocalDirectory(true);
source.setLocalFilter(new FileSystemPersistentAcceptOnceFileListFilter(metadataStore(),""));
source.setMaxFetchSize(10);
return source;
}
@Bean(name="fileStore")
public PropertiesPersistingMetadataStore metadataStore() {
PropertiesPersistingMetadataStore metadataStore = new PropertiesPersistingMetadataStore();
metadataStore.setBaseDirectory("filestore");
metadataStore.setFileName("filestore.properties");
metadataStore.afterPropertiesSet();
return metadataStore;
}
As every file is processed, I make an entry of filename to fileStore.properties file.
metadataStore.put(file.getName(),file.getName());
One issue, I ran into next time processing (w/o restarting server and kicking off the same process again) is - process gets same set of files again for processing.
I do not want to SFTP processed files, can you please point out which configuration I am missing to avoid downloading same files again.
答案1
得分: 2
> 并在处理后删除文件。
因此，本地目录中不再存在该文件。由于您没有筛选远程文件，它们将作为新的本地副本再次下载。
FileSystemPersistentAcceptOnceFileListFilter 逻辑基于 file.lastModified()，如果它与现有条目不同，则会替换，因此会向下游推送。
考虑在 sftpInboundFileSynchronizer 上使用 SftpPersistentAcceptOnceFileListFilter，这样相同的文件（如果lastModified相同）将不会从SFTP中拉取。
在文档中了解更多信息：https://docs.spring.io/spring-integration/docs/5.3.2.RELEASE/reference/html/sftp.html#sftp-inbound
英文:
> and delete files post processing.
So, file does not exist in local directory any more. Since you don't filter remote files, they are downloaded again as a new local copy.
The FileSystemPersistentAcceptOnceFileListFilter logic is based on the file.lastModified() and if it is different from an existing entry, it is replaced and therefore pushed downstream.
Consider using an SftpPersistentAcceptOnceFileListFilter on the sftpInboundFileSynchronizer. This way the same file (if the same lastModified) is not going to be pulled from SFTP.
See more in docs: https://docs.spring.io/spring-integration/docs/5.3.2.RELEASE/reference/html/sftp.html#sftp-inbound
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论