英文:
Error 'No such file or directory' while sending files from s3 bucket using airflow 'EmailOperator'
问题
我需要使用Airflow EmailOperator发送位于AWS S3存储桶中的CSV文件。
嗨，
当我尝试使用Airflow EmailOperator将S3存储桶中的2个CSV文件发送给收件人时，我遇到了'No such file or directory'错误。
我是否正确地从S3存储桶发送文件？或者这是否可能？
此外，我不希望在这方面使用任何本地存储，原因是我的MWAA具有最小的临时存储，如果在过程中更改了工作节点，文件可能不会在临时存储中可用于工作节点进行拾取。
请协助。
英文:
I need to send csv files present in aws s3 bucket using airflow EmailOperator.
Hi,
I am facing 'No such file or directory' error when I am trying to send 2 csv files present in s3 bucket to a recipient using Airflow EmailOperator.
Am I doing right, sending files from s3 bucket ? Or Is it possible ?
Also, I do not want any local storage to be used in this regards, the reason being, my MWAA has minimal temp storage and file may not be available in the temp for the worker node to pick it if worker node gets changed in the process.
Please assist.
答案1
得分: 1
我尚未尝试过这样做，但我对EmailOperator的工作原理有一定了解，文件需要位于本地文件系统上。如果您可以控制任务执行的位置，可能可以创建一个任务来下载这些文件，然后在第二个任务中发送电子邮件。但是，如果您正在使用类似MWAA的东西，这将不起作用，因为执行任务的工作进程在某种程度上是“无状态”的，不能保证两个任务将在同一底层计算主机上执行。
一个替代方法可能是创建一个生成已签名的S3 URL 并链接到S3附件的函数。
英文:
I have not tried to do this, but my understanding of how the EmailOperator works is that the files need to be on the local filesystem. If you have control over where your tasks are executed, its possible you could create a task that downloads those files in one task, and then email them in a second. However, if you are using something like MWAA this is not going to work as workers that execute tasks are "stateless" in so far as you cannot guarantee that two tasks will execute on the same underlying compute host.
One alternative might be to have a function that generates a signed S3 URL and then links to the S3 attachments that way.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论