英文:
Can file just pass trough the server and to AWS S3?
问题
我正在尝试将大文件上传到S3，但是：
- 不在服务器上存储文件
- 不在“上传文件到服务器”后再“上传文件到S3”
我想要实现的是：一旦文件上传开始，就直接将文件上传到S3，这样我就不必等待文件上传到服务器，然后再等待文件上传到S3。
我刚开始使用GoLang，对我在做什么大部分时间都不太清楚。
到目前为止，我正在使用echo (https://echo.labstack.com)。
file, err := c.FormFile("file")
if err != nil {
return err
}
src, err := file.Open()
if err != nil {
return err
}
defer src.Close()
如果我理解正确，此时我有一个io.Reader（src变量），在我可以使用该读取器开始上传到S3之前，我不必等待文件上传到服务器？
基本上，当文件被发送到服务器时，服务器同时将文件发送到S3。
或者我理解错了吗？
使用io.Pipe会更好吗？
英文:
I'm trying to upload large file to S3 but:
- without file being stored on the server
- without "upload file to server" and when it's done "upload file to S3"
What I'm trying to accomplish is: as soon as file upload stars - pipe that file upload directly to S3 so that I don't have to wait for file to be uploaded to the server and then wait again for file to be uploaded to S3.
I've only started using GoLang and I mostly have no idea what I'm doing.
So far I'm using echo (https://echo.labstack.com).
file, err := c.FormFile("file")
if err != nil {
return err
}
src, err := file.Open()
if err != nil {
return err
}
defer src.Close()
If I understood correctly at this point i have io.Reader (src variable) and I don't have to wait for file to be uploaded to server, before I can use that reader to start uploading to S3?
Basically while file is being sent to the server, at the same time server is sending the file to S3.
Or am I wrong?
Is it better to use io.Pipe?
答案1
得分: 2
我假设你想要从客户端（Web、移动、桌面应用程序）上传文件到S3，而不将文件存储在你的服务器上。
使用官方的AWS SDK，可以采用以下方法：
https://stackoverflow.com/questions/47621804/upload-object-to-aws-s3-without-creating-a-file-using-aws-sdk-go
另外，如果你想要完全跳过服务器进行上传，并让客户端应用程序直接上传到你的S3存储桶，你可以参考S3预签名上传URL。
现在，你的服务器的角色是预签名上传请求，这意味着它将授权客户端将文件上传到你的S3存储桶。对于大文件，你将需要使用多部分上传，并且你的服务器将需要预签名每个部分。
英文:
I'm assuming you want to upload a file from a client (web, mobile, desktop applications) to S3 without storing the file in your server.
A possible approach using official AWS SDK is answered here:
https://stackoverflow.com/questions/47621804/upload-object-to-aws-s3-without-creating-a-file-using-aws-sdk-go
Alternatively, if you want to skip your server entirely for uploading and let your client apps upload directly to your S3 bucket, you might want to take a look at S3 pre-signed upload URL.
Your server now plays the role of pre-signing the upload request, meaning it will authorize clients to upload the files to your S3 bucket. For large file, you will have to use multi-part upload and your server will need to pre-sign each part.
答案2
得分: 1
实际上，你并没有真正地“拥有”io.Reader。它只是由io包提供的一个接口。如果你感兴趣的话，io.Pipe会返回(*io.PipeReader, *io.PipeWriter)。
上面的io.PipeReader是io.Reader接口的一个实际实现，因此在上传时可以直接在golang aws sdk for s3中使用。
英文:
Actually you don't really "have" io.Reader. It is just an interface provided by the io package. io.Pipe, if you are interested, returns
(*io.PipeReader, *io.PipeWriter)
The io.PipeReader above is an actual implementation of the io.Reader interface - and hence can be used directly in golang aws sdk for s3 when uploading.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论