英文:
S3 golang Multipart upload using Presigned url
问题
我们正在使用"go"语言编写一个应用程序,在我的服务中生成一个预签名URL(使用PutObjectRequest()和Presign()),然后将其传递给外部服务,以便外部服务可以使用预签名URL将较大的文件进行分块上传到S3。我们不想将AWS凭证提供给外部服务,这就是为什么我们计划提供一个预签名URL的原因。另外,由于文件较大,我们需要使用AWS SDK提供的分块上传功能。
我进行了快速搜索,没有找到AWS SDK中提供此功能的函数。
AWS go语言SDK是否支持使用预签名URL进行分块上传?有没有办法实现这个需求?
英文:
We are writing an application in "go" language where I generate a Presigned url from my service (using PutObjectRequest() and Presign()) and pass it to an external service so that the external service can perform the multipart upload of larger files to S3 (using presigned url). We do not want to give the AWS credentials to the external service, that is why we plan to provide a presigned url. Also, since the files are large we need to use the multipart upload feature provided by AWS SDK.
I did a quick search and could not find any functions that do this in the aws sdk.
Does the AWS sdk for go language support multipart upload using presigned url? Is there any way to achieve this?
答案1
得分: 1
创建请求到Amazon以获取用于分块上传的UploadID,从中创建多个部分并正常预签名每个部分。将这些发送给客户端以供使用。
func (s *s3) UploadMultipartPresignedCreate(key string) ([]string) {
// 使用存储桶和键查询以获取UploadID
out, err := s.s3.CreateMultipartUploadRequest(&s3.CreateMultipartUploadInput{
Bucket: &s.bucketName,
Key: aws.String(key),
})
if err != nil {
return nil
}
// 生成部分
var parts = 10 // 部分数量
requests := make([]string, 0, parts)
for i := 0; i < parts; i++ {
req, _ := i.s3.UploadPartRequest(&s3.UploadPartInput{
Bucket: &s.bucketName,
Key: aws.String(key),
PartNumber: aws.Int64((int64)(i)),
UploadId: out.UploadId,
})
// 每次使用所需的到期时间进行预签名
url, err := req.Presign(time.Hour)
if err != nil {
return nil
}
requests = append(requests, url)
}
// 返回预签名URL列表
return requests, nil
}
此时,客户端应该成功或失败地上传并通过另一个端点(通常)回传完成的过程。此时,您将需要执行上传完成操作。
func (s *S3) UploadMultipartComplete(key string, uploadId string) (string) {
out, err := s.s3.CompleteMultipartUpload(&s3.CompleteMultipartUploadInput{
Bucket: &s.bucketName,
Key: aws.String(key),
UploadId: aws.String(uploadId),
})
if err != nil {
return "" // 给出合理的错误
}
return *out.Key, nil
}
如果客户端告知您由于某种原因(可能是过期或改变主意)分块上传失败,您可以中止分块上传(并且应该这样做,因为直到过期时您将被收费)。这将替代上述的CompleteMultipartUpload
操作。
out, err := i.s3.AbortMultipartUpload(&s3.AbortMultipartUploadInput{
Bucket: &i.bucketName,
Key: aws.String(key),
UploadId: aws.String(uploadId),
})
英文:
Create the request to Amazon to get an UploadID for multipart, create several parts from it and pre-sign each normally. Send these to the client to use them.
func (s *s3) UploadMultipartPresignedCreate(key string) ([]string) {
// Query with bucket and key to fetch UploadID
out, err := s.s3.CreateMultipartUploadRequest(&s3.CreateMultipartUploadInput{
Bucket: &s.bucketName,
Key: aws.String(key),
})
if err != nil {
return nil
}
// Generate parts
var parts = 10 // Number of parts
requests := make([]string, 0, parts)
for i := 0; i < parts; i++ {
req, _ := i.s3.UploadPartRequest(&s3.UploadPartInput{
Bucket: &s.bucketName,
Key: aws.String(key),
PartNumber: aws.Int64((int64)(i)),
UploadId: out.UploadId,
})
// Presign each time with whatever expiration time you want
url, err := req.Presign(time.Hour)
if err != nil {
return nil
}
requests = append(requests, url)
}
// Returning a list of presigned URLs
return requests, nil
}
At this point the client should succeed or fail their upload and ping you back with completion of process (on a separate endpoint, typically). At this point you'll want to carry out the Upload completion.
func (s *S3) UploadMultipartComplete(key string, uploadId string) (string) {
out, err := s.s3.CompleteMultipartUpload(&s3.CompleteMultipartUploadInput{
Bucket: &s.bucketName,
Key: aws.String(key),
UploadId: aws.String(uploadId),
})
if err != nil {
return "" // Give a reasonable error
}
return *out.Key, nil
}
If the client informs you that for whatever reason (possibly expiry, or change of mind) the multipart upload has failed, you can instead abort the multipart (and you should, since you'll be charged until it expires). This would be done in place of the CompleteMultipartUpload
above.
out, err := i.s3.AbortMultipartUpload(&s3.AbortMultipartUploadInput{
Bucket: &i.bucketName,
Key: aws.String(key),
UploadId: aws.String(uploadId),
})
答案2
得分: -1
AWS Go SDK文档提供了一个使用预签名PUT URL进行上传的详细示例:https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/s3-example-presigned-urls.html
例如(从文档中复制相关部分):
// 创建S3服务客户端
svc := s3.New(sess)
resp, _ := svc.PutObjectRequest(&s3.PutObjectInput{
Bucket: aws.String("testBucket"),
Key: aws.String("testKey"),
})
md5s := base64.StdEncoding.EncodeToString(h.Sum(nil))
resp.HTTPRequest.Header.Set("Content-MD5", md5s)
url, err := resp.Presign(15 * time.Minute)
if err != nil {
fmt.Println("error presigning request", err)
return
}
req, err := http.NewRequest("PUT", url, strings.NewReader(""))
req.Header.Set("Content-MD5", md5s)
if err != nil {
fmt.Println("error creating request", url)
return
}
英文:
The AWS Go SDK documentation provides a spelled out example of using presigned PUT URLs for uploading: https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/s3-example-presigned-urls.html
For example (copied relevant part from the documentation)
// Create S3 service client
svc := s3.New(sess)
resp, _ := svc.PutObjectRequest(&s3.PutObjectInput{
Bucket: aws.String("testBucket"),
Key: aws.String("testKey"),
})
md5s := base64.StdEncoding.EncodeToString(h.Sum(nil))
resp.HTTPRequest.Header.Set("Content-MD5", md5s)
url, err := resp.Presign(15 * time.Minute)
if err != nil {
fmt.Println("error presigning request", err)
return
}
req, err := http.NewRequest("PUT", url, strings.NewReader(""))
req.Header.Set("Content-MD5", md5s)
if err != nil {
fmt.Println("error creating request", url)
return
}
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论