S3使用预签名URL进行Golang多部分上传

huangapple go评论114阅读模式
英文:

S3 golang Multipart upload using Presigned url

问题

我们正在使用"go"语言编写一个应用程序,在我的服务中生成一个预签名URL(使用PutObjectRequest()和Presign()),然后将其传递给外部服务,以便外部服务可以使用预签名URL将较大的文件进行分块上传到S3。我们不想将AWS凭证提供给外部服务,这就是为什么我们计划提供一个预签名URL的原因。另外,由于文件较大,我们需要使用AWS SDK提供的分块上传功能。

我进行了快速搜索,没有找到AWS SDK中提供此功能的函数。
AWS go语言SDK是否支持使用预签名URL进行分块上传?有没有办法实现这个需求?

英文:

We are writing an application in "go" language where I generate a Presigned url from my service (using PutObjectRequest() and Presign()) and pass it to an external service so that the external service can perform the multipart upload of larger files to S3 (using presigned url). We do not want to give the AWS credentials to the external service, that is why we plan to provide a presigned url. Also, since the files are large we need to use the multipart upload feature provided by AWS SDK.

I did a quick search and could not find any functions that do this in the aws sdk.
Does the AWS sdk for go language support multipart upload using presigned url? Is there any way to achieve this?

答案1

得分: 1

创建请求到Amazon以获取用于分块上传的UploadID,从中创建多个部分并正常预签名每个部分。将这些发送给客户端以供使用。

func (s *s3) UploadMultipartPresignedCreate(key string) ([]string) {
  // 使用存储桶和键查询以获取UploadID
	out, err := s.s3.CreateMultipartUploadRequest(&s3.CreateMultipartUploadInput{
		Bucket: &s.bucketName,
		Key:    aws.String(key),
	})

	if err != nil {
		return nil
	}

  // 生成部分
	var parts = 10 // 部分数量
	requests := make([]string, 0, parts)

	for i := 0; i < parts; i++ {
		req, _ := i.s3.UploadPartRequest(&s3.UploadPartInput{
			Bucket:     &s.bucketName,
			Key:        aws.String(key),
			PartNumber: aws.Int64((int64)(i)),
			UploadId:   out.UploadId,
		})

    // 每次使用所需的到期时间进行预签名
		url, err := req.Presign(time.Hour)

		if err != nil {
			return nil
		}

		requests = append(requests, url)
	}

  // 返回预签名URL列表
	return requests, nil
}

此时,客户端应该成功或失败地上传并通过另一个端点(通常)回传完成的过程。此时,您将需要执行上传完成操作。

func (s *S3) UploadMultipartComplete(key string, uploadId string) (string) {
	out, err := s.s3.CompleteMultipartUpload(&s3.CompleteMultipartUploadInput{
		Bucket:   &s.bucketName,
		Key:      aws.String(key),
		UploadId: aws.String(uploadId),
	})

	if err != nil {
		return "" // 给出合理的错误
	}

	return *out.Key, nil
}

如果客户端告知您由于某种原因(可能是过期或改变主意)分块上传失败,您可以中止分块上传(并且应该这样做,因为直到过期时您将被收费)。这将替代上述的CompleteMultipartUpload操作。

out, err := i.s3.AbortMultipartUpload(&s3.AbortMultipartUploadInput{
	Bucket:   &i.bucketName,
	Key:      aws.String(key),
	UploadId: aws.String(uploadId),
})
英文:

Create the request to Amazon to get an UploadID for multipart, create several parts from it and pre-sign each normally. Send these to the client to use them.

func (s *s3) UploadMultipartPresignedCreate(key string) ([]string) {
  // Query with bucket and key to fetch UploadID
	out, err := s.s3.CreateMultipartUploadRequest(&s3.CreateMultipartUploadInput{
		Bucket: &s.bucketName,
		Key:    aws.String(key),
	})

	if err != nil {
		return nil
	}

  // Generate parts
	var parts = 10 // Number of parts
	requests := make([]string, 0, parts)

	for i := 0; i < parts; i++ {
		req, _ := i.s3.UploadPartRequest(&s3.UploadPartInput{
			Bucket:     &s.bucketName,
			Key:        aws.String(key),
			PartNumber: aws.Int64((int64)(i)),
			UploadId:   out.UploadId,
		})

    // Presign each time with whatever expiration time you want
		url, err := req.Presign(time.Hour)

		if err != nil {
			return nil
		}

		requests = append(requests, url)
	}

  // Returning a list of presigned URLs
	return requests, nil
}

At this point the client should succeed or fail their upload and ping you back with completion of process (on a separate endpoint, typically). At this point you'll want to carry out the Upload completion.

func (s *S3) UploadMultipartComplete(key string, uploadId string) (string) {
	out, err := s.s3.CompleteMultipartUpload(&s3.CompleteMultipartUploadInput{
		Bucket:   &s.bucketName,
		Key:      aws.String(key),
		UploadId: aws.String(uploadId),
	})

	if err != nil {
		return "" // Give a reasonable error
	}

	return *out.Key, nil
}

If the client informs you that for whatever reason (possibly expiry, or change of mind) the multipart upload has failed, you can instead abort the multipart (and you should, since you'll be charged until it expires). This would be done in place of the CompleteMultipartUpload above.

out, err := i.s3.AbortMultipartUpload(&s3.AbortMultipartUploadInput{
	Bucket:   &i.bucketName,
	Key:      aws.String(key),
	UploadId: aws.String(uploadId),
})

答案2

得分: -1

AWS Go SDK文档提供了一个使用预签名PUT URL进行上传的详细示例:https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/s3-example-presigned-urls.html

例如(从文档中复制相关部分):

// 创建S3服务客户端
svc := s3.New(sess)

resp, _ := svc.PutObjectRequest(&s3.PutObjectInput{
    Bucket: aws.String("testBucket"),
    Key:    aws.String("testKey"),
})

md5s := base64.StdEncoding.EncodeToString(h.Sum(nil))
resp.HTTPRequest.Header.Set("Content-MD5", md5s)

url, err := resp.Presign(15 * time.Minute)
if err != nil {
    fmt.Println("error presigning request", err)
    return
}

req, err := http.NewRequest("PUT", url, strings.NewReader(""))
req.Header.Set("Content-MD5", md5s)
if err != nil {
    fmt.Println("error creating request", url)
    return
}
英文:

The AWS Go SDK documentation provides a spelled out example of using presigned PUT URLs for uploading: https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/s3-example-presigned-urls.html

For example (copied relevant part from the documentation)

    // Create S3 service client
    svc := s3.New(sess)

    resp, _ := svc.PutObjectRequest(&s3.PutObjectInput{
        Bucket: aws.String("testBucket"),
        Key:    aws.String("testKey"),
    })

    md5s := base64.StdEncoding.EncodeToString(h.Sum(nil))
    resp.HTTPRequest.Header.Set("Content-MD5", md5s)

    url, err := resp.Presign(15 * time.Minute)
    if err != nil {
        fmt.Println("error presigning request", err)
        return
    }

    req, err := http.NewRequest("PUT", url, strings.NewReader(""))
    req.Header.Set("Content-MD5", md5s)
    if err != nil {
        fmt.Println("error creating request", url)
        return
    }

huangapple
  • 本文由 发表于 2017年4月28日 00:05:31
  • 转载请务必保留本文链接:https://go.coder-hub.com/43662755.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定