GCP云存储 – 使用Golang aws sdk2 – 使用S3互操作凭证上传文件

huangapple
go评论160阅读模式
英文:

GCP Cloud Storage - Golang aws sdk2 - Upload file with s3 INTEROPERABILITY Creds

问题

我正在尝试使用s3 go sdk aws-sdk-go-v2通过互操作性功能从/向云存储桶下载/上传文件。

下载按预期工作，但上传不起作用，出现以下错误消息：SDK 2022/09/14 11:24:43 DEBUG request failed with unretryable error https response error StatusCode: 403, RequestID: , HostID: , api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method.

由于我在下载和上传中使用相同的access_key和secret_key，所以似乎不是凭据的问题。
此外，hmac密钥背后的服务帐号具有存储对象管理员角色。

以下是代码：

main.go

  1. package main
  3. import (
  4. 	"context"
  5. 	"fmt"
  6. 	"os"
  7. 	"strings"
  9. 	"github.com/aws/aws-sdk-go-v2/aws"
  10. 	"github.com/aws/aws-sdk-go-v2/config"
  11. 	"github.com/aws/aws-sdk-go-v2/credentials"
  12. 	"github.com/aws/aws-sdk-go-v2/feature/s3/manager"
  13. 	"github.com/aws/aws-sdk-go-v2/service/s3"
  14. )
  16. var BUCKET_NAME = ""
  18. func main() {
  20. 	//准备gcp解析器
  21. 	gcpResolver := aws.EndpointResolverWithOptionsFunc(func(service, region string, options ...interface{}) (aws.Endpoint, error) {
  22. 		return aws.Endpoint{
  23. 			URL:               "https://storage.googleapis.com",
  24. 			SigningRegion:     "auto",
  25. 			Source:            aws.EndpointSourceCustom,
  26. 			HostnameImmutable: true,
  27. 		}, nil
  28. 	})
  29. 	//文件格式：$accessKey:$secretKey
  30. 	file, _ := os.ReadFile("/home/bapt/creds/amz-keys-gcp-2")
  31. 	keys := strings.Split(string(file), ":")
  33. 	//初始化配置选项
  34. 	optConfig := []func(*config.LoadOptions) error{
  35. 		config.WithRegion("auto"),
  36. 		config.WithCredentialsProvider(credentials.NewStaticCredentialsProvider(keys[0], strings.TrimRight(keys[1], "\n"), "")),
  37. 		config.WithClientLogMode(aws.LogRetries | aws.LogRequestWithBody | aws.LogResponseWithBody | aws.LogRequestEventMessage | aws.LogResponseEventMessage | aws.LogSigning),
  38. 		config.WithEndpointResolverWithOptions(gcpResolver),
  39. 	}
  41. 	//初始化配置
  42. 	cfg, _ := config.LoadDefaultConfig(context.TODO(), optConfig...)
  44. 	//初始化服务
  45. 	svc := s3.NewFromConfig(cfg)
  46. 	tempFile, _ := os.CreateTemp("/tmp", "test-gcp-*")
  47. 	defer tempFile.Close()
  48. 	downloader := manager.NewDownloader(svc)
  49. 	downloader.Download(context.TODO(), tempFile, &s3.GetObjectInput{
  50. 		Bucket: aws.String(BUCKET_NAME),
  51. 		Key:    aws.String("file-test.txt"),
  52. 	})
  53. 	//初始化上传器（非分块）
  54. 	uploader := manager.NewUploader(svc, func(u *manager.Uploader) {
  55. 		u.Concurrency = 1
  56. 		u.MaxUploadParts = 1
  57. 	})
  58. 	//上传
  59. 	_, err := uploader.Upload(context.TODO(), &s3.PutObjectInput{
  60. 		Bucket: aws.String(BUCKET_NAME),
  61. 		Key:    aws.String("file-test.txt"),
  62. 		Body:   strings.NewReader("HELLO"),
  63. 	})
  64. 	fmt.Println(err)
  65. }

go.mod

  1. module gcps3/v2
  3. go 1.18
  5. require (
  6. 	github.com/aws/aws-sdk-go-v2 v1.16.14
  7. 	github.com/aws/aws-sdk-go-v2/config v1.17.5
  8. 	github.com/aws/aws-sdk-go-v2/credentials v1.12.18
  9. 	github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.11.31
  10. 	github.com/aws/aws-sdk-go-v2/service/s3 v1.27.9
  11. )
  13. require (
  14. 	github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.4.7 // indirect
  15. 	github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.12.15 // indirect
  16. 	github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.21 // indirect
  17. 	github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.4.15 // indirect
  18. 	github.com/aws/aws-sdk-go-v2/internal/ini v1.3.22 // indirect
  19. 	github.com/aws/aws-sdk-go-v2/internal/v4a v1.0.12 // indirect
  20. 	github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.9.8 // indirect
  21. 	github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.1.16 // indirect
  22. 	github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.9.15 // indirect
  23. 	github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.13.15 // indirect
  24. 	github.com/aws/aws-sdk-go-v2/service/sso v1.11.21 // indirect
  25. 	github.com/aws/aws-sdk-go-v2/service/ssooidc v1.13.3 // indirect
  26. 	github.com/aws/aws-sdk-go-v2/service/sts v1.16.17 // indirect
  27. 	github.com/aws/smithy-go v1.13.2 // indirect
  28. 	github.com/jmespath/go-jmespath v0.4.0 // indirect
  29. )

这是PUT的调试跟踪（我的桶名已替换为BUCKET，access_key替换为GOOG1ID）：

  1. SDK 2022/09/14 14:52:37 DEBUG Request Signature:
  2. ---[ CANONICAL STRING  ]-----------------------------
  3. PUT
  4. /BUCKET/file-test.txt
  5. x-id=PutObject
  6. accept-encoding:identity
  7. amz-sdk-invocation-id:d6776820-e336-4bdf-afa3-0ca3b6d5b0a0
  8. amz-sdk-request:attempt=1; max=3
  9. content-length:5
  10. content-type:application/octet-stream
  11. host:storage.googleapis.com
  12. x-amz-content-sha256:UNSIGNED-PAYLOAD
  13. x-amz-date:20220914T125237Z
  15. accept-encoding;amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-content-sha256;x-amz-date
  16. UNSIGNED-PAYLOAD
  17. ---[ STRING TO SIGN ]--------------------------------
  18. AWS4-HMAC-SHA256
  19. 20220914T125237Z
  20. 20220914/auto/s3/aws4_request
  21. bc09.....daf520
  22. -----------------------------------------------------
  23. SDK 2022/09/14 14:52:37 DEBUG Request
  24. PUT /BUCKET/file-test.txt?x-id=PutObject HTTP/1.1
  25. Host: storage.googleapis.com
  26. User-Agent: aws-sdk-go-v2/1.16.14 os/linux lang/go/1.18.1 md/GOOS/linux md/GOARCH/amd64 api/s3/1.27.9 ft/s3-transfer
  27. Content-Length: 5
  28. Accept-Encoding: identity
  29. Amz-Sdk-Invocation-Id: d6776820-e336-4bdf-afa3-0ca3b6d5b0a0
  30. Amz-Sdk-Request: attempt=1; max=3
  31. Authorization: AWS4-HMAC-SHA256 Credential=GOOG1ID/20220914/auto/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-content-sha256;x-amz-date, Signature=c994....37d0
  32. Content-Type: application/octet-stream
  33. X-Amz-Content-Sha256: UNSIGNED-PAYLOAD
  34. X-Amz-Date: 20220914T125237Z
  36. HELLO
  37. SDK 2022/09/14 14:52:37 DEBUG Response
  38. HTTP/2.0 403 Forbidden
  39. Content-Length: 883
  40. Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
  41. Content-Type: application/xml; charset=UTF-8
  42. Date: Wed, 14 Sep 2022 12:52:38 GMT
  43. Server: UploadServer
  44. X-Guploader-Uploadid: ADPycdt0aCu6BTmzWQl2Ehc4q2sP8rtexDb4Keyn6cQL_GigREvc8T1CzX0HH-ZXgw_6XWLJPPYXufwRCr0Sl7uSsiIi0Q
  46. <?xml version='1.0' encoding='UTF-8'?><Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method.</Message><StringToSign>AWS4-HMAC-SHA256
  47. 20220914T125237Z
  48. 20220914/auto/s3/aws4_request
  49. 0a10....d63e</StringToSign><CanonicalRequest>PUT
  50. /BUCKET/file-test.txt
  51. x-id=PutObject
  52. accept-encoding:identity,gzip(gfe)
  53. amz-sdk-invocation-id:d6776820-e336-4bdf-afa3-0ca3b6d5b0a0
  54. amz-sdk-request:attempt=1; max=3
  55. content-length:5
  56. content-type:application/octet-stream
  57. host:storage.googleapis.com
  58. x-amz-content-sha256:UNSIGNED-PAYLOAD
  59. x-amz-date:20220914T125237Z
  61. accept-encoding;amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-content-sha256;x-amz-date
  62. UNSIGNED-PAYLOAD</CanonicalRequest></Error>
  63. SDK 2022/09/14 14:52:37 DEBUG request failed with unretryable error https response error StatusCode: 403, RequestID: , HostID: , api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method.
  64. operation error S3: PutObject, https response error StatusCode: 403, RequestID: , HostID: , api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method.

我使用此Python脚本测试了上传，并且它可以正常工作（相同的凭据）：

  1. boto3.set_stream_logger('', logging.DEBUG)
  2. GCP_BUCKET = True
  3. FILE_TO_BE_UPLOADED = '/tmp/toto'
  5. if GCP_BUCKET:
  6.     ACCESS_KEY = "ACESS"
  7.     SECRET_KEY = "SECRET"
  8.     bucket_name = "BUCKET"    
  9.     region_name="auto"
  10.     endpoint_url="https://storage.googleapis.com"
  11.     
  12. def makeS3Client():
  13.     s3 = boto3.client("s3", 
  14.                 region_name=region_name,
  15.                 endpoint_url=endpoint_url,
  16.                 aws_access_key_id=ACCESS_KEY,
  17.                 aws_secret_access_key=SECRET_KEY,
  18.                 )
  20.     return s3
  23. def upload_file(s3,bucket_name,fname):
  24.     """
  25.     Uploads file to S3 bucket using S3 client object
  26.     :return: None
  27.     """
  28.     object_name = os.path.basename(fname)
  29.     file_name = os.path.abspath(fname)
  30.     #file_name = os.path.join(pathlib.Path(__file__).parent.resolve(), fname)
  32.     response = s3.upload_file(file_name, bucket_name, object_name)
  33.     #print(response)  # prints None

我注意到Go和Python之间唯一的区别是所使用的signedHeaders。

但是，使用s3 aws存储桶的Go代码正常工作。

我是否遗漏了某个选项？

谢谢你的帮助。

英文:

I'm trying to implements download/upload a file from/to a bucket in cloud storage via the s3 go sdk aws-sdk-go-v2 using the Interoperability feature

The download is working as expected, but the upload isnt working, with this error message: SDK 2022/09/14 11:24:43 DEBUG request failed with unretryable error https response error StatusCode: 403, RequestID: , HostID: , api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method.

As I use same access_key and secret_key for both download and upload, it does not seems to be a credentials problem.
Plus, the service account behind the hmac keys has the Storage object Admin Role.

Here the code:

main.go

  1. package main
  2. import (
  3. &quot;context&quot;
  4. &quot;fmt&quot;
  5. &quot;os&quot;
  6. &quot;strings&quot;
  7. &quot;github.com/aws/aws-sdk-go-v2/aws&quot;
  8. &quot;github.com/aws/aws-sdk-go-v2/config&quot;
  9. &quot;github.com/aws/aws-sdk-go-v2/credentials&quot;
  10. &quot;github.com/aws/aws-sdk-go-v2/feature/s3/manager&quot;
  11. &quot;github.com/aws/aws-sdk-go-v2/service/s3&quot;
  12. )
  13. var BUCKET_NAME = &quot;&quot;
  14. func main() {
  15. //prepare gcp resolver
  16. gcpResolver := aws.EndpointResolverWithOptionsFunc(func(service, region string, options ...interface{}) (aws.Endpoint, error) {
  17. return aws.Endpoint{
  18. URL:               &quot;https://storage.googleapis.com&quot;,
  19. SigningRegion:     &quot;auto&quot;,
  20. Source:            aws.EndpointSourceCustom,
  21. HostnameImmutable: true,
  22. }, nil
  23. })
  24. //file with fornat : $accessKey:$secretKey
  25. file, _ := os.ReadFile(&quot;/home/bapt/creds/amz-keys-gcp-2&quot;)
  26. keys := strings.Split(string(file), &quot;:&quot;)
  27. //init the config options
  28. optConfig := []func(*config.LoadOptions) error{
  29. config.WithRegion(&quot;auto&quot;),
  30. config.WithCredentialsProvider(credentials.NewStaticCredentialsProvider(keys[0], strings.TrimRight(keys[1], &quot;\n&quot;), &quot;&quot;)),
  31. config.WithClientLogMode(aws.LogRetries | aws.LogRequestWithBody | aws.LogResponseWithBody | aws.LogRequestEventMessage | aws.LogResponseEventMessage | aws.LogSigning),
  32. config.WithEndpointResolverWithOptions(gcpResolver),
  33. }
  34. //init config
  35. cfg, _ := config.LoadDefaultConfig(context.TODO(), optConfig...)
  36. //init service
  37. svc := s3.NewFromConfig(cfg)
  38. tempFile, _ := os.CreateTemp(&quot;/tmp&quot;, &quot;test-gcp-*&quot;)
  39. defer tempFile.Close()
  40. downloader := manager.NewDownloader(svc)
  41. downloader.Download(context.TODO(),tempFile, &amp;s3.GetObjectInput{
  42. Bucket: aws.String(BUCKET_NAME),
  43. Key:    aws.String(&quot;file-test.txt&quot;),
  44. })
  45. //init uploader ( no multipart)
  46. uploader := manager.NewUploader(svc, func(u *manager.Uploader) {
  47. u.Concurrency = 1
  48. u.MaxUploadParts = 1
  49. })
  50. //upload
  51. _, err := uploader.Upload(context.TODO(), &amp;s3.PutObjectInput{
  52. Bucket: aws.String(BUCKET_NAME),
  53. Key:    aws.String(&quot;file-test.txt&quot;),
  54. Body:   strings.NewReader(&quot;HELLO&quot;),
  55. })
  56. fmt.Println(err)
  57. }

go.mod

  1. module gcps3/v2
  2. go 1.18
  3. require (
  4. github.com/aws/aws-sdk-go-v2 v1.16.14
  5. github.com/aws/aws-sdk-go-v2/config v1.17.5
  6. github.com/aws/aws-sdk-go-v2/credentials v1.12.18
  7. github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.11.31
  8. github.com/aws/aws-sdk-go-v2/service/s3 v1.27.9
  9. )
  10. require (
  11. github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.4.7 // indirect
  12. github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.12.15 // indirect
  13. github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.21 // indirect
  14. github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.4.15 // indirect
  15. github.com/aws/aws-sdk-go-v2/internal/ini v1.3.22 // indirect
  16. github.com/aws/aws-sdk-go-v2/internal/v4a v1.0.12 // indirect
  17. github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.9.8 // indirect
  18. github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.1.16 // indirect
  19. github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.9.15 // indirect
  20. github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.13.15 // indirect
  21. github.com/aws/aws-sdk-go-v2/service/sso v1.11.21 // indirect
  22. github.com/aws/aws-sdk-go-v2/service/ssooidc v1.13.3 // indirect
  23. github.com/aws/aws-sdk-go-v2/service/sts v1.16.17 // indirect
  24. github.com/aws/smithy-go v1.13.2 // indirect
  25. github.com/jmespath/go-jmespath v0.4.0 // indirect
  26. )

And here the debug trace of the PUT ( my bucket name is replaced by BUCKET, and access_key with GOOG1ID:

  1. SDK 2022/09/14 14:52:37 DEBUG Request Signature:
  2. ---[ CANONICAL STRING  ]-----------------------------
  3. PUT
  4. /BUCKET/file-test.txt
  5. x-id=PutObject
  6. accept-encoding:identity
  7. amz-sdk-invocation-id:d6776820-e336-4bdf-afa3-0ca3b6d5b0a0
  8. amz-sdk-request:attempt=1; max=3
  9. content-length:5
  10. content-type:application/octet-stream
  11. host:storage.googleapis.com
  12. x-amz-content-sha256:UNSIGNED-PAYLOAD
  13. x-amz-date:20220914T125237Z
  14. accept-encoding;amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-content-sha256;x-amz-date
  15. UNSIGNED-PAYLOAD
  16. ---[ STRING TO SIGN ]--------------------------------
  17. AWS4-HMAC-SHA256
  18. 20220914T125237Z
  19. 20220914/auto/s3/aws4_request
  20. bc09.....daf520
  21. -----------------------------------------------------
  22. SDK 2022/09/14 14:52:37 DEBUG Request
  23. PUT /BUCKET/file-test.txt?x-id=PutObject HTTP/1.1
  24. Host: storage.googleapis.com
  25. User-Agent: aws-sdk-go-v2/1.16.14 os/linux lang/go/1.18.1 md/GOOS/linux md/GOARCH/amd64 api/s3/1.27.9 ft/s3-transfer
  26. Content-Length: 5
  27. Accept-Encoding: identity
  28. Amz-Sdk-Invocation-Id: d6776820-e336-4bdf-afa3-0ca3b6d5b0a0
  29. Amz-Sdk-Request: attempt=1; max=3
  30. Authorization: AWS4-HMAC-SHA256 Credential=GOOG1ID/20220914/auto/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-content-sha256;x-amz-date, Signature=c994....37d0
  31. Content-Type: application/octet-stream
  32. X-Amz-Content-Sha256: UNSIGNED-PAYLOAD
  33. X-Amz-Date: 20220914T125237Z
  34. HELLO
  35. SDK 2022/09/14 14:52:37 DEBUG Response
  36. HTTP/2.0 403 Forbidden
  37. Content-Length: 883
  38. Alt-Svc: h3=&quot;:443&quot;; ma=2592000,h3-29=&quot;:443&quot;; ma=2592000,h3-Q050=&quot;:443&quot;; ma=2592000,h3-Q046=&quot;:443&quot;; ma=2592000,h3-Q043=&quot;:443&quot;; ma=2592000,quic=&quot;:443&quot;; ma=2592000; v=&quot;46,43&quot;
  39. Content-Type: application/xml; charset=UTF-8
  40. Date: Wed, 14 Sep 2022 12:52:38 GMT
  41. Server: UploadServer
  42. X-Guploader-Uploadid: ADPycdt0aCu6BTmzWQl2Ehc4q2sP8rtexDb4Keyn6cQL_GigREvc8T1CzX0HH-ZXgw_6XWLJPPYXufwRCr0Sl7uSsiIi0Q
  43. &lt;?xml version=&#39;1.0&#39; encoding=&#39;UTF-8&#39;?&gt;&lt;Error&gt;&lt;Code&gt;SignatureDoesNotMatch&lt;/Code&gt;&lt;Message&gt;The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method.&lt;/Message&gt;&lt;StringToSign&gt;AWS4-HMAC-SHA256
  44. 20220914T125237Z
  45. 20220914/auto/s3/aws4_request
  46. 0a10....d63e&lt;/StringToSign&gt;&lt;CanonicalRequest&gt;PUT
  47. /BUCKET/file-test.txt
  48. x-id=PutObject
  49. accept-encoding:identity,gzip(gfe)
  50. amz-sdk-invocation-id:d6776820-e336-4bdf-afa3-0ca3b6d5b0a0
  51. amz-sdk-request:attempt=1; max=3
  52. content-length:5
  53. content-type:application/octet-stream
  54. host:storage.googleapis.com
  55. x-amz-content-sha256:UNSIGNED-PAYLOAD
  56. x-amz-date:20220914T125237Z
  57. accept-encoding;amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-content-sha256;x-amz-date
  58. UNSIGNED-PAYLOAD&lt;/CanonicalRequest&gt;&lt;/Error&gt;
  59. SDK 2022/09/14 14:52:37 DEBUG request failed with unretryable error https response error StatusCode: 403, RequestID: , HostID: , api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method.
  60. operation error S3: PutObject, https response error StatusCode: 403, RequestID: , HostID: , api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method.

I tested the upload, with this python script, and it works (same creds)

  1. boto3.set_stream_logger(&#39;&#39;, logging.DEBUG)
  2. GCP_BUCKET = True
  3. FILE_TO_BE_UPLOADED = &#39;/tmp/toto&#39;
  4. if GCP_BUCKET:
  5. ACCESS_KEY = &quot;ACESS&quot;
  6. SECRET_KEY = &quot;SECRET&quot;
  7. bucket_name = &quot;BUCKET&quot;    
  8. region_name=&quot;auto&quot;
  9. endpoint_url=&quot;https://storage.googleapis.com&quot;
  10. def makeS3Client():
  11. s3 = boto3.client(&quot;s3&quot;, 
  12. region_name=region_name,
  13. endpoint_url=endpoint_url,
  14. aws_access_key_id=ACCESS_KEY,
  15. aws_secret_access_key=SECRET_KEY,
  16. )
  17. return s3
  18. def upload_file(s3,bucket_name,fname):
  19. &quot;&quot;&quot;
  20. Uploads file to S3 bucket using S3 client object
  21. :return: None
  22. &quot;&quot;&quot;
  23. object_name = os.path.basename(fname)
  24. file_name = os.path.abspath(fname)
  25. #file_name = os.path.join(pathlib.Path(__file__).parent.resolve(), fname)
  26. response = s3.upload_file(file_name, bucket_name, object_name)
  27. #print(response)  # prints None

The only different between go and python i saw in the https requests made is the signedHeaders used.

But the go code with an s3 aws bucket is working fine...

Am I missing an option ?

Thanks for your help.

答案1

得分: 3

@h3yduck提到的问题是，似乎需要在v2库的签名签名过程中排除accept-encoding。由于签名依赖于SignedHeaders中提到的标头，我们将不得不自己重新计算签名。

AWS配置对象公开了一个用于所有服务的自定义HTTP客户端的变量。我们可以通过定义自己的RoundTripper来使用它，该RoundTripper修改请求的签名，使其不考虑accept-encoding。

示例代码如下：

  1. package main
  3. import (
  4. 	"context"
  5. 	"fmt"
  6. 	"net/http"
  7. 	"net/http/httputil"
  8. 	"strings"
  9. 	"time"
  11. 	"github.com/aws/aws-sdk-go-v2/aws"
  12. 	v4 "github.com/aws/aws-sdk-go-v2/aws/signer/v4"
  13. 	"github.com/aws/aws-sdk-go-v2/config"
  14. 	"github.com/aws/aws-sdk-go-v2/credentials"
  15. 	"github.com/aws/aws-sdk-go-v2/feature/s3/manager"
  16. 	"github.com/aws/aws-sdk-go-v2/service/s3"
  17. )
  19. var BUCKET_NAME = "test"
  21. type RecalculateV4Signature struct {
  22. 	next   http.RoundTripper
  23. 	signer *v4.Signer
  24. 	cfg    aws.Config
  25. }
  27. func (lt *RecalculateV4Signature) RoundTrip(req *http.Request) (*http.Response, error) {
  28. 	// 存储以备后用
  29. 	val := req.Header.Get("Accept-Encoding")
  31. 	// 删除该标头，以便该标头不计入签名
  32. 	req.Header.Del("Accept-Encoding")
  34. 	// 使用相同的日期进行签名
  35. 	timeString := req.Header.Get("X-Amz-Date")
  36. 	timeDate, _ := time.Parse("20060102T150405Z", timeString)
  38. 	creds, _ := lt.cfg.Credentials.Retrieve(req.Context())
  39. 	err := lt.signer.SignHTTP(req.Context(), creds, req, v4.GetPayloadHash(req.Context()), "s3", lt.cfg.Region, timeDate)
  40. 	if err != nil {
  41. 		return nil, err
  42. 	}
  43. 	// 如果需要，重置Accept-Encoding
  44. 	req.Header.Set("Accept-Encoding", val)
  46. 	fmt.Println("AfterAdjustment")
  47. 	rrr, _ := httputil.DumpRequest(req, false)
  48. 	fmt.Println(string(rrr))
  50. 	// 继续原始的RoundTripper
  51. 	return lt.next.RoundTrip(req)
  52. }
  54. func main() {
  56. 	// 准备GCP解析器
  57. 	gcpResolver := aws.EndpointResolverWithOptionsFunc(func(service, region string, options ...interface{}) (aws.Endpoint, error) {
  58. 		return aws.Endpoint{
  59. 			URL:               "https://storage.googleapis.com",
  60. 			SigningRegion:     "auto",
  61. 			Source:            aws.EndpointSourceCustom,
  62. 			HostnameImmutable: true,
  63. 		}, nil
  64. 	})
  65. 	// 文件格式：$accessKey:$secretKey
  67. 	// 初始化配置选项
  68. 	optConfig := []func(*config.LoadOptions) error{
  69. 		config.WithRegion("auto"),
  70. 		config.WithCredentialsProvider(credentials.NewStaticCredentialsProvider("test", "test", "session")),
  71. 		config.WithClientLogMode(aws.LogRetries | aws.LogRequestWithBody | aws.LogResponseWithBody | aws.LogRequestEventMessage | aws.LogResponseEventMessage | aws.LogSigning),
  72. 		config.WithEndpointResolverWithOptions(gcpResolver),
  73. 	}
  75. 	// 初始化配置
  76. 	cfg, _ := config.LoadDefaultConfig(context.TODO(), optConfig...)
  77. 	// 使用我们自己的传输方式分配自定义客户端
  78. 	cfg.HTTPClient = &http.Client{Transport: &RecalculateV4Signature{http.DefaultTransport, v4.NewSigner(), cfg}}
  79. 	// 初始化服务
  80. 	svc := s3.NewFromConfig(cfg)
  81. 	uploader := manager.NewUploader(svc, func(u *manager.Uploader) {
  82. 		u.Concurrency = 1
  83. 		u.MaxUploadParts = 1
  84. 	})
  85. 	// 上传
  86. 	_, err := uploader.Upload(context.TODO(), &s3.PutObjectInput{
  87. 		Bucket: aws.String(BUCKET_NAME),
  88. 		Key:    aws.String("file-test.txt"),
  89. 		Body:   strings.NewReader("HELLO"),
  90. 	})
  91. 	fmt.Println(err)
  92. }

附注：客户端最好基于aws的BuildableClient构建，因为它们为其服务提供了合理的默认值，我将把这个问题留给读者。

英文:

The issue that @h3yduck mentioned, it seems that accept-encoding needs to be excluded from the signature signing process in the v2 library. Since the signature is dependent on the headers mentioned in SignedHeaders, we will have to recalculate the signature ourselves.

The AWS Configuration object exposes a variable for a custom http client to be used for all services. We can use this by defining our own RoundTripper that modifies the request's signature to one that doesn't account for accept-encoding.

Example:

  1. package main
  3. import (
  4. 	&quot;context&quot;
  5. 	&quot;fmt&quot;
  6. 	&quot;net/http&quot;
  7. 	&quot;net/http/httputil&quot;
  8. 	&quot;strings&quot;
  9. 	&quot;time&quot;
  11. 	&quot;github.com/aws/aws-sdk-go-v2/aws&quot;
  12. 	v4 &quot;github.com/aws/aws-sdk-go-v2/aws/signer/v4&quot;
  13. 	&quot;github.com/aws/aws-sdk-go-v2/config&quot;
  14. 	&quot;github.com/aws/aws-sdk-go-v2/credentials&quot;
  15. 	&quot;github.com/aws/aws-sdk-go-v2/feature/s3/manager&quot;
  16. 	&quot;github.com/aws/aws-sdk-go-v2/service/s3&quot;
  17. )
  19. var BUCKET_NAME = &quot;test&quot;
  21. type RecalculateV4Signature struct {
  22. 	next   http.RoundTripper
  23. 	signer *v4.Signer
  24. 	cfg    aws.Config
  25. }
  27. func (lt *RecalculateV4Signature) RoundTrip(req *http.Request) (*http.Response, error) {
  28. 	// store for later use
  29. 	val := req.Header.Get(&quot;Accept-Encoding&quot;)
  31. 	// delete the header so the header doesn&#39;t account for in the signature
  32. 	req.Header.Del(&quot;Accept-Encoding&quot;)
  34. 	// sign with the same date
  35. 	timeString := req.Header.Get(&quot;X-Amz-Date&quot;)
  36. 	timeDate, _ := time.Parse(&quot;20060102T150405Z&quot;, timeString)
  38. 	creds, _ := lt.cfg.Credentials.Retrieve(req.Context())
  39. 	err := lt.signer.SignHTTP(req.Context(), creds, req, v4.GetPayloadHash(req.Context()), &quot;s3&quot;, lt.cfg.Region, timeDate)
  40. 	if err != nil {
  41. 		return nil, err
  42. 	}
  43. 	// Reset Accept-Encoding if desired
  44. 	req.Header.Set(&quot;Accept-Encoding&quot;, val)
  46. 	fmt.Println(&quot;AfterAdjustment&quot;)
  47. 	rrr, _ := httputil.DumpRequest(req, false)
  48. 	fmt.Println(string(rrr))
  50. 	// follows up the original round tripper
  51. 	return lt.next.RoundTrip(req)
  52. }
  54. func main() {
  56. 	//prepare gcp resolver
  57. 	gcpResolver := aws.EndpointResolverWithOptionsFunc(func(service, region string, options ...interface{}) (aws.Endpoint, error) {
  58. 		return aws.Endpoint{
  59. 			URL:               &quot;https://storage.googleapis.com&quot;,
  60. 			SigningRegion:     &quot;auto&quot;,
  61. 			Source:            aws.EndpointSourceCustom,
  62. 			HostnameImmutable: true,
  63. 		}, nil
  64. 	})
  65. 	//file with format : $accessKey:$secretKey
  67. 	//init the config options
  68. 	optConfig := []func(*config.LoadOptions) error{
  69. 		config.WithRegion(&quot;auto&quot;),
  70. 		config.WithCredentialsProvider(credentials.NewStaticCredentialsProvider(&quot;test&quot;, &quot;test&quot;, &quot;session&quot;)),
  71. 		config.WithClientLogMode(aws.LogRetries | aws.LogRequestWithBody | aws.LogResponseWithBody | aws.LogRequestEventMessage | aws.LogResponseEventMessage | aws.LogSigning),
  72. 		config.WithEndpointResolverWithOptions(gcpResolver),
  73. 	}
  75. 	//init config
  76. 	cfg, _ := config.LoadDefaultConfig(context.TODO(), optConfig...)
  77. 	// Assign custom client with our own transport
  78. 	cfg.HTTPClient = &amp;http.Client{Transport: &amp;RecalculateV4Signature{http.DefaultTransport, v4.NewSigner(), cfg}}
  79. 	//init service
  80. 	svc := s3.NewFromConfig(cfg)
  81. 	uploader := manager.NewUploader(svc, func(u *manager.Uploader) {
  82. 		u.Concurrency = 1
  83. 		u.MaxUploadParts = 1
  84. 	})
  85. 	//upload
  86. 	_, err := uploader.Upload(context.TODO(), &amp;s3.PutObjectInput{
  87. 		Bucket: aws.String(BUCKET_NAME),
  88. 		Key:    aws.String(&quot;file-test.txt&quot;),
  89. 		Body:   strings.NewReader(&quot;HELLO&quot;),
  90. 	})
  91. 	fmt.Println(err)
  92. }

P.S. the client should ideally be built on BuildableClient from aws as they provide sensible defaults for their services, I will leave that up to the reader

答案2

得分: 1

根据https://github.com/aws/aws-sdk-go-v2/issues/1816的说法，v2 SDK不支持GCP GCS。

对我来说，从签名标头中删除Accept-Encoding解决了这个问题，但我不确定这是否是一个建议的做法。此外，我没有找到一种方法来从签名标头列表中删除此标头（可能有一个支持此操作的中间件），除非修改库的源代码（将其添加到IgnoredHeaders全局变量中）。

英文:

According to https://github.com/aws/aws-sdk-go-v2/issues/1816 the v2 SDK doesn't support GCP GCS.

Removing Accept-Encoding from the sighed headers solved the issue for me, but I'm not sure if it's an advised thing to do. Also, I didn't find a way to remove this header from the list of signed headers (there might be a middleware that supports this), other than modifying the lib's source code (adding it to the IgnoredHeaders global variable).

huangapple
  • 本文由 发表于 2022年9月14日 21:11:38
  • 转载请务必保留本文链接：https://go.coder-hub.com/73717477.html
  • amazon-s3
  • go
  • google-cloud-platform
  • google-cloud-storage
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定