使用s3manager上传文件

huangapple
go评论119阅读模式
英文:

Uploading a file with s3manager

问题

我正在尝试从我的网站下载一些文件并将它们上传到Amazon S3。

以下是我的代码：

  1. import (
  2. 	"log"
  3. 	"net/http"
  5. 	"github.com/aws/aws-sdk-go/aws"
  6. 	"github.com/aws/aws-sdk-go/aws/credentials"
  7. 	"github.com/aws/aws-sdk-go/aws/session"
  8. 	"github.com/aws/aws-sdk-go/service/s3/s3manager"
  9. )
  11. func main() {
  12. 	// 发送GET请求以下载并上传到s3的图像
  13. 	resp, err := http.Get("http://my-site.com/image.jpg")
  14. 	if err != nil {
  15. 		log.Fatal(err)
  16. 	}
  18. 	// 获取AWS凭证
  19. 	creds := credentials.NewSharedCredentials("/Users/name/.aws/credentials", "default")
  21. 	// 设置配置
  22. 	config := &aws.Config{
  23. 		Region:      aws.String("US Standard"),
  24. 		Credentials: creds,
  25. 	}
  27. 	// 创建新的AWS会话
  28. 	sess := session.New(config)
  30. 	// 创建新的s3manager客户端
  31. 	uploader := s3manager.NewUploader(sess)
  33. 	result, err := uploader.Upload(&s3manager.UploadInput{
  34. 		Body:   resp.Body,
  35. 		Bucket: aws.String("mimi-movies"),
  36. 		Key:    aws.String("mimi.jpg"),
  37. 	})
  38. 	if err != nil {
  39. 		log.Fatalln("上传失败", err)
  40. 	}
  42. 	log.Println("成功上传至", result.Location)
  43. }

我一直收到一个运行时错误，指向空指针解引用。错误信息并没有提供太多帮助，但我认为可能与SSL有关。以下是完整的错误信息：

  1. panic: runtime error: invalid memory address or nil pointer dereference
  2. [signal 0xb code=0x1 addr=0x0 pc=0x236ffe]
  4. goroutine 1 [running]:
  5. panic(0x4743c0, 0xc82000a0e0)
  6. 	/usr/local/Cellar/go/1.6/libexec/src/runtime/panic.go:464 +0x3e6
  7. github.com/aws/aws-sdk-go/service/s3.validateSSERequiresSSL(0xc820066380)
  8. 	/Users/name/work/src/github.com/aws/aws-sdk-go/service/s3/sse.go:15 +0x17e
  9. github.com/aws/aws-sdk-go/aws/request.(*HandlerList).Run(0xc820066478, 0xc820066380)
  10. 	/Users/name/work/src/github.com/aws/aws-sdk-go/aws/request/handlers.go:136 +0xc3
  11. github.com/aws/aws-sdk-go/aws/request.(*Request).Build(0xc820066380, 0x0, 0x0)
  12. 	/Users/name/work/src/github.com/aws/aws-sdk-go/aws/request/request.go:189 +0x69
  13. github.com/aws/aws-sdk-go/aws/request.(*Request).Sign(0xc820066380, 0x0, 0x0)
  14. 	/Users/name/work/src/github.com/aws/aws-sdk-go/aws/request/request.go:210 +0x31
  15. github.com/aws/aws-sdk-go/aws/request.(*Request).Send(0xc820066380, 0x0, 0x0)
  16. 	/Users/name/work/src/github.com/aws/aws-sdk-go/aws/request/request.go:261 +0x6f4
  17. github.com/aws/aws-sdk-go/service/s3/s3manager.(*uploader).singlePart(0xc8201520f0, 0x8bb980, 0xc82014a030, 0xc82000a140, 0x0, 0x0)
  18. 	/Users/name/work/src/github.com/aws/aws-sdk-go/service/s3/s3manager/upload.go:449 +0x1fc
  19. github.com/aws/aws-sdk-go/service/s3/s3manager.(*uploader).upload(0xc8201520f0, 0xc8201520f0, 0x0, 0x0)
  20. 	/Users/name/work/src/github.com/aws/aws-sdk-go/service/s3/s3manager/upload.go:355 +0x563
  21. github.com/aws/aws-sdk-go/service/s3/s3manager.Uploader.Upload(0x500000, 0x5, 0x0, 0x2710, 0xe00240, 0xc820156018, 0xc82016e000, 0x0, 0x0, 0x0, ...)
  22. 	/Users/name/work/src/github.com/aws/aws-sdk-go/service/s3/s3manager/upload.go:329 +0x15a
  23. main.main()
  24. 	/Users/name/work/src/gitlab.com/user/project/test/main.go:39 +0x59f
  25. exit status 2

希望这可以帮助你解决问题。

英文:

I'm trying to download a few files from my website and upload them to amazons3.

Here is my code:

  1. import (
  2. 	"log"
  3. 	"net/http"
  5. 	"github.com/aws/aws-sdk-go/aws"
  6. 	"github.com/aws/aws-sdk-go/aws/credentials"
  7. 	"github.com/aws/aws-sdk-go/aws/session"
  8. 	"github.com/aws/aws-sdk-go/service/s3/s3manager"
  9. )
  11. func main() {
  12. 	// Make a GET request to an image to be downloaded and uploaded to s3
  13. 	resp, err := http.Get("http://my-site.com/image.jpg")
  14. 	if err != nil {
  15. 		log.Fatal(err)
  16. 	}
  18. 	// Get the AWS credentials
  19. 	creds := credentials.NewSharedCredentials("/Users/name/.aws/credentials", "default")
  21. 	// Set up config
  22. 	config := &aws.Config{
  23. 		Region:      aws.String("US Standard"),
  24. 		Credentials: creds,
  25. 	}
  27. 	// Set up a new AWS session
  28. 	sess := session.New(config)
  30. 	// Set up a new s3manager client
  31. 	uploader := s3manager.NewUploader(sess)
  33. 	result, err := uploader.Upload(&s3manager.UploadInput{
  34. 		Body:   resp.Body,
  35. 		Bucket: aws.String("mimi-movies"),
  36. 		Key:    aws.String("mimi.jpg"),
  37. 	})
  38. 	if err != nil {
  39. 		log.Fatalln("Failed to upload", err)
  40. 	}
  42. 	log.Println("Successfully uploaded to", result.Location)
  43. }

I keep getting a run time error of a nil pointer dereference the error doesn't help much but for some reason I think it might have to do with ssl but here is the entire error:

  1. panic: runtime error: invalid memory address or nil pointer dereference
  2. [signal 0xb code=0x1 addr=0x0 pc=0x236ffe]
  4. goroutine 1 [running]:
  5. panic(0x4743c0, 0xc82000a0e0)
  6. 	/usr/local/Cellar/go/1.6/libexec/src/runtime/panic.go:464 +0x3e6
  7. github.com/aws/aws-sdk-go/service/s3.validateSSERequiresSSL(0xc820066380)
  8. 	/Users/name/work/src/github.com/aws/aws-sdk-go/service/s3/sse.go:15 +0x17e
  9. github.com/aws/aws-sdk-go/aws/request.(*HandlerList).Run(0xc820066478, 0xc820066380)
  10. 	/Users/name/work/src/github.com/aws/aws-sdk-go/aws/request/handlers.go:136 +0xc3
  11. github.com/aws/aws-sdk-go/aws/request.(*Request).Build(0xc820066380, 0x0, 0x0)
  12. 	/Users/name/work/src/github.com/aws/aws-sdk-go/aws/request/request.go:189 +0x69
  13. github.com/aws/aws-sdk-go/aws/request.(*Request).Sign(0xc820066380, 0x0, 0x0)
  14. 	/Users/name/work/src/github.com/aws/aws-sdk-go/aws/request/request.go:210 +0x31
  15. github.com/aws/aws-sdk-go/aws/request.(*Request).Send(0xc820066380, 0x0, 0x0)
  16. 	/Users/name/work/src/github.com/aws/aws-sdk-go/aws/request/request.go:261 +0x6f4
  17. github.com/aws/aws-sdk-go/service/s3/s3manager.(*uploader).singlePart(0xc8201520f0, 0x8bb980, 0xc82014a030, 0xc82000a140, 0x0, 0x0)
  18. 	/Users/name/work/src/github.com/aws/aws-sdk-go/service/s3/s3manager/upload.go:449 +0x1fc
  19. github.com/aws/aws-sdk-go/service/s3/s3manager.(*uploader).upload(0xc8201520f0, 0xc8201520f0, 0x0, 0x0)
  20. 	/Users/name/work/src/github.com/aws/aws-sdk-go/service/s3/s3manager/upload.go:355 +0x563
  21. github.com/aws/aws-sdk-go/service/s3/s3manager.Uploader.Upload(0x500000, 0x5, 0x0, 0x2710, 0xe00240, 0xc820156018, 0xc82016e000, 0x0, 0x0, 0x0, ...)
  22. 	/Users/name/work/src/github.com/aws/aws-sdk-go/service/s3/s3manager/upload.go:329 +0x15a
  23. main.main()
  24. 	/Users/name/work/src/gitlab.com/user/project/test/main.go:39 +0x59f
  25. exit status 2

答案1

得分: 4

如@user3666882所提到的，aws-go-sdk中存在一个错误，你应该检查一下你的区域。

在你的代码中，你已经编写了获取AWS凭证的代码。

  1. // 获取AWS凭证
  2. creds := credentials.NewSharedCredentials("/Users/name/.aws/credentials", "default")

你可以尝试在终端中执行以下命令：

aws configure

并配置你的aws-sdk。

否则，你可以使用Minio。但是你必须将你的ACCESS_KEY_ID和SECRET_ACCESS_KEY放入你的代码中，这对于生产环境来说是非常不好的。

  1. s3Client, err := minio.New("s3.amazonaws.com", "YOUR-ACCESSKEYID", "YOUR-SECRETACCESSKEY", false)

另外，你可以使用外部API Viper 来隐藏这些密钥。但我建议你使用官方的AWS SDK for Go。

英文:

As @user3666882 mentioned there's bug in aws-go-sdk and you should probably check your region.

In your code, you have written code to get AWS Credential.

  1. // Get the AWS credentials
  2.     creds := credentials.NewSharedCredentials("/Users/name/.aws/credentials", "default")

You might try this in your terminal :

> aws configure

And configure your aws-sdk.

Else you can use Minio. But your have to put your ACCESS_KEY_ID and SECRET_ACCESS_KEY in your code, which is very bad for production use.

  1.  s3Client, err := minio.New("s3.amazonaws.com", "YOUR-ACCESSKEYID", "YOUR-SECRETACCESSKEY", false)

Also, you can use external API Viper to hide these keys. But I recommend you to use official AWS SDK for Go.

答案2

得分: 2

有一个 aws-go-sdk 的 bug，但解决方法是将区域更改为 "us-east-1"

// 设置配置
config := &aws.Config{
Region: aws.String("us-east-1"),
Credentials: creds,
}

英文:

There is a bug on the aws-go-sdk but the solution is to change the region to "us-east-1"

  1. // Set up config
  2. config := &aws.Config{
  3.     Region:      aws.String("us-east-1"),
  4.     Credentials: creds,
  5. }

答案3

得分: 0

你可以选择使用Minio-go客户端库，它是开源的，并且与AWS S3兼容。

这里是一个简单的示例来上传对象

  1. package main
  3. import (
  4. 	"log"
  5. 	"os"
  7. 	"github.com/minio/minio-go"
  8. )
  10. func main() {
  11. 	// 注意：YOUR-ACCESSKEYID、YOUR-SECRETACCESSKEY、my-testfile、my-bucketname和my-objectname是虚拟值，请将它们替换为实际值。
  13. 	// 默认情况下，请求始终是安全的（HTTPS）。设置insecure=true以启用不安全的（HTTP）访问。
  14. 	// 这个布尔值是New()的最后一个参数。
  16. 	// New返回一个兼容Amazon S3的客户端对象。API兼容性（v2或v4）会根据Endpoint值自动确定。
  17. 	s3Client, err := minio.New("s3.amazonaws.com", "YOUR-ACCESSKEYID", "YOUR-SECRETACCESSKEY", false)
  18. 	if err != nil {
  19. 		log.Fatalln(err)
  20. 	}
  22. 	object, err := os.Open("my-testfile")
  23. 	if err != nil {
  24. 		log.Fatalln(err)
  25. 	}
  26. 	defer object.Close()
  28. 	n, err := s3Client.PutObject("my-bucketname", "my-objectname", object, "application/octet-stream")
  29. 	if err != nil {
  30. 		log.Fatalln(err)
  31. 	}
  32. 	log.Println("已上传", "my-objectname", "大小为:", n, "成功.")
  34. }

希望对你有帮助。

免责声明：我为Minio工作。

英文:

You could have alternatively used Minio-go client libraries, its Open Source & compatible with AWS S3.

Here is a simple example to Put an object

<pre>
package main

import (
"log"
"os"

  1. &quot;github.com/minio/minio-go&quot;

)

func main() {
// Note: YOUR-ACCESSKEYID, YOUR-SECRETACCESSKEY, my-testfile, my-bucketname and
// my-objectname are dummy values, please replace them with original values.

  1. // Requests are always secure (HTTPS) by default. Set insecure=true to enable insecure (HTTP) access.
  2. // This boolean value is the last argument for New().
  4. // New returns an Amazon S3 compatible client object. API compatibility (v2 or v4) is automatically
  5. // determined based on the Endpoint value.
  6. s3Client, err := minio.New(&quot;s3.amazonaws.com&quot;, &quot;YOUR-ACCESSKEYID&quot;, &quot;YOUR-SECRETACCESSKEY&quot;, false)
  7. if err != nil {
  8. 	log.Fatalln(err)
  9. }
  11. object, err := os.Open(&quot;my-testfile&quot;)
  12. if err != nil {
  13. 	log.Fatalln(err)
  14. }
  15. defer object.Close()
  17. n, err := s3Client.PutObject(&quot;my-bucketname&quot;, &quot;my-objectname&quot;, object, &quot;application/octet-stream&quot;)
  18. if err != nil {
  19. 	log.Fatalln(err)
  20. }
  21. log.Println(&quot;Uploaded&quot;, &quot;my-objectname&quot;, &quot; of size: &quot;, n, &quot;Successfully.&quot;)

}
Status API Training Shop Blog About

</pre>

Hope it helps.

Disclaimer: I work for Minio

huangapple
  • 本文由 发表于 2016年3月26日 01:04:58
  • 转载请务必保留本文链接：https://go.coder-hub.com/36224477.html
  • amazon-s3
  • amazon-web-services
  • go
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定