使用aws-sdk-go将文件保存到S3

huangapple go评论89阅读模式
英文:

Saving file to S3 using aws-sdk-go

问题

我在使用AWS S3 Go SDK(https://github.com/awslabs/aws-sdk-go)保存文件时遇到了一些问题。

这是我的代码:

import (
    "fmt"
    "bytes"

    "github.com/awslabs/aws-sdk-go/aws"
    "github.com/awslabs/aws-sdk-go/aws/awsutil"
    "github.com/awslabs/aws-sdk-go/service/s3"
)

func main() {    
    cred := aws.DefaultChainCredentials
    cred.Get() // 我正在使用环境变量凭证,并且已经检查了它们是否存在
    svc := s3.New(&aws.Config{Region: "us-west-2", Credentials:cred, LogLevel: 1})
    params := &s3.PutObjectInput{
        Bucket: aws.String("my-bucket-123"),
        Key:    aws.String("test/123/"),
        Body:   bytes.NewReader([]byte("testing!")),
    }
    resp, err := svc.PutObject(params)
    fmt.Printf("response %s", awsutil.StringValue(resp))
}

我一直收到301 Moved Permanently的响应。

编辑:我手动创建了存储桶。
编辑#2:示例响应:

---[ RESPONSE ]--------------------------------------
HTTP/1.1 301 Moved Permanently
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Tue, 05 May 2015 18:42:03 GMT
Server: AmazonS3

POST sign is http as well.

请注意,这只是代码的翻译部分,不包括问题的回答。

英文:

I'm having a bit of trouble saving a file in golang with the AWS S3 go sdk (https://github.com/awslabs/aws-sdk-go).

This is what I have:

import (
        "fmt"
        "bytes"

        "github.com/awslabs/aws-sdk-go/aws"
        "github.com/awslabs/aws-sdk-go/aws/awsutil"
        "github.com/awslabs/aws-sdk-go/service/s3"
)

func main() {    
    cred := aws.DefaultChainCredentials
    cred.Get() // i'm using environment variable credentials and yes, I checked if they were in here
    svc := s3.New(&aws.Config{Region: "us-west-2", Credentials:cred, LogLevel: 1})
    params := &s3.PutObjectInput{
                    Bucket:         aws.String("my-bucket-123"),
                    Key:            aws.String("test/123/"),
                    Body:         bytes.NewReader([]byte("testing!")),
            }
    resp, err := svc.PutObject(params)
    fmt.Printf("response %s", awsutil.StringValue(resp))
}

I keep receiving a 301 Moved Permanently response.

Edit: I created the bucket manually.
Edit #2: Example response:

---[ RESPONSE ]--------------------------------------
HTTP/1.1 301 Moved Permanently
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Tue, 05 May 2015 18:42:03 GMT
Server: AmazonS3

POST sign is http as well.

答案1

得分: 3

根据Amazon的说明:

Amazon S3在所有区域都支持虚拟主机样式(virtual-hosted-style)和路径样式(path-style)访问。然而,路径样式语法要求在尝试访问存储桶时使用特定于区域的终端节点。例如,如果你有一个名为mybucket的存储桶位于欧洲(EU)区域,你想使用路径样式语法,并且对象名为puppy.jpg,正确的URI是http://s3-eu-west-1.amazonaws.com/mybucket/puppy.jpg。如果你尝试使用路径样式语法访问位于美国标准(US Standard)区域之外的存储桶,使用以下任一方式的URI:

  • http://s3.amazonaws.com
  • 与存储桶所在区域不同的区域终端节点,例如,对于在美国西部(北加利福尼亚)区域创建的存储桶,使用http://s3-eu-west-1.amazonaws.com

我认为问题在于你正在尝试访问错误区域的存储桶。你的请求正在发送到这里:

https://my-bucket-123.s3-us-west-2.amazonaws.com/test/123

所以请确保my-bucket-123实际上位于us-west-2区域。(我使用自己的存储桶尝试过,可以正常工作)

我还验证了它是否使用了HTTPS,通过包装调用来进行日志记录(他们的日志消息只是错误的):

type LogReadCloser struct {
    io.ReadCloser
}

func (lr *LogReadCloser) Read(p []byte) (int, error) {
    n, err := lr.ReadCloser.Read(p)
    log.Println(string(p))
    return n, err
}

type LogRoundTripper struct {
    http.RoundTripper
}

func (lrt *LogRoundTripper) RoundTrip(req *http.Request) (*http.Response, error) {
    log.Println("REQUEST", req)
    res, err := lrt.RoundTripper.RoundTrip(req)
    log.Println("RESPONSE", res, err)
    res.Body = &LogReadCloser{res.Body}
    return res, err
}

然后:

svc := s3.New(&aws.Config{
    Region:      "us-west-2",
    Credentials: cred,
    LogLevel:    0,
    HTTPClient:  &http.Client{Transport: &LogRoundTripper{http.DefaultTransport}},
})
英文:

According to Amazon:

> Amazon S3 supports virtual-hosted-style and path-style access in all regions. The path-style syntax, however, requires that you use the region-specific endpoint when attempting to access a bucket. For example, if you have a bucket called mybucket that resides in the EU, you want to use path-style syntax, and the object is named puppy.jpg, the correct URI is http://s3-eu-west-1.amazonaws.com/mybucket/puppy.jpg. You will receive a "PermanentRedirect" error, an HTTP response code 301, and a message indicating what the correct URI is for your resource if you try to access a bucket outside the US Standard region with path-style syntax that uses either of the following:
>
> * http://s3.amazonaws.com
>
> * An endpoint for a region different from the one where the bucket resides, for example, http://s3-eu-west-1.amazonaws.com for a bucket that was created in the US West (Northern California) region

I think the problem is that you are trying to access a bucket in the wrong region. Your request is going here:

https://my-bucket-123.s3-us-west-2.amazonaws.com/test/123

So make sure that my-bucket-123 is actually in us-west-2. (I tried this with my own bucket and it worked fine)

I also verified that it's using HTTPS by wrapping the calls: (their log message is just wrong)

type LogReadCloser struct {
    io.ReadCloser
}

func (lr *LogReadCloser) Read(p []byte) (int, error) {
    n, err := lr.ReadCloser.Read(p)
    log.Println(string(p))
    return n, err
}

type LogRoundTripper struct {
    http.RoundTripper
}

func (lrt *LogRoundTripper) RoundTrip(req *http.Request) (*http.Response, error) {
    log.Println("REQUEST", req)
    res, err := lrt.RoundTripper.RoundTrip(req)
    log.Println("RESPONSE", res, err)
    res.Body = &LogReadCloser{res.Body}
    return res, err
}

And then:

svc := s3.New(&aws.Config{
    Region:      "us-west-2",
    Credentials: cred,
    LogLevel:    0,
    HTTPClient:  &http.Client{Transport: &LogRoundTripper{http.DefaultTransport}},
})

答案2

得分: 1

我认为你最好使用S3 Uploader。以下是我的代码示例,它是一个Web应用程序,我使用gin框架,在我的情况下,我从Web表单中获取文件,将其上传到S3,并检索URL以在其他HTML页面中呈现页面:

// 创建一个S3 Uploader
uploader := s3manager.NewUploader(sess)

// 上传
result, err := uploader.Upload(&s3manager.UploadInput{
    Bucket: aws.String(bucket),
    Key:    aws.String(fileHeader.Filename),
    Body:   f,
})
if err != nil {
    c.HTML(http.StatusBadRequest, "create-project.html", gin.H{
        "ErrorTitle":   "S3上传失败",
        "ErrorMessage": err.Error(),
    })
} else {
    // 上传成功,将URL打印到控制台。
    // "result.Location是图像的URL"
    fmt.Println("成功上传至", result.Location)
}

你可以在这里找到一个完整的示例,逐步解释:
https://www.matscloud.com/docs/cloud-sdk/go-and-s3/

英文:

I think you're better off using the S3 Uploader. Here's an example from my code, it's a web app, I use gin framework, and in my case I get a file from a web form, upload it to s3, and retrieve the URL to present a page in other HTMLs:

// Create an S3 Uploader
  uploader := s3manager.NewUploader(sess)

  // Upload 
  result, err := uploader.Upload(&s3manager.UploadInput{
   Bucket: aws.String(bucket),
   Key: aws.String(fileHeader.Filename),
   Body: f,
  })
  if err != nil {
   c.HTML(http.StatusBadRequest, "create-project.html", gin.H{
    "ErrorTitle":   "S3 Upload Failed",
    "ErrorMessage": err.Error()})
  } else {
   // Success, print URL to Console.
   ///"result.Location is the URL of an Image"
   fmt.Println("Successfully uploaded to", result.Location)
  }

You can find an entire example here, explained step by step:
https://www.matscloud.com/docs/cloud-sdk/go-and-s3/

huangapple
  • 本文由 发表于 2015年5月6日 01:41:01
  • 转载请务必保留本文链接:https://go.coder-hub.com/30059723.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定