AWS S3大文件反向代理使用golang的http.ResponseWriter

huangapple go评论85阅读模式
英文:

AWS S3 large file reverse proxying with golang's http.ResponseWriter

问题

我有一个名为Download的请求处理程序,我想从Amazon S3访问一个大文件,并将其推送到用户的浏览器。我的目标是:

  • 在向用户授予文件访问权限之前记录一些请求信息
  • 不要将文件缓冲到内存中太多。文件可能会变得太大。

以下是我目前探索的内容:

func Download(w http.ResponseWriter, r *http.Request) {

    sess := session.New(&aws.Config{
        Region:            aws.String("eu-west-1"),
        Endpoint:          aws.String("s3-eu-west-1.amazonaws.com"),
        S3ForcePathStyle:  aws.Bool(true),
        Credentials:       cred,
    })

    downloader := s3manager.NewDownloader(sess)
    // 我不能直接写入ResponseWriter。它没有实现WriteAt方法。
    // 而且,这似乎不是正确的做法。
    _, err := downloader.Download(w, &s3.GetObjectInput{
        Bucket: aws.String(BUCKET),
        Key:    aws.String(filename),
    })
    if err != nil {
        log.Error(4, err.Error())
        return
    }

}

我想知道是否有更好的方法(考虑到我要实现的目标)。

欢迎提出任何建议。提前谢谢你 AWS S3大文件反向代理使用golang的http.ResponseWriter

英文:

I have a request handler named Download which I want to access a large file from Amazon S3 and push it to the user's browser. My goals are:

  • To record some request information before granting the user access to the file
  • To not buffer the file into memory too much. Files may become too large.

Here is what I've explored so far:

func Download(w http.ResponseWriter, r *http.Request) {

    sess := session.New(&aws.Config{
	    Region: 			aws.String("eu-west-1"),
	    Endpoint: 			aws.String("s3-eu-west-1.amazonaws.com"),
	    S3ForcePathStyle: 	aws.Bool(true),
	    Credentials: 		cred,
    })

    downloader := s3manager.NewDownloader(sess)
    // I can't write directly into the ResponseWriter. It doesn't implement WriteAt. 
    // Besides, it doesn't seem like the right thing to do.
    _, err := downloader.Download(w, &s3.GetObjectInput{
	    Bucket: aws.String(BUCKET),
	    Key: aws.String(filename),
    })
    if err != nil {
	    log.Error(4, err.Error())
	    return
    }

}

I'm wondering if there isn't a better approach (given the goals I'm trying to achieve).

Any suggestions are welcome. Thank you in advance AWS S3大文件反向代理使用golang的http.ResponseWriter

答案1

得分: 13

如果您确实希望通过您的服务流式传输文件(而不是直接下载,如接受的答案中建议的那样),可以使用以下代码:

import (
	...

	"github.com/aws/aws-sdk-go/aws"
	"github.com/aws/aws-sdk-go/service/s3"
)

func StreamDownloadHandler(w http.ResponseWriter, r *http.Request) {

	sess, awsSessErr := session.NewSession(&aws.Config{
		Region:      aws.String("eu-west-1"),
		Credentials: credentials.NewStaticCredentials("my-aws-id", "my-aws-secret", ""),
	})
	if awsSessErr != nil {
		http.Error(w, fmt.Sprintf("创建 AWS 会话时出错:%s", awsSessErr.Error()), http.StatusInternalServerError)
		return
	}

	result, err := s3.New(sess).GetObject(&s3.GetObjectInput{
		Bucket: aws.String("my-bucket"),
		Key:    aws.String("my-file-id"),
	})
	if err != nil {
		http.Error(w, fmt.Sprintf("从 S3 获取文件时出错:%s", err.Error()), http.StatusInternalServerError)
		return
	}

	w.Header().Set("Content-Disposition", fmt.Sprintf("attachment; filename=\"%s\"", "my-file.csv"))
	w.Header().Set("Cache-Control", "no-store")

	bytesWritten, copyErr := io.Copy(w, result.Body)
	if copyErr != nil {
		http.Error(w, fmt.Sprintf("将文件复制到 HTTP 响应时出错:%s", copyErr.Error()), http.StatusInternalServerError)
		return
	}
	log.Printf("下载完成:“%s”。已写入 %s 字节", "my-file.csv", strconv.FormatInt(bytesWritten, 10))
}

以上是用于通过服务流式传输文件的代码示例。

英文:

If you do want to stream the file through your service (rather than download directly as recommended in the accepted answer) -

import (
	...

	"github.com/aws/aws-sdk-go/aws"
	"github.com/aws/aws-sdk-go/service/s3"
)

func StreamDownloadHandler(w http.ResponseWriter, r *http.Request) {

	sess, awsSessErr := session.NewSession(&aws.Config{
		Region:      aws.String("eu-west-1"),
		Credentials: credentials.NewStaticCredentials("my-aws-id", "my-aws-secret", ""),
	})
	if awsSessErr != nil {
		http.Error(w, fmt.Sprintf("Error creating aws session %s", awsSessErr.Error()), http.StatusInternalServerError)
		return
	}

	result, err := s3.New(sess).GetObject(&s3.GetObjectInput{
		Bucket: aws.String("my-bucket"),
		Key:    aws.String("my-file-id"),
	})
	if err != nil {
		http.Error(w, fmt.Sprintf("Error getting file from s3 %s", err.Error()), http.StatusInternalServerError)
		return
	}

	w.Header().Set("Content-Disposition", fmt.Sprintf("attachment; filename=\"%s\"", "my-file.csv"))
	w.Header().Set("Cache-Control", "no-store")

	bytesWritten, copyErr := io.Copy(w, result.Body)
	if copyErr != nil {
		http.Error(w, fmt.Sprintf("Error copying file to the http response %s", copyErr.Error()), http.StatusInternalServerError)
		return
	}
	log.Printf("Download of \"%s\" complete. Wrote %s bytes", "my-file.csv", strconv.FormatInt(bytesWritten, 10))
}

答案2

得分: 9

如果文件可能很大,你不希望它通过你自己的服务器传输。在我看来,最好的方法是让用户直接从S3下载。

你可以通过生成一个"预签名URL"来实现这一点:

func Download(w http.ResponseWriter, r *http.Request) {

    ...

    sess := session.New(&aws.Config{
        Region:             aws.String("eu-west-1"),
        Endpoint:           aws.String("s3-eu-west-1.amazonaws.com"),
        S3ForcePathStyle:   aws.Bool(true),
        Credentials:        cred,
    })

    s3svc := s3.New(sess)
    req, _ := s3svc.GetObjectRequest(&s3.GetObjectInput{
        Bucket: aws.String(BUCKET),
        Key: aws.String(filename),
    })

    url, err := req.Presign(5 * time.Minute)
    if err != nil {
        //处理错误
    }

    http.Redirect(w, r, url, http.StatusTemporaryRedirect)
}

预签名URL只在有限的时间内有效(在此示例中为5分钟,根据需要进行调整),并直接将用户引导到S3。不再需要担心下载了!

英文:

If the file is potentially large, you don't want it to go trough your own server.
The best approach (in my opinion) is to have the user download it directly from S3.

You can do this by generating a presigned url:

func Download(w http.ResponseWriter, r *http.Request) {

    ...
    
    sess := session.New(&aws.Config{
        Region:             aws.String("eu-west-1"),
        Endpoint:           aws.String("s3-eu-west-1.amazonaws.com"),
        S3ForcePathStyle:   aws.Bool(true),
        Credentials:        cred,
    })

    s3svc := s3.New(sess)
    req, _ := s3svc.GetObjectRequest(&s3.GetObjectInput{
        Bucket: aws.String(BUCKET),
        Key: aws.String(filename),
    })
    
    url, err := req.Presign(5 * time.Minute)
    if err != nil {
        //handle error
    }

    http.Redirect(w, r, url, http.StatusTemporaryRedirect)
}

The presigned url is only valid for a limited time (5 minutes in this example, adjust to your needs) and takes the user directly to S3. No need to worry about downloads anymore!

huangapple
  • 本文由 发表于 2016年2月7日 03:48:18
  • 转载请务必保留本文链接:https://go.coder-hub.com/35245649.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定