golang s3 download to buffer using s3manager.downloader

huangapple go评论73阅读模式
英文:

golang s3 download to buffer using s3manager.downloader

问题

我正在使用Amazon S3 SDK下载文件,代码如下:

file, err := os.Create("/tmp/download_file")
downloader := s3manager.NewDownloader(session.New(&aws.Config{
    Region: aws.String("us-west-2"),
}))
numBytes, err := downloader.Download(file,
    &s3.GetObjectInput{
        Bucket: aws.String(bucketName),
        Key:    aws.String(fileName),
    })

它将文件下载到一个文件中。

如何将下载内容直接存入[]byte切片(缓冲区)中?我尝试了以下代码:

var tBuf bytes.Buffer
tBufIo := bufio.NewWriter(&tBuf)

而不是使用file。但是我得到了一个关于io.WriterAt接口的错误:

无法将*tBufIo(类型为bufio.Writer)作为downloader.Download的io.WriterAt参数使用。

英文:

I'm using the Amazon s3 SDK to download files like below:

file, err := os.Create("/tmp/download_file")
downloader := s3manager.NewDownloader(session.New(&aws.Config{
                                 Region: aws.String("us-west-2")}))
numBytes, err := downloader.Download(file,
	&s3.GetObjectInput{
		Bucket: aws.String(bucketName),
		Key:    aws.String(fileName),
	})

It downloads to a file.

How do I get the download content into a []byte slice (buffer) directly.
I tried something like:

var tBuf bytes.Buffer
tBufIo := bufio.NewWriter(&tBuf)

instead of "file". But I get an error for io.WriterAt interface

> cannot use *tBufIo (type bufio.Writer) as type io.WriterAt in argument to downloader.Download

答案1

得分: 47

从链接中找到了这段代码:

buff := &aws.WriteAtBuffer{}
downloader := s3manager.NewDownloader(session.New(&aws.Config{
                                      Region: aws.String(S3_Region)}))
numBytes, err := downloader.Download(buff,....
data := buff.Bytes() // 现在 data 是我的 []byte 数组

这段代码可以满足需求。

另请参阅:https://docs.aws.amazon.com/sdk-for-go/api/aws/#WriteAtBuffer

英文:

Found it from the link
https://groups.google.com/forum/#!topic/Golang-Nuts/4z8rcWEZ8Os

buff := &aws.WriteAtBuffer{}
downloader := s3manager.NewDownloader(session.New(&aws.Config{
                                      Region: aws.String(S3_Region)}))
numBytes, err := downloader.Download(buff,....
data := buff.Bytes() // now data is my []byte array

Works and fits the need.

See also: https://docs.aws.amazon.com/sdk-for-go/api/aws/#WriteAtBuffer

答案2

得分: 2

如果您正在使用AWS SDK v2,可以按照以下方式创建一个WriterAt:

import (
	"context"
	"errors"
	"github.com/aws/aws-sdk-go-v2/aws"
	"github.com/aws/aws-sdk-go-v2/config"
	"github.com/aws/aws-sdk-go-v2/credentials"
	"github.com/aws/aws-sdk-go-v2/service/s3"
	"github.com/aws/aws-sdk-go-v2/feature/s3/manager"
)

func DownloadS3File(objectKey string, bucket string, s3Client *s3.Client) ([]byte, error) {

	buffer := manager.NewWriteAtBuffer([]byte{})

	downloader := manager.NewDownloader(s3Client)

	numBytes, err := downloader.Download(context.TODO(), buffer, &s3.GetObjectInput{
		Bucket: aws.String(bucket),
		Key:    aws.String(objectKey),
	})
	if err != nil {
		return nil, err
	}

	if numBytes < 1 {
		return nil, errors.New("zero bytes written to memory")
	}

	return buffer.Bytes(), nil
}

这段代码使用了AWS SDK v2来下载S3文件。它创建了一个WriteAt缓冲区buffer,然后使用manager.NewDownloader创建一个下载器downloader。最后,调用downloader.Download方法来下载指定的S3对象,并将结果存储在buffer中。如果下载成功,将返回buffer.Bytes()作为字节数组;如果下载失败或写入的字节数为零,则会返回相应的错误。

英文:

If you are running AWS SDK v2 then you can create a WriterAt like this

import (
	&quot;context&quot;
	&quot;errors&quot;
	&quot;github.com/aws/aws-sdk-go-v2/aws&quot;
	&quot;github.com/aws/aws-sdk-go-v2/config&quot;
	&quot;github.com/aws/aws-sdk-go-v2/credentials&quot;
	&quot;github.com/aws/aws-sdk-go-v2/service/s3&quot;
 	&quot;github.com/aws/aws-sdk-go-v2/feature/s3/manager&quot;
)
func DownloadS3File(objectKey string, bucket string, s3Client *s3.Client) ([]byte, error) {

	buffer := manager.NewWriteAtBuffer([]byte{})

	downloader := manager.NewDownloader(s3Client)

	numBytes, err := downloader.Download(context.TODO(), buffer, &amp;s3.GetObjectInput{
		Bucket: aws.String(bucket),
		Key:    aws.String(objectKey),
	})
	if err != nil {
		return nil, err
	}

	if numBytes &lt; 1 {
		return nil, errors.New(&quot;zero bytes written to memory&quot;)
	}

	return buffer.Bytes(), nil
}

huangapple
  • 本文由 发表于 2017年1月14日 08:16:45
  • 转载请务必保留本文链接:https://go.coder-hub.com/41645377.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定