英文:
golang s3 download to buffer using s3manager.downloader
问题
我正在使用Amazon S3 SDK下载文件,代码如下:
file, err := os.Create("/tmp/download_file")
downloader := s3manager.NewDownloader(session.New(&aws.Config{
Region: aws.String("us-west-2"),
}))
numBytes, err := downloader.Download(file,
&s3.GetObjectInput{
Bucket: aws.String(bucketName),
Key: aws.String(fileName),
})
它将文件下载到一个文件中。
如何将下载内容直接存入[]byte
切片(缓冲区)中?我尝试了以下代码:
var tBuf bytes.Buffer
tBufIo := bufio.NewWriter(&tBuf)
而不是使用file
。但是我得到了一个关于io.WriterAt
接口的错误:
无法将*tBufIo(类型为bufio.Writer)作为downloader.Download的io.WriterAt参数使用。
英文:
I'm using the Amazon s3 SDK to download files like below:
file, err := os.Create("/tmp/download_file")
downloader := s3manager.NewDownloader(session.New(&aws.Config{
Region: aws.String("us-west-2")}))
numBytes, err := downloader.Download(file,
&s3.GetObjectInput{
Bucket: aws.String(bucketName),
Key: aws.String(fileName),
})
It downloads to a file.
How do I get the download content into a []byte
slice (buffer) directly.
I tried something like:
var tBuf bytes.Buffer
tBufIo := bufio.NewWriter(&tBuf)
instead of "file". But I get an error for io.WriterAt
interface
> cannot use *tBufIo (type bufio.Writer) as type io.WriterAt in argument to downloader.Download
答案1
得分: 47
从链接中找到了这段代码:
buff := &aws.WriteAtBuffer{}
downloader := s3manager.NewDownloader(session.New(&aws.Config{
Region: aws.String(S3_Region)}))
numBytes, err := downloader.Download(buff,....
data := buff.Bytes() // 现在 data 是我的 []byte 数组
这段代码可以满足需求。
另请参阅:https://docs.aws.amazon.com/sdk-for-go/api/aws/#WriteAtBuffer
英文:
Found it from the link
https://groups.google.com/forum/#!topic/Golang-Nuts/4z8rcWEZ8Os
buff := &aws.WriteAtBuffer{}
downloader := s3manager.NewDownloader(session.New(&aws.Config{
Region: aws.String(S3_Region)}))
numBytes, err := downloader.Download(buff,....
data := buff.Bytes() // now data is my []byte array
Works and fits the need.
See also: https://docs.aws.amazon.com/sdk-for-go/api/aws/#WriteAtBuffer
答案2
得分: 2
如果您正在使用AWS SDK v2,可以按照以下方式创建一个WriterAt:
import (
"context"
"errors"
"github.com/aws/aws-sdk-go-v2/aws"
"github.com/aws/aws-sdk-go-v2/config"
"github.com/aws/aws-sdk-go-v2/credentials"
"github.com/aws/aws-sdk-go-v2/service/s3"
"github.com/aws/aws-sdk-go-v2/feature/s3/manager"
)
func DownloadS3File(objectKey string, bucket string, s3Client *s3.Client) ([]byte, error) {
buffer := manager.NewWriteAtBuffer([]byte{})
downloader := manager.NewDownloader(s3Client)
numBytes, err := downloader.Download(context.TODO(), buffer, &s3.GetObjectInput{
Bucket: aws.String(bucket),
Key: aws.String(objectKey),
})
if err != nil {
return nil, err
}
if numBytes < 1 {
return nil, errors.New("zero bytes written to memory")
}
return buffer.Bytes(), nil
}
这段代码使用了AWS SDK v2来下载S3文件。它创建了一个WriteAt
缓冲区buffer
,然后使用manager.NewDownloader
创建一个下载器downloader
。最后,调用downloader.Download
方法来下载指定的S3对象,并将结果存储在buffer
中。如果下载成功,将返回buffer.Bytes()
作为字节数组;如果下载失败或写入的字节数为零,则会返回相应的错误。
英文:
If you are running AWS SDK v2 then you can create a WriterAt like this
import (
"context"
"errors"
"github.com/aws/aws-sdk-go-v2/aws"
"github.com/aws/aws-sdk-go-v2/config"
"github.com/aws/aws-sdk-go-v2/credentials"
"github.com/aws/aws-sdk-go-v2/service/s3"
"github.com/aws/aws-sdk-go-v2/feature/s3/manager"
)
func DownloadS3File(objectKey string, bucket string, s3Client *s3.Client) ([]byte, error) {
buffer := manager.NewWriteAtBuffer([]byte{})
downloader := manager.NewDownloader(s3Client)
numBytes, err := downloader.Download(context.TODO(), buffer, &s3.GetObjectInput{
Bucket: aws.String(bucket),
Key: aws.String(objectKey),
})
if err != nil {
return nil, err
}
if numBytes < 1 {
return nil, errors.New("zero bytes written to memory")
}
return buffer.Bytes(), nil
}
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论