英文:
Upload Image URL To S3 With Buffering Process
问题
我发现了许多与将图像上传到S3相关的问题，但我还没有找到一个可以完全帮助我的帖子。所以，我决定提出我的问题。
我有一个案例，需要将图像从URL上传到S3，所以我所做的就是首先从URL下载图像，然后上传到S3。
// 从URL下载图像
resp, _ := http.Get("https://sample-videos.com/img/Sample-jpg-image-50kb.jpg")
byteImage, _ = ioutil.ReadAll(resp.Body)
// 上传到S3（使用s3manager）
_, err = s3ManagerClient.Upload(&s3manager.UploadInput{
Bucket: aws.String(awsS3Bucket),
Key: aws.String(awsS3Destination),
Body: bytes.NewReader(byteImage),
})
但是，我的代码存在一个问题，即上传过程必须等待下载过程完成。我的目标是在不等待下载过程完成的情况下将图像上传到S3，也就是说，下载和上传过程同时进行。
这是我的新代码（为了更易读，删除了错误处理部分）
resp, _ := http.Get("https://sample-videos.com/img/Sample-jpg-image-50kb.jpg")
// 初始化S3
sess := session.Must(session.NewSession(&aws.Config{
Region: aws.String(awsRegion),
Credentials: credentials.NewStaticCredentials(awsS3Key, awsS3Secret, ""),
}))
s3ManagerClient := s3manager.NewUploader(sess, func(u *s3manager.Uploader) {
u.PartSize = int64(DefaultS3OptionPartSizeInMB)
u.Concurrency = DefaultS3OptionConcurrency
})
buf := make([]byte, 1024) // 每kb
for {
n, _ := resp.Body.Read(buf)
if n == 0 {
break
}
_, err = s3ManagerClient.Upload(&s3manager.UploadInput{
Bucket: aws.String(awsS3Bucket),
Key: aws.String(awsS3Destination),
Body: bytes.NewReader(buf[:n]), // 每部分上传图像
})
}
这段代码成功将图像上传到S3，但文件损坏，我怀疑图像没有完全上传（过程中出现了问题）。
英文:
I found many questions related to uploading image to S3, but I haven't found a post that can help me fully. So, I decided to ask my problem.
I have the case to upload image from the URL to S3, so what I did is just download the image from URL first, and then upload to S3.
// download image form url
resp, _ := http.Get("https://sample-videos.com/img/Sample-jpg-image-50kb.jpg")
byteImage, _ = ioutil.ReadAll(resp.Body)
// upload to s3 (using s3manager)
_, err = s3ManagerClient.Upload(&s3manager.UploadInput{
Bucket: aws.String(awsS3Bucket),
Key: aws.String(awsS3Destination),
Body: bytes.NewReader(byteImage),
})
But, there is a problem with my code, the upload process must wait until the download process is completed. My goal is to upload image to s3 without waiting for the download process to complete, meaning that the download and upload process occurs simultaneously.
Here is my new code (handling error is removed for more readable)
resp, _ := http.Get("https://sample-videos.com/img/Sample-jpg-image-50kb.jpg")
// init s3
sess := session.Must(session.NewSession(&aws.Config{
Region: aws.String(awsRegion),
Credentials: credentials.NewStaticCredentials(awsS3Key, awsS3Secret, ""),
}))
s3ManagerClient := s3manager.NewUploader(sess, func(u *s3manager.Uploader) {
u.PartSize = int64(DefaultS3OptionPartSizeInMB)
u.Concurrency = DefaultS3OptionConcurrency
})
buf := make([]byte, 1024) // per kb
for {
n, _ := resp.Body.Read(buf)
if n == 0 {
break
}
_, err = s3ManagerClient.Upload(&s3manager.UploadInput{
Bucket: aws.String(awsS3Bucket),
Key: aws.String(awsS3Destination),
Body: bytes.NewReader(buf[:n]), // upload image per part
})
}
This code successfully uploads the image to S3, but the file is broken, I suspect that the image is not fully uploaded (there is something wrong with the process).
答案1
得分: 1
你的问题是for循环会将图像以字节块的形式上传到awsS3Destination。实际上，你正在逐个字节块地替换图像的内容。
你可以尝试将响应体传递给S3上传器：
resp, err := http.Get("https://sample-videos.com/img/Sample-jpg-image-50kb.jpg")
if err != nil {
// 处理错误
}
_, err = s3ManagerClient.Upload(&s3manager.UploadInput{
Bucket: aws.String(awsS3Bucket),
Key: aws.String(awsS3Destination),
Body: resp.Body, // 上传图像
})
英文:
Your problem is that the for-loop will upload the image in chunks of bytes onto the awsS3Destination. What you really are doing is you are replacing the contents of the image with individual chunks of bytes, one at a time.
You can try instead to pass the response body into the s3 uploader:
resp, err := http.Get("https://sample-videos.com/img/Sample-jpg-image-50kb.jpg")
if err != nil {
// handle error
}
_, err = s3ManagerClient.Upload(&s3manager.UploadInput{
Bucket: aws.String(awsS3Bucket),
Key: aws.String(awsS3Destination),
Body: resp.Body, // upload image
})
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论