使用Golang通过multipart处理图像并流式传输到Azure

huangapple
go评论132阅读模式
英文:

Golang processing images via multipart and streaming to Azure

问题

在学习golang的过程中，我正在尝试编写一个具有多图上传功能的Web应用程序。

我正在使用Azure Blob Storage来存储图像，但是我在将图像从多部分请求流式传输到Blob Storage时遇到了问题。

以下是我目前编写的处理程序：

  1. func (imgc *ImageController) UploadInstanceImageHandler(w http.ResponseWriter, r *http.Request, p httprouter.Params) {
  2.     reader, err := r.MultipartReader()
  4.     if err != nil {
  5.         http.Error(w, err.Error(), http.StatusInternalServerError)
  6.         return
  7.     }
  9.     for {
  10.         part, partErr := reader.NextPart()
  12.         // 没有更多的部分需要处理
  13.         if partErr == io.EOF {
  14.             break
  15.         }
  17.         // 如果part.FileName()为空，则跳过此次迭代。
  18.         if part.FileName() == "" {
  19.             continue
  20.         }
  22.         // 检查文件类型
  23.         if part.Header["Content-Type"][0] != "image/jpeg" {
  24.             fmt.Printf("\nNot image/jpeg!")
  25.             break
  26.         }
  28.         var read uint64
  29.         fileName := uuid.NewV4().String() + ".jpg"
  30.         buffer := make([]byte, 100000000)
  32.         // 获取大小
  33.         for {
  34.             cBytes, err := part.Read(buffer)
  36.             if err == io.EOF {
  37.                 fmt.Printf("\nLast buffer read!")
  38.                 break
  39.             }
  41.             read = read + uint64(cBytes)
  42.         }
  44.         stream := bytes.NewReader(buffer[0:read])
  45.         err = imgc.blobClient.CreateBlockBlobFromReader(imgc.imageContainer, fileName, read, stream, nil)
  47.         if err != nil {
  48.             fmt.Println(err)
  49.             break
  50.         }
  51.     }
  53.     w.WriteHeader(http.StatusOK)
  54. }

在我的研究过程中，我阅读了使用r.FormFile、ParseMultipartForm的方法，但决定尝试学习如何使用MultiPartReader。

我能够使用MultiPartReader将图像上传到golang后端并将文件保存到我的计算机上。

目前，我能够将文件上传到Azure，但它们最终会损坏。文件大小似乎是正确的，但显然有些地方出了问题。

我是否误解了如何为CreateBlockBlobFromReader创建io.Reader？

非常感谢任何帮助！

英文:

In the process of learning golang, I'm trying to write a web app with multiple image upload functionality.

I'm using Azure Blob Storage to store images, but I am having trouble streaming the images from the multipart request to Blob Storage.

Here's the handler I've written so far:

  1. func (imgc *ImageController) UploadInstanceImageHandler(w http.ResponseWriter, r *http.Request, p httprouter.Params) {
  2. reader, err := r.MultipartReader()
  3. if err != nil {
  4. http.Error(w, err.Error(), http.StatusInternalServerError)
  5. return
  6. }
  7. for {
  8. part, partErr := reader.NextPart()
  9. // No more parts to process
  10. if partErr == io.EOF {
  11. break
  12. }
  13. // if part.FileName() is empty, skip this iteration.
  14. if part.FileName() == "" {
  15. continue
  16. }
  17. // Check file type
  18. if part.Header["Content-Type"][0] != "image/jpeg" {
  19. fmt.Printf("\nNot image/jpeg!")
  20. break
  21. }
  22. var read uint64
  23. fileName := uuid.NewV4().String() + ".jpg"
  24. buffer := make([]byte, 100000000)
  25. // Get Size
  26. for {
  27. cBytes, err := part.Read(buffer)
  28. if err == io.EOF {
  29. fmt.Printf("\nLast buffer read!")
  30. break
  31. }
  32. read = read + uint64(cBytes)
  33. }
  34. stream := bytes.NewReader(buffer[0:read])
  35. err = imgc.blobClient.CreateBlockBlobFromReader(imgc.imageContainer, fileName, read, stream, nil)
  36. if err != nil {
  37. fmt.Println(err)
  38. break
  39. }
  40. }
  41. w.WriteHeader(http.StatusOK)

}

In the process of my research, I've read through using r.FormFile, ParseMultipartForm, but decided on trying to learn how to use MultiPartReader.

I was able to upload an image to the golang backend and save the file to my machine using MultiPartReader.

At the moment, I'm able to upload files to Azure but they end up being corrupted. The file sizes seem on point but clearly something is not working.

Am I misunderstanding how to create a io.Reader for CreateBlockBlobFromReader?

Any help is much appreciated!

答案1

得分: 4

如@Mark所说，您可以使用ioutil.ReadAll将内容读取到字节数组中，代码如下所示。

  1. import (
  2.    "bytes"
  3.    "io/ioutil"
  4. )
  6. partBytes, _ := ioutil.ReadAll(part)
  7. size := uint64(len(partBytes))
  8. blob := bytes.NewReader(partBytes)
  9. err := blobClient.CreateBlockBlobFromReader(container, fileName, size, blob, nil)

根据CreateBlockBlobFromReader的godoc，如下所示。

该API拒绝大小大于64 MiB的请求（但此限制未由SDK检查）。要写入更大的Blob，请使用CreateBlockBlob，PutBlock和PutBlockList。

因此，如果大小大于64MB，则代码应如下所示。

  1. import "encoding/base64"
  3. const BLOB_LENGTH_LIMITS uint64 = 64 * 1024 * 1024
  5. partBytes, _ := ioutil.ReadAll(part)
  6. size := uint64(len(partBytes))
  7. if size <= BLOB_LENGTH_LIMITS {
  8.    blob := bytes.NewReader(partBytes)
  9.    err := blobClient.CreateBlockBlobFromReader(container, fileName, size, blob, nil)
  10. } else {
  11.    // 创建一个空的Blob
  12.    blobClient.CreateBlockBlob(container, fileName)
  13.    // 创建一个块列表，并上传每个块
  14.    length := size / BLOB_LENGTH_LIMITS
  15.    if length%limits != 0 {
  16.        length = length + 1
  17.    }
  18.    blocks := make([]Block, length)
  19.    for i := uint64(0); i < length; i++ {
  20.         start := i * BLOB_LENGTH_LIMITS
  21.         end := (i+1) * BLOB_LENGTH_LIMITS
  22.         if end > size {
  23.             end = size
  24.         }
  25.         chunk := partBytes[start: end]
  26.         blockId := base64.StdEncoding.EncodeToString(chunk)
  27.         block := Block{blockId, storage.BlockStatusCommitted}
  28.         blocks[i] = block
  29.         err = blobClient.PutBlock(container, fileName, blockID, chunk)
  30.         if err != nil {
  31.         .......
  32.         }
  33.    }
  34.    err = blobClient.PutBlockList(container, fileName, blocks)
  35.    if err != nil {
  36.       .......
  37.    }
  38. }

希望对您有所帮助。

英文:

As @Mark said, you can use ioutil.ReadAll to read the content into a byte array, the code like below.

  1. import (
  2. &quot;bytes&quot;
  3. &quot;io/ioutil&quot;
  4. )
  5. partBytes, _ := ioutil.ReadAll(part)
  6. size := uint64(len(partBytes))
  7. blob := bytes.NewReader(partBytes)
  8. err := blobClient.CreateBlockBlobFromReader(container, fileName, size, blob, nil)

According to the godoc for CreateBlockBlobFromReader, as below.

> The API rejects requests with size > 64 MiB (but this limit is not checked by the SDK). To write a larger blob, use CreateBlockBlob, PutBlock, and PutBlockList.

So if the size is larger than 64MB, the code shoule be like below.

  1. import &quot;encoding/base64&quot;
  2. const BLOB_LENGTH_LIMITS uint64 = 64 * 1024 * 1024
  3. partBytes, _ := ioutil.ReadAll(part)
  4. size := uint64(len(partBytes))
  5. if size &lt;= BLOB_LENGTH_LIMITS {
  6. blob := bytes.NewReader(partBytes)
  7. err := blobClient.CreateBlockBlobFromReader(container, fileName, size, blob, nil)
  8. } else {
  9. // Create an empty blob
  10. blobClient.CreateBlockBlob(container, fileName)
  11. // Create a block list, and upload each block
  12. length := size / BLOB_LENGTH_LIMITS
  13. if length%limits != 0 {
  14. length = length + 1
  15. }
  16. blocks := make([]Block, length)
  17. for i := uint64(0); i &lt; length; i++ {
  18. start := i * BLOB_LENGTH_LIMITS
  19. end := (i+1) * BLOB_LENGTH_LIMITS
  20. if end &gt; size {
  21. end = size
  22. }
  23. chunk := partBytes[start: end]
  24. blockId := base64.StdEncoding.EncodeToString(chunk)
  25. block := Block{blockId, storage.BlockStatusCommitted}
  26. blocks[i] = block
  27. err = blobClient.PutBlock(container, fileName, blockID, chunk)
  28. if err != nil {
  29. .......
  30. }
  31. }
  32. err = blobClient.PutBlockList(container, fileName, blocks)
  33. if err != nil {
  34. .......
  35. }
  36. }

Hope it helps.

答案2

得分: 0

一个Reader可以同时返回io.EOF和有效的最后读取的字节，看起来最后的字节（cBytes）没有添加到read的总字节数中。另外要小心：如果part.Read(buffer)返回的错误不是io.EOF，读取循环可能不会退出。考虑使用ioutil.ReadAll。

CreateBlockBlobFromReader接受一个Reader，part也是一个Reader，所以你可以直接传递part。

你可能还需要考虑Azure块大小限制可能比图像小，参见Asure blobs。

英文:

A Reader can return both an io.EOF and a valid final bytes read, it looks like the final bytes (cBytes) is not added to read total bytes. Also, careful: if an error is returned by part.Read(buffer) other than io.EOF, the read loop might not exit. Consider ioutil.ReadAll instead.

CreateBlockBlobFromReader takes a Reader, and part is a Reader, so you may be able to pass the part in directly.

You may also want to consider Azure block size limits might be smaller than the image, see Asure blobs.

huangapple
  • 本文由 发表于 2017年4月3日 22:30:43
  • 转载请务必保留本文链接：https://go.coder-hub.com/43187362.html
  • azure
  • azure-blob-storage
  • go
  • image-uploading
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定