Golang io.copy在请求体上执行两次复制。

huangapple
go评论118阅读模式
英文:

Golang io.copy twice on the request body

问题

我正在构建一个 Blob 存储系统，并选择使用 Go 作为编程语言。
我创建了一个流来实现从客户端到 Blob 服务器的多部分文件上传。

流工作正常，但我想从请求体中创建一个 SHA1 哈希值。我需要对请求体进行两次 io.Copy 操作。SHA1 哈希值可以创建成功，但在此之后，多部分流变成了 0 字节。

有什么办法可以解决这个问题吗？

客户端上传代码如下：

  1. func (c *Client) Upload(h *UploadHandle) (*PutResult, error) {
  2.     body, bodySize, err := h.Read()
  3.     if err != nil {
  4.         return nil, err
  5.     }
  7.     // 从 body 字节创建一个 SHA1 哈希值
  8.     dropRef, err := drop.Sha1FromReader(body)
  9.     if err != nil {
  10.         return nil, err
  11.     }
  13.     bodyReader, bodyWriter := io.Pipe()
  14.     writer := multipart.NewWriter(bodyWriter)
  16.     errChan := make(chan error, 1)
  17.     go func() {
  18.         defer bodyWriter.Close()
  19.         part, err := writer.CreateFormFile(dropRef, dropRef)
  20.         if err != nil {
  21.             errChan <- err
  22.             return
  23.         }
  24.         if _, err := io.Copy(part, body); err != nil {
  25.             errChan <- err
  26.             return
  27.         }
  28.         if err = writer.Close(); err != nil {
  29.             errChan <- err
  30.         }
  31.     }()
  33.     req, err := http.NewRequest("POST", c.Server+"/drops/upload", bodyReader)
  34.     req.Header.Add("Content-Type", writer.FormDataContentType())
  35.     resp, err := c.Do(req)
  36.     if err != nil {
  37.         return nil, err
  38.     }
  39.     // ...
  40. }

SHA1 函数代码如下：

  1. func Sha1FromReader(src io.Reader) (string, error) {
  2.     hash := sha1.New()
  3.     _, err := io.Copy(hash, src)
  4.     if err != nil {
  5.         return "", err
  6.     }
  7.     return hex.EncodeToString(hash.Sum(nil)), nil
  8. }

上传处理代码如下：

  1. func (h *UploadHandle) Read() (io.Reader, int64, error) {
  2.     var b bytes.Buffer
  4.     hw := &Hasher{&b, sha1.New()}
  5.     n, err := io.Copy(hw, h.Contents)
  7.     if err != nil {
  8.         return nil, 0, err
  9.     }
  11.     return &b, n, nil
  12. }

以上是你提供的代码的翻译。

英文:

I am building a blob storage system and i picked Go as the programming language.
I create a stream to do a multipart file upload from client to the blob server.

The stream works fine, but i want to make a sha1 hash from the request body. I need to io.Copy the body twice. The sha1 gets created but the multipart streams 0 bytes after that.

  1. For creating the hash
  2. For streaming the body as multipart

any idea how i can do this?

the client upload

  1. func (c *Client) Upload(h *UploadHandle) (*PutResult, error) {
  2. body, bodySize, err := h.Read()
  3. if err != nil {
  4. 	return nil, err
  5. }
  7. // Creating a sha1 hash from the bytes of body
  8. dropRef, err := drop.Sha1FromReader(body)
  9. if err != nil {
  10. 	return nil, err
  11. }
  13. bodyReader, bodyWriter := io.Pipe()
  14. writer := multipart.NewWriter(bodyWriter)
  16. errChan := make(chan error, 1)
  17. go func() {
  18. 	defer bodyWriter.Close()
  19. 	part, err := writer.CreateFormFile(dropRef, dropRef)
  20. 	if err != nil {
  21. 		errChan &lt;- err
  22. 		return
  23. 	}
  24. 	if _, err := io.Copy(part, body); err != nil {
  25. 		errChan &lt;- err
  26. 		return
  27. 	}
  28. 	if err = writer.Close(); err != nil {
  29. 		errChan &lt;- err
  30. 	}
  31. }()
  33. req, err := http.NewRequest(&quot;POST&quot;, c.Server+&quot;/drops/upload&quot;, bodyReader)
  34. req.Header.Add(&quot;Content-Type&quot;, writer.FormDataContentType())
  35. resp, err := c.Do(req)
  36. if err != nil {
  37. 	return nil, err
  38. }
  39.   .....
  40.  }

the sha1 func

  1. func Sha1FromReader(src io.Reader) (string, error) {
  2. hash := sha1.New()
  3. _, err := io.Copy(hash, src)
  4. if err != nil {
  5. 	return &quot;&quot;, err
  6. }
  7. return hex.EncodeToString(hash.Sum(nil)), nil

}

upload handle

  1. func (h *UploadHandle) Read() (io.Reader, int64, error) {
  2. var b bytes.Buffer
  4. hw := &amp;Hasher{&amp;b, sha1.New()}
  5. n, err := io.Copy(hw, h.Contents)
  7. if err != nil {
  8. 	return nil, 0, err
  9. }
  11. return &amp;b, n, nil

}

答案1

得分: 26

我建议使用io.TeeReader，如果你想要并发地将所有从 blob 读取的内容传递给 sha1。

  1. bodyReader := io.TeeReader(body, hash)

现在，当上传过程中消耗 bodyReader 时，哈希值会自动更新。

英文:

I would suggest using an io.TeeReader if you want to push all reads from the blob through the sha1 concurrently.

  1. bodyReader := io.TeeReader(body, hash)

Now as the bodyReader is consumed during upload, the hash is automatically updated.

答案2

得分: 12

你可以编写一个包装器，在io.Copy上进行哈希处理，但不能直接进行哈希处理。

  1. // 这个包装器适用于读取器或写入器，但如果同时使用两者，哈希将会出错。
  2. type Hasher struct {
  3. 	io.Writer
  4. 	io.Reader
  5. 	hash.Hash
  6. 	Size uint64
  7. }
  9. func (h *Hasher) Write(p []byte) (n int, err error) {
  10. 	n, err = h.Writer.Write(p)
  11. 	h.Hash.Write(p)
  12. 	h.Size += uint64(n)
  13. 	return
  14. }
  16. func (h *Hasher) Read(p []byte) (n int, err error) {
  17. 	n, err = h.Reader.Read(p)
  18. 	h.Hash.Write(p[:n]) // 如果出错，n将为0，所以这仍然是安全的。
  19. 	return
  20. }
  22. func (h *Hasher) Sum() string {
  23. 	return hex.EncodeToString(h.Hash.Sum(nil))
  24. }
  26. func (h *UploadHandle) Read() (io.Reader, string, int64, error) {
  27. 	var b bytes.Buffer
  29. 	hashedReader := &Hasher{Reader: h.Contents, Hash: sha1.New()}
  30. 	n, err := io.Copy(&b, hashedReader)
  32. 	if err != nil {
  33. 		return nil, "", 0, err
  34. 	}
  36. 	return &b, hashedReader.Sum(), n, nil
  37. }
  39. // 基于@Dustin的评论更新的版本，因为我完全忘记了`io.TeeReader`的存在。
  41. func (h *UploadHandle) Read() (io.Reader, string, int64, error) {
  42. 	var b bytes.Buffer
  44. 	hash := sha1.New()
  45. 	n, err := io.Copy(&b, io.TeeReader(h.Contents, hash))
  47. 	if err != nil {
  48. 		return nil, "", 0, err
  49. 	}
  51. 	return &b, hex.EncodeToString(hash.Sum(nil)), n, nil
  52. }
英文:

You can't do that directly but you can write a wrapper that does the hashing on io.Copy

  1. // this works for either a reader or writer, 
  2. //  but if you use both in the same time the hash will be wrong.
  3. type Hasher struct {
  4. io.Writer
  5. io.Reader
  6. hash.Hash
  7. Size uint64
  8. }
  9. func (h *Hasher) Write(p []byte) (n int, err error) {
  10. n, err = h.Writer.Write(p)
  11. h.Hash.Write(p)
  12. h.Size += uint64(n)
  13. return
  14. }
  15. func (h *Hasher) Read(p []byte) (n int, err error) {
  16. n, err = h.Reader.Read(p)
  17. h.Hash.Write(p[:n]) //on error n is gonna be 0 so this is still safe.
  18. return
  19. }
  20. func (h *Hasher) Sum() string {
  21. return hex.EncodeToString(h.Hash.Sum(nil))
  22. }
  23. func (h *UploadHandle) Read() (io.Reader, string, int64, error) {
  24. var b bytes.Buffer
  25. hashedReader := &amp;Hasher{Reader: h.Contents, Hash: sha1.New()}
  26. n, err := io.Copy(&amp;b, hashedReader)
  27. if err != nil {
  28. return nil, &quot;&quot;, 0, err
  29. }
  30. return &amp;b, hashedReader.Sum(), n, nil
  31. }

// updated version based on @Dustin's comment since I complete forgot io.TeeReader existed.

  1. func (h *UploadHandle) Read() (io.Reader, string, int64, error) {
  2. var b bytes.Buffer
  3. hash := sha1.New()
  4. n, err := io.Copy(&amp;b, io.TeeReader(h.Contents, hash))
  5. if err != nil {
  6. return nil, &quot;&quot;, 0, err
  7. }
  8. return &amp;b, hex.EncodeToString(hash.Sum(nil)), n, nil
  9. }

答案3

得分: 2

你有两个选项。

最直接的方法是使用io.MultiWriter。

但是，如果你需要哈希函数生成多部分输出，那么你需要将其复制到一个bytes.Buffer中，然后将缓冲区写回每个写入器。

英文:

You have two options.

The most direct way is to use io.MultiWriter.

But if you need the hash to produce the multipart output, then you will have to copy to a bytes.Buffer and then write the buffer back to each writer.

答案4

得分: 1

我们可以将流转换为字符串，并根据需要多次创建它。

例如：

  1. readerStream := 来源的流
  2. buf := new(bytes.Buffer)
  3. buf.ReadFrom(readerStream)
  4. rawBody := buf.String()
  5. newReader1 := strings.NewReader(rawBody)
  6. newReader2 := strings.NewReader(rawBody)

但如果能够避免这样做，那将是很好的。

我不确定这是否是最佳方法，但它对我起作用了。

英文:

We can convert the stream into string and create it again as many times we need.
e.g.

  1. readerStream := your stream from source
  2. buf := new(bytes.Buffer)
  3. buf.ReadFrom(readerStream)
  4. rawBody := buf.String()
  5. newReader1 := strings.NewReader(rawBody)
  6. newReader2 := strings.NewReader(rawBody)

But it will be great if it can be avoided.

I am not sure it is the best approach. But it worked for me.

答案5

得分: 0

你可以使用Request.GetBody方法：

  1. package main
  2. import (
  3. "io"
  4. "net/http"
  5. "os"
  6. "strings"
  7. )
  8. func main() {
  9. read := strings.NewReader("north east south west")
  10. req, e := http.NewRequest("GET", "https://stackoverflow.com", read)
  11. if e != nil {
  12. panic(e)
  13. }
  14. // one
  15. io.Copy(os.Stdout, req.Body)
  16. // two
  17. body, e := req.GetBody()
  18. if e != nil {
  19. panic(e)
  20. }
  21. io.Copy(os.Stdout, body)
  22. }

https://golang.org/pkg/net/http#Request.GetBody

英文:

You can use Request.GetBody:

  1. package main
  2. import (
  3. &quot;io&quot;
  4. &quot;net/http&quot;
  5. &quot;os&quot;
  6. &quot;strings&quot;
  7. )
  8. func main() {
  9. read := strings.NewReader(&quot;north east south west&quot;)
  10. req, e := http.NewRequest(&quot;GET&quot;, &quot;https://stackoverflow.com&quot;, read)
  11. if e != nil {
  12. panic(e)
  13. }
  14. // one
  15. io.Copy(os.Stdout, req.Body)
  16. // two
  17. body, e := req.GetBody()
  18. if e != nil {
  19. panic(e)
  20. }
  21. io.Copy(os.Stdout, body)
  22. }

https://golang.org/pkg/net/http#Request.GetBody

huangapple
  • 本文由 发表于 2014年9月5日 01:10:25
  • 转载请务必保留本文链接：https://go.coder-hub.com/25671305.html
  • go
  • io
  • multipart
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定