Go – multipart Part.Read 的正确用法

huangapple go评论81阅读模式
英文:

Go - correct usage of multipart Part.Read

问题

我一直在尝试使用multipart.Part来帮助读取HTTP上传的非常大的文件(>20GB),所以我编写了下面的代码,看起来工作得很好:

func ReceiveMultipartRoute(w http.ResponseWriter, r *http.Request) {
	mediatype, p, err := mime.ParseMediaType(r.Header.Get("Content-Type"))
	if err != nil {
	   //...
	}
	if mediatype != "multipart/form-data" {
		//...
	}
	boundary := p["boundary"]
	reader := multipart.NewReader(r.Body, boundary)
	buffer := make([]byte, 8192)
	for {
		part, err := reader.NextPart()
		if err != nil {
			// ... 
		}
	
		f, err := os.CreateTemp("", part.FileName())
		if err != nil {
			// ...
		}
	
		for {
			numBytesRead, err := part.Read(buffer)
			// People say not to read if there's an err, but then I miss the last chunk?
			f.Write(buffer[:numBytesRead])
			if err != nil {
				if err == io.EOF {
					break
				} else {
					// error, abort ...
					return
				}
			}
		}
	}
}

然而,在最内层的循环中,我发现我必须在检查EOF之前从part.Read中读取数据,因为我注意到如果我事先这样做并且中断,我会错过最后一块数据。然而,我注意到在许多其他文章/帖子中,人们会检查错误/EOF,并且如果有错误/EOF,则使用最后一次读取而不进行读取。我是否错误/安全地使用了multipart.Part.Read()?

英文:

I've been trying to use multipart.Part to help read very large file uploads (>20GB) from HTTP - so I've written the below code which seems to work nicely:

func ReceiveMultipartRoute(w http.ResponseWriter, r *http.Request) {
	mediatype, p, err := mime.ParseMediaType(r.Header.Get("Content-Type"))
	if err != nil {
	   //...
	}
	if mediatype != "multipart/form-data" {
		//...
	}
	boundary := p["boundary"]
	reader := multipart.NewReader(r.Body, boundary)
	buffer := make([]byte, 8192)
	for {
		part, err := reader.NextPart()
		if err != nil {
			// ... 
		}
	
		f, err := os.CreateTemp("", part.FileName())
		if err != nil {
			// ...
		}
	
		for {
			numBytesRead, err := part.Read(buffer)
			// People say not to read if there's an err, but then I miss the last chunk?
			f.Write(buffer[:numBytesRead])
			if err != nil {
				if err == io.EOF {
					break
				} else {
					// error, abort ...
					return
				}
			}
		}
	}
}

However, in the innermost for loop, I found out that I have to read from part.Read before even checking for EOF, as I notice that I will miss the last chunk if I do so beforehand and break. However, I notice on many other articles/posts where people check for errors/EOF, and break-ing if there is without using the last read. Am I using multipart.Part.Read() wrongly/safely?

答案1

得分: 2

你在正确使用multipart.Part

multipart.Partio.Reader的一个特定实现。因此,你应该遵循惯例并遵循io.Reader的建议。引用文档中的一段话:

>在考虑错误err之前,调用者应始终处理返回的n > 0字节。这样做可以正确处理在读取一些字节后发生的I/O错误,以及两种允许的EOF行为。

此外,请注意,在你的示例中,你正在从io.Reader复制数据到os.Fileos.File实现了io.ReaderFrom接口,因此你可以使用File.ReadFrom()方法来复制数据。

_, err := file.ReadFrom(part)
// 非 io.EOF
if err != nil {
    return fmt.Errorf("复制数据:%w", err)
}

如果你需要使用缓冲区,可以使用io.CopyBuffer()函数。但请注意,你需要隐藏io.ReaderFrom的实现,否则缓冲区将不会被用于执行复制操作。参见示例:123

_, err := io.CopyBuffer(writeFunc(file.Write), part, buffer)
// 非 io.EOF
if err != nil {
    return fmt.Errorf("复制数据:%w", err)
}

type writeFunc func([]byte) (int, error)

func (write writeFunc) Write(data []byte) (int, error) {
        return write(data)
}
英文:

You use multipart.Part in a proper way.

multipart.Part is a particular implementation of io.Reader. Accordingly, you should be guided by the conventions and follow the recommendations for io.Reader. Quote from the documentation:

>Callers should always process the n > 0 bytes returned before considering the error err. Doing so correctly handles I/O errors that happen after reading some bytes and also both of the allowed EOF behaviors.


Also note that in the example you are copying data from io.Reader to os.File. os.File implements io.ReaderFrom interface, so you can use File.ReadFrom() method to copy the data.

_, err := file.ReadFrom(part)
// non io.EOF
if err != nil {
    return fmt.Errorf("copy data: %w", err)
}

If you need to use a buffer, you can use io.CopyBuffer() function. But note that you need to hide io.ReaderFrom implementation, otherwise the buffer will not be used to perform the copy. See examples: 1, 2, 3.

_, err := io.CopyBuffer(writeFunc(file.Write), part, buffer)
// non io.EOF
if err != nil {
    return fmt.Errorf("copy data: %w", err)
}

type writeFunc func([]byte) (int, error)

func (write writeFunc) Write(data []byte) (int, error) {
        return write(data)
}

huangapple
  • 本文由 发表于 2022年11月21日 17:22:50
  • 转载请务必保留本文链接:https://go.coder-hub.com/74516664.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定