英文:
Go - correct usage of multipart Part.Read
问题
我一直在尝试使用multipart.Part来帮助读取HTTP上传的非常大的文件(>20GB),所以我编写了下面的代码,看起来工作得很好:
func ReceiveMultipartRoute(w http.ResponseWriter, r *http.Request) {
mediatype, p, err := mime.ParseMediaType(r.Header.Get("Content-Type"))
if err != nil {
//...
}
if mediatype != "multipart/form-data" {
//...
}
boundary := p["boundary"]
reader := multipart.NewReader(r.Body, boundary)
buffer := make([]byte, 8192)
for {
part, err := reader.NextPart()
if err != nil {
// ...
}
f, err := os.CreateTemp("", part.FileName())
if err != nil {
// ...
}
for {
numBytesRead, err := part.Read(buffer)
// People say not to read if there's an err, but then I miss the last chunk?
f.Write(buffer[:numBytesRead])
if err != nil {
if err == io.EOF {
break
} else {
// error, abort ...
return
}
}
}
}
}
然而,在最内层的循环中,我发现我必须在检查EOF之前从part.Read中读取数据,因为我注意到如果我事先这样做并且中断,我会错过最后一块数据。然而,我注意到在许多其他文章/帖子中,人们会检查错误/EOF,并且如果有错误/EOF,则使用最后一次读取而不进行读取。我是否错误/安全地使用了multipart.Part.Read()?
英文:
I've been trying to use multipart.Part to help read very large file uploads (>20GB) from HTTP - so I've written the below code which seems to work nicely:
func ReceiveMultipartRoute(w http.ResponseWriter, r *http.Request) {
mediatype, p, err := mime.ParseMediaType(r.Header.Get("Content-Type"))
if err != nil {
//...
}
if mediatype != "multipart/form-data" {
//...
}
boundary := p["boundary"]
reader := multipart.NewReader(r.Body, boundary)
buffer := make([]byte, 8192)
for {
part, err := reader.NextPart()
if err != nil {
// ...
}
f, err := os.CreateTemp("", part.FileName())
if err != nil {
// ...
}
for {
numBytesRead, err := part.Read(buffer)
// People say not to read if there's an err, but then I miss the last chunk?
f.Write(buffer[:numBytesRead])
if err != nil {
if err == io.EOF {
break
} else {
// error, abort ...
return
}
}
}
}
}
However, in the innermost for loop, I found out that I have to read from part.Read before even checking for EOF, as I notice that I will miss the last chunk if I do so beforehand and break. However, I notice on many other articles/posts where people check for errors/EOF, and break
-ing if there is without using the last read. Am I using multipart.Part.Read() wrongly/safely?
答案1
得分: 2
你在正确使用multipart.Part。
multipart.Part是io.Reader的一个特定实现。因此,你应该遵循惯例并遵循io.Reader的建议。引用文档中的一段话:
>在考虑错误err之前,调用者应始终处理返回的n > 0字节。这样做可以正确处理在读取一些字节后发生的I/O错误,以及两种允许的EOF行为。
此外,请注意,在你的示例中,你正在从io.Reader复制数据到os.File。os.File实现了io.ReaderFrom接口,因此你可以使用File.ReadFrom()方法来复制数据。
_, err := file.ReadFrom(part)
// 非 io.EOF
if err != nil {
return fmt.Errorf("复制数据:%w", err)
}
如果你需要使用缓冲区,可以使用io.CopyBuffer()函数。但请注意,你需要隐藏io.ReaderFrom的实现,否则缓冲区将不会被用于执行复制操作。参见示例:1,2,3。
_, err := io.CopyBuffer(writeFunc(file.Write), part, buffer)
// 非 io.EOF
if err != nil {
return fmt.Errorf("复制数据:%w", err)
}
type writeFunc func([]byte) (int, error)
func (write writeFunc) Write(data []byte) (int, error) {
return write(data)
}
英文:
You use multipart.Part in a proper way.
multipart.Part is a particular implementation of io.Reader. Accordingly, you should be guided by the conventions and follow the recommendations for io.Reader. Quote from the documentation:
>Callers should always process the n > 0 bytes returned before considering the error err. Doing so correctly handles I/O errors that happen after reading some bytes and also both of the allowed EOF behaviors.
Also note that in the example you are copying data from io.Reader to os.File. os.File implements io.ReaderFrom interface, so you can use File.ReadFrom() method to copy the data.
_, err := file.ReadFrom(part)
// non io.EOF
if err != nil {
return fmt.Errorf("copy data: %w", err)
}
If you need to use a buffer, you can use io.CopyBuffer() function. But note that you need to hide io.ReaderFrom implementation, otherwise the buffer will not be used to perform the copy. See examples: 1, 2, 3.
_, err := io.CopyBuffer(writeFunc(file.Write), part, buffer)
// non io.EOF
if err != nil {
return fmt.Errorf("copy data: %w", err)
}
type writeFunc func([]byte) (int, error)
func (write writeFunc) Write(data []byte) (int, error) {
return write(data)
}
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论