英文:
Consistent off by one errors in go io.copy function for large content
问题
这个代码段可以正常工作：
_, err = io.Copy(out, resp.Body)
if err != nil {
ErrLog.Fatal(err)
}
这个代码段在处理较大的响应（几兆字节）时会产生一致的偏移一个字节的错误（在下载内容的最后一个字节被省略，例如在 JSON 响应中省略了一个闭合的 ]）：
if _, err := io.Copy(out, resp.Body); err != nil {
ErrLog.Fatal(err)
}
根据官方 Golang 博客上的示例，这个语法应该是有效的。
编辑：更多细节和背景信息
这是我在第二个版本的代码（更紧凑的错误处理）中遇到的错误：
ERROR: 2015/08/05 08:09:31 pull.go:257: unexpected end of JSON input
以下是另一个函数中的代码：
err = json.Unmarshal(dat, &all_data)
if err != nil {
return err
}
我通过查看每种情况下文件的前10个和后10个字符来发现了偏移一个字节的问题。以下是修改前后的对比：
# 修改前（存在错误）
START: [{"tags":[ END: ersion":1}
START: [{"_create END: "tags":[]}
# 修改后
START: [{"tags":[ END: rsion":1}]
START: [{"_create END: tags":[]}]
这些文件是 15-20 MB 的 JSON 字符串。
英文:
This one works consistently.
_, err = io.Copy(out, resp.Body)
if err != nil {
ErrLog.Fatal(err)
}
This one gives pretty consistent off by one errors (the last byte of the downloaded content is left off, in my case a closing ] in a json response) for large-ish responses (MBs).
if _, err := io.Copy(out, resp.Body); err != nil {
ErrLog.Fatal(err)
}
From the examples on the official golang blog, it looks like this should be valid syntax.
Edit: Some more details and context
This is the error I get with the 2nd version of the code (more compact error handling)
ERROR: 2015/08/05 08:09:31 pull.go:257: unexpected end of JSON input
From this code in another function
err = json.Unmarshal(dat, &all_data)
if err != nil {
return err
}
I found the off by one issue by looking at the first 10 and last 10 characters of the file in each case. Here are the before and afters:
# Before (with error)
START: [{"tags":[ END: ersion":1}
START: [{"_create END: "tags":[]}
# After
START: [{"tags":[ END: rsion":1}]
START: [{"_create END: tags":[]}]
The files are 15-20 Mb json strings.
答案1
得分: 0
问题至少部分是由于竞态条件引起的。
在退出函数之前，我没有对文件out调用.Close()。添加了这个之后，问题就没有再出现了。
为什么仅仅是最后一个字节有时会丢失，对我来说还是个谜。
英文:
It turned out the issue was at least partially the result of a race condition.
I was not calling .Close() on the file out before exiting the function. After adding that I have not been any more issues.
Why exactly this was causing just the last byte of the file to get dropped sometimes is a mystery to me.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论