英文:
Can not fetch the complete file from nginx with http.Get in GO
问题
我已经编写了一个函数，使用http.Get()从Nginx服务器获取文件（该服务器与获取文件的主机位于同一台主机上，仅用于测试）。代码如下：
res,err := http.Get(addr) // addr是Nginx服务器上的文件地址
defer res.Body.Close()
for {
v := &vFile{path,0} // path是要写入的文件路径
bv :=bufio.NewWriterSize(v,1024*1024) // 每次写入1MB
_, err:= io.Copy(bv,res.Body)
if err == nil { err = bv.Flush() }
}
其中vFile和Write函数定义如下：
type vFile struct {
path string
cur int64
}
func (wtr *vFile) Write(buf []byte) {
var f *os.File
if wtr.cur == 0 { f,wtr.err = os.Create(wtr.path) }
else { f,wtr.err = os.OpenFile(wtr.path,os.O_RDWR|os.O_APPEND,0666) }
_, err := f.WriteAt(buf,twr.path)
}
然而，在高并发情况下（例如，并发数为500），大量文件无法完全获取，并且在Nginx日志中，HTTP响应为200，但文件长度不正确：
"GET /videos/4b42d6e8e138233c7eb62939.mp4 HTTP/1.1" 200 37863424 "-" "Go 1.1 package http" "-"
该文件的大小为75273523字节，但只有37863424字节被获取。如果我将大小从1MB更改为32KB，情况会好得多，但仍有一些文件不完整。那么代码可能出了什么问题？
英文:
I have written a function that uses http.Get() to fetch files from the Nginx server (which is on the same host as the one fetching files just for test). The code is like this:
res,err := http.Get(addr) // addr is the file address on Nginx server
defer res.Body.Close()
for {
v := &vFile{path,0} // path is the file path to write
bv :=bufio.NewWriterSize(v,1024*1024) // Write 1MB each time
_, err:= io.Copy(bv,res.Body)
if err == nil { err = bv.Flush() }
}
with vFile and the Write function defined like
type vFile struct {
path string
cur int64
}
func (wtr *vFile) Write(buf []byte) {
var f *os.File
if wtr.cur == 0 { f,wtr.err = os.Create(wtr.path) }
else { f,wtr.err = os.OpenFile(wtr.path,os.O_RDWR|os.O_APPEND,0666) }
_, err := f.WriteAt(buf,twr.path)
}
However, in high concurrency situation (for example, the concurrency number is 500), a large number of files are not fully fetched, and in Nginx log the HTTP response is 200 while the file length is not correct:
"GET /videos/4b42d6e8e138233c7eb62939.mp4 HTTP/1.1" 200 37863424 "-" "Go 1.1 package http" "-"
The file has a size of 75273523 Bytes while only 37863424 Bytes are fetched. If I change the size from 1MB to 32KB, the situation can be much better, but there is still a few files that is not complete. So what might be wrong with the code?
答案1
得分: 1
在这个函数中，你没有关闭 os.File。我怀疑这就是出错的原因。
func (wtr *vFile) Write(buf []byte) {
var f *os.File
if wtr.cur == 0 { f,wtr.err = os.Create(wtr.path) }
else { f,wtr.err = os.OpenFile(wtr.path,os.O_RDWR|os.O_APPEND,0666) }
_, err := f.WriteAt(buf,twr.path)
}
如果不是这个问题，请提供一个完整的、经过编译和格式化的示例！
英文:
In this function you aren't closing the os.File. I suspect that is why it is going wrong.
func (wtr *vFile) Write(buf []byte) {
var f *os.File
if wtr.cur == 0 { f,wtr.err = os.Create(wtr.path) }
else { f,wtr.err = os.OpenFile(wtr.path,os.O_RDWR|os.O_APPEND,0666) }
_, err := f.WriteAt(buf,twr.path)
}
If not post a complete compiling gofmt-ed example!
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论