英文:
io.copyN cannot work when it is not called the first time
问题
我正在尝试从一个网站下载图片,步骤如下:
- 使用
http.Get
获取图片 - 使用
os.Create
在当前文件夹中创建一个新文件 - 使用
io.CopyN
将图片复制到文件中
但是如果 io.CopyN
在第一次失败后,后续似乎永远无法成功
代码片段:
download_again:
copy_byte, copy_err := io.CopyN(file, res.Body, res.ContentLength)
fmt.Fprintf(os.Stderr, "img(%s) size: %d\n", name, res.ContentLength)
if copy_err == nil && res.ContentLength == copy_byte {
fmt.Fprintf(os.Stderr, "成功下载图片(%s)[%f KB(%d B)]: %s\n", img_url, float64(copy_byte)/1024, copy_byte, name)
} else {
if try_i > download_times {
fmt.Fprintf(os.Stderr, "[致命错误] 下载图片(%s)失败 %s\n", img_url, name)
fout.WriteString(name + "\n")
return
}
fmt.Fprintf(os.Stderr, "下载图片(%s)[%f KB(%d B)]出错: %s, 尝试第 %d 次\n", img_url, float64(copy_byte)/1024, copy_byte, name, try_i)
try_i++
goto download_again
}
输出信息:
img(11085) size: 273047
下载图片(./style/images/dszp/11085.jpg)[171.447266 KB(175562 B)]出错: 11085,尝试第 1 次
img(11085) size: 273047
下载图片(./style/images/dszp/11085.jpg)[0.000000 KB(0 B)]出错: 11085,尝试第 2 次
img(11085) size: 273047
下载图片(./style/images/dszp/11085.jpg)[0.000000 KB(0 B)]出错: 11085,尝试第 3 次
img(11085) size: 273047
下载图片(./style/images/dszp/11085.jpg)[0.000000 KB(0 B)]出错: 11085,尝试第 4 次
img(11085) size: 273047
下载图片(./style/images/dszp/11085.jpg)[0.000000 KB(0 B)]出错: 11085,尝试第 5 次
img(11085) size: 273047
下载图片(./style/images/dszp/11085.jpg)[0.000000 KB(0 B)]出错: 11085,尝试第 6 次
任何帮助将不胜感激
英文:
I'm trying to download a image from a site, and the steps are as follow:
- use
http.Get
to fetch the image - use
os.Create
to create a new file in current folder - use
io.copyN
to copy the image into the file
But it is weird if the io.CopyN
is failed at the first time, it seems never success later
code fragment:
download_again:
copy_byte, copy_err := io.CopyN(file, res.Body, res.ContentLength)
fmt.Fprintf(os.Stderr, "img(%s) size: %d\n", name, res.ContentLength)
if copy_err == nil && res.ContentLength == copy_byte {
fmt.Fprintf(os.Stderr, "Success to download img(%s)[%f KB(%d B)]: %s\n", img_url, float64(copy_byte)/1024, copy_byte, name)
} else {
if try_i > download_times {
fmt.Fprintf(os.Stderr, "[fatal] fail to download img(%s) %s\n", img_url, name)
fout.WriteString(name + "\n")
return
}
fmt.Fprintf(os.Stderr, "error in download img(%s)[%f KB(%d B)]: %s, try %d times\n", img_url, float64(copy_byte)/1024, copy_byte, name, try_i)
try_i++
goto download_again
}
and the output message:
img(11085) size: 273047
error in download img(./style/images/dszp/11085.jpg)[171.447266 KB(175562 B)]: 11085 , try 1 times
img(11085) size: 273047
error in download img(./style/images/dszp/11085.jpg)[0.000000 KB(0 B)]: 11085 , try 2 times
img(11085) size: 273047
error in download img(./style/images/dszp/11085.jpg)[0.000000 KB(0 B)]: 11085, try 3 times
img(11085) size: 273047
error in download img(./style/images/dszp/11085.jpg)[0.000000 KB(0 B)]: 11085, try 4 times
img(11085) size: 273047
error in download img(./style/images/dszp/11085.jpg)[0.000000 KB(0 B)]: 11085, try 5 times
img(11085) size: 273047
error in download img(./style/images/dszp/11085.jpg)[0.000000 KB(0 B)]: 11085, try 6 times
Any help will be appreciated
答案1
得分: 0
你需要重新尝试HTTP请求(不仅仅是io.CopyN
)。
这是一个(大部分未经测试的)可恢复下载的实现示例:
func downloadFile(dst *os.File, url string, offset int64) (int64, error) {
req, err := http.NewRequest("GET", url, nil)
if err != nil {
return 0, err
}
// 尝试使用Range头部
if offset > 0 {
req.Header.Set("Range", fmt.Sprintf("bytes=%d-", offset))
}
res, err := http.DefaultClient.Do(req)
if err != nil {
return 0, err
}
defer res.Body.Close()
// 一些HTTP服务器不支持Range,所以在这种情况下我们丢弃前面的offset字节
if offset > 0 && res.StatusCode != http.StatusPartialContent {
_, err = io.CopyN(ioutil.Discard, res.Body, offset)
if err != nil {
return offset, err
}
}
n, err := io.CopyN(dst, res.Body, res.ContentLength)
return offset + n, err
}
你可以向该函数传递一个偏移量,它将尝试从该点继续下载(通过告诉HTTP服务器或丢弃字节)。它还返回你已经完成的进度,因此你可以将此函数包装在重试循环中:
func example() error {
f, err := os.Create("/tmp/pkg.png")
if err != nil {
return err
}
defer f.Close()
offset := int64(0)
delay := time.Second
// 尝试5次
for i := 0; i < 5; i++ {
offset, err = downloadFile(f, "http://golang.org/doc/gopher/pkg.png", offset)
if err == nil {
return nil
}
// 等待一段时间后再次尝试
time.Sleep(delay)
delay *= 2
}
return fmt.Errorf("在5次尝试后无法下载文件")
}
顺便说一下,你应该尽量避免在循环中使用goto
。
英文:
You need to retry the http request. (not just the io.CopyN
)
Here is a (mostly untested) implementation of a resumable download:
func downloadFile(dst *os.File, url string, offset int64) (int64, error) {
req, err := http.NewRequest("GET", url, nil)
if err != nil {
return 0, err
}
// try to use a range header
if offset > 0 {
req.Header.Set("Range", fmt.Sprintf("bytes=%d-", offset))
}
res, err := http.DefaultClient.Do(req)
if err != nil {
return 0, err
}
defer res.Body.Close()
// some http servers don't support range, so in that case we discard the first
// offset bytes
if offset > 0 && res.StatusCode != http.StatusPartialContent {
_, err = io.CopyN(ioutil.Discard, res.Body, offset)
if err != nil {
return offset, err
}
}
n, err := io.CopyN(dst, res.Body, res.ContentLength)
return offset + n, err
}
You can pass that function an offset and it will try to pick up from that point (either by telling the HTTP server, or by discarding the bytes). It also returns the progress you've made, so you can then wrap this function in a retry loop:
func example() error {
f, err := os.Create("/tmp/pkg.png")
if err != nil {
return err
}
defer f.Close()
offset := int64(0)
delay := time.Second
// try 5 times
for i := 0; i < 5; i++ {
offset, err = downloadFile(f, "http://golang.org/doc/gopher/pkg.png", offset)
if err == nil {
return nil
}
// wait a little while before trying again
time.Sleep(delay)
delay *= 2
}
return fmt.Errorf("failed to download file after 5 tries")
}
Incidentally, you should really avoid using goto
for loops.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论