io.copyN在不是第一次调用时无法工作。

huangapple go评论78阅读模式
英文:

io.copyN cannot work when it is not called the first time

问题

我正在尝试从一个网站下载图片,步骤如下:

  • 使用 http.Get 获取图片
  • 使用 os.Create 在当前文件夹中创建一个新文件
  • 使用 io.CopyN 将图片复制到文件中

但是如果 io.CopyN 在第一次失败后,后续似乎永远无法成功

代码片段:

	download_again:
		copy_byte, copy_err := io.CopyN(file, res.Body, res.ContentLength)
		fmt.Fprintf(os.Stderr, "img(%s) size: %d\n", name, res.ContentLength)
		if copy_err == nil && res.ContentLength == copy_byte {
			fmt.Fprintf(os.Stderr, "成功下载图片(%s)[%f KB(%d B)]: %s\n", img_url, float64(copy_byte)/1024, copy_byte, name)
		} else {
			if try_i > download_times {
				fmt.Fprintf(os.Stderr, "[致命错误] 下载图片(%s)失败 %s\n", img_url, name)
				fout.WriteString(name + "\n")
				return
			}
			fmt.Fprintf(os.Stderr, "下载图片(%s)[%f KB(%d B)]出错: %s, 尝试第 %d 次\n", img_url, float64(copy_byte)/1024, copy_byte, name, try_i)
			try_i++
			goto download_again
		}

输出信息:

img(11085) size: 273047
下载图片(./style/images/dszp/11085.jpg)[171.447266 KB(175562 B)]出错: 11085,尝试第 1 次
img(11085) size: 273047
下载图片(./style/images/dszp/11085.jpg)[0.000000 KB(0 B)]出错: 11085,尝试第 2 次
img(11085) size: 273047
下载图片(./style/images/dszp/11085.jpg)[0.000000 KB(0 B)]出错: 11085,尝试第 3 次
img(11085) size: 273047
下载图片(./style/images/dszp/11085.jpg)[0.000000 KB(0 B)]出错: 11085,尝试第 4 次
img(11085) size: 273047
下载图片(./style/images/dszp/11085.jpg)[0.000000 KB(0 B)]出错: 11085,尝试第 5 次
img(11085) size: 273047
下载图片(./style/images/dszp/11085.jpg)[0.000000 KB(0 B)]出错: 11085,尝试第 6 次

任何帮助将不胜感激 io.copyN在不是第一次调用时无法工作。

英文:

I'm trying to download a image from a site, and the steps are as follow:

  • use http.Get to fetch the image
  • use os.Create to create a new file in current folder
  • use io.copyN to copy the image into the file

But it is weird if the io.CopyN is failed at the first time, it seems never success later

code fragment:

	download_again:
		copy_byte, copy_err := io.CopyN(file, res.Body, res.ContentLength)
		fmt.Fprintf(os.Stderr, "img(%s) size: %d\n", name, res.ContentLength)
		if copy_err == nil && res.ContentLength == copy_byte {
			fmt.Fprintf(os.Stderr, "Success to download img(%s)[%f KB(%d B)]: %s\n", img_url, float64(copy_byte)/1024, copy_byte, name)
		} else {
			if try_i > download_times {
				fmt.Fprintf(os.Stderr, "[fatal] fail to download img(%s) %s\n", img_url, name)
				fout.WriteString(name + "\n")
				return
			}
			fmt.Fprintf(os.Stderr, "error in download img(%s)[%f KB(%d B)]: %s, try %d times\n", img_url, float64(copy_byte)/1024, copy_byte, name, try_i)
			try_i++
			goto download_again
		}

and the output message:

img(11085) size: 273047
error in download img(./style/images/dszp/11085.jpg)[171.447266 KB(175562 B)]: 11085 , try 1 times
img(11085) size: 273047
error in download img(./style/images/dszp/11085.jpg)[0.000000 KB(0 B)]: 11085 , try 2 times
img(11085) size: 273047
error in download img(./style/images/dszp/11085.jpg)[0.000000 KB(0 B)]: 11085, try 3 times
img(11085) size: 273047
error in download img(./style/images/dszp/11085.jpg)[0.000000 KB(0 B)]: 11085, try 4 times
img(11085) size: 273047
error in download img(./style/images/dszp/11085.jpg)[0.000000 KB(0 B)]: 11085, try 5 times
img(11085) size: 273047
error in download img(./style/images/dszp/11085.jpg)[0.000000 KB(0 B)]: 11085, try 6 times

Any help will be appreciated io.copyN在不是第一次调用时无法工作。

答案1

得分: 0

你需要重新尝试HTTP请求(不仅仅是io.CopyN)。

这是一个(大部分未经测试的)可恢复下载的实现示例:

func downloadFile(dst *os.File, url string, offset int64) (int64, error) {
    req, err := http.NewRequest("GET", url, nil)
    if err != nil {
        return 0, err
    }
    // 尝试使用Range头部
    if offset > 0 {
        req.Header.Set("Range", fmt.Sprintf("bytes=%d-", offset))
    }
    res, err := http.DefaultClient.Do(req)
    if err != nil {
        return 0, err
    }
    defer res.Body.Close()

    // 一些HTTP服务器不支持Range,所以在这种情况下我们丢弃前面的offset字节
    if offset > 0 && res.StatusCode != http.StatusPartialContent {
        _, err = io.CopyN(ioutil.Discard, res.Body, offset)
        if err != nil {
            return offset, err
        }
    }

    n, err := io.CopyN(dst, res.Body, res.ContentLength)
    return offset + n, err
}

你可以向该函数传递一个偏移量,它将尝试从该点继续下载(通过告诉HTTP服务器或丢弃字节)。它还返回你已经完成的进度,因此你可以将此函数包装在重试循环中:

func example() error {
    f, err := os.Create("/tmp/pkg.png")
    if err != nil {
        return err
    }
    defer f.Close()

    offset := int64(0)
    delay := time.Second
    // 尝试5次
    for i := 0; i < 5; i++ {
        offset, err = downloadFile(f, "http://golang.org/doc/gopher/pkg.png", offset)
        if err == nil {
            return nil
        }
        // 等待一段时间后再次尝试
        time.Sleep(delay)
        delay *= 2
    }
    return fmt.Errorf("在5次尝试后无法下载文件")
}

顺便说一下,你应该尽量避免在循环中使用goto

英文:

You need to retry the http request. (not just the io.CopyN)

Here is a (mostly untested) implementation of a resumable download:

func downloadFile(dst *os.File, url string, offset int64) (int64, error) {
	req, err := http.NewRequest(&quot;GET&quot;, url, nil)
	if err != nil {
		return 0, err
	}
	// try to use a range header
	if offset &gt; 0 {
		req.Header.Set(&quot;Range&quot;, fmt.Sprintf(&quot;bytes=%d-&quot;, offset))
	}
	res, err := http.DefaultClient.Do(req)
	if err != nil {
		return 0, err
	}
	defer res.Body.Close()

	// some http servers don&#39;t support range, so in that case we discard the first
	// offset bytes
	if offset &gt; 0 &amp;&amp; res.StatusCode != http.StatusPartialContent {
		_, err = io.CopyN(ioutil.Discard, res.Body, offset)
		if err != nil {
			return offset, err
		}
	}

	n, err := io.CopyN(dst, res.Body, res.ContentLength)
	return offset + n, err
}

You can pass that function an offset and it will try to pick up from that point (either by telling the HTTP server, or by discarding the bytes). It also returns the progress you've made, so you can then wrap this function in a retry loop:

func example() error {
	f, err := os.Create(&quot;/tmp/pkg.png&quot;)
	if err != nil {
		return err
	}
	defer f.Close()

	offset := int64(0)
	delay := time.Second
	// try 5 times
	for i := 0; i &lt; 5; i++ {
		offset, err = downloadFile(f, &quot;http://golang.org/doc/gopher/pkg.png&quot;, offset)
		if err == nil {
			return nil
		}
		// wait a little while before trying again
		time.Sleep(delay)
		delay *= 2
	}
	return fmt.Errorf(&quot;failed to download file after 5 tries&quot;)
}

Incidentally, you should really avoid using goto for loops.

huangapple
  • 本文由 发表于 2015年5月20日 09:42:29
  • 转载请务必保留本文链接:https://go.coder-hub.com/30338913.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定