使用goroutines下载文件。

huangapple go评论90阅读模式
英文:

downloading files with goroutines?

问题

我是你的中文翻译助手,以下是翻译好的内容:

我刚开始学习Go语言,正在学习如何使用goroutine。

我有一个下载图片的函数:

func imageDownloader(uri string, filename string) {
    fmt.Println("starting download for ", uri)

    outFile, err := os.Create(filename)
    defer outFile.Close()
    if err != nil {
        os.Exit(1)
    }

    client := &http.Client{}

    req, err := http.NewRequest("GET", uri, nil)

    resp, err := client.Do(req)
    defer resp.Body.Close()

    if err != nil {
        panic(err)
    }

    header := resp.ContentLength
    bar := pb.New(int(header))
    rd := bar.NewProxyReader(resp.Body)
    // and copy from reader
    io.Copy(outFile, rd)
}

当我单独调用这个函数作为另一个函数的一部分时,它可以完整地下载图片,没有截断的数据。

然而,当我尝试修改它使其成为一个goroutine时,图片经常被截断或者生成长度为零的文件。

func imageDownloader(uri string, filename string, wg *sync.WaitGroup) {
    ...
    io.Copy(outFile, rd)
    wg.Done()
}

func main() {
    var wg sync.WaitGroup
    wg.Add(1)
    go imageDownloader(url, file, &wg)
    wg.Wait()
}

我是否错误地使用了WaitGroup?这可能是什么原因,我该如何解决?

更新:

问题已解决。我将wg.Add()函数放在了循环外面。 使用goroutines下载文件。

英文:

I'm new to Go and I'm learning how to work with goroutines.

I have a function that downloads images:

func imageDownloader(uri string, filename string) {
	fmt.Println("starting download for ", uri)

	outFile, err := os.Create(filename)
	defer outFile.Close()
	if err != nil {
		os.Exit(1)
	}

	client := &http.Client{}

	req, err := http.NewRequest("GET", uri, nil)

	resp, err := client.Do(req)
	defer resp.Body.Close()

	if err != nil {
		panic(err)
	}

	header := resp.ContentLength
	bar := pb.New(int(header))
	rd := bar.NewProxyReader(resp.Body)
	// and copy from reader
	io.Copy(outFile, rd)
}

When I call by itself as part of another function, it downloads images completely and there is no truncated data.

However, when I try to modify it to make it a goroutine, images are often truncated or zero length files.

func imageDownloader(uri string, filename string, wg *sync.WaitGroup) {
    ...
    io.Copy(outFile, rd)
    wg.Done()
}

func main() {
var wg sync.WaitGroup
wg.Add(1)
go imageDownloader(url, file, &wg)
wg.Wait()
}

Am I using WaitGroups incorrectly? What could cause this and how can I fix it?

Update:

Solved it. I had placed the wg.add() function outside of a loop. 使用goroutines下载文件。

答案1

得分: 3

虽然我不确定导致你的问题的具体原因,但以下是两种解决方法,可以帮助你恢复正常工作。

首先,参考sync库中的使用waitgroup的示例,尝试在函数开头调用defer wg.Done(),以确保即使goroutine意外结束,waitgroup也能正确递减。

其次,io.Copy返回一个错误,你没有进行检查。这不是一个好的实践,但在你的特定情况下,它会阻止你查看复制过程中是否存在错误。检查并适当处理该错误。它还返回写入的字节数,这可能对你有所帮助。

英文:

While I'm not sure exactly what's causing your issue, here's two options for how to get it back into working order.

First, looking to the example of how to use waitgroups from the sync library, try calling defer wg.Done() at the beginning of your function to ensure that even if the goroutine ends unexpectedly, that the waitgroup is properly decremented.

Second, io.Copy returns an error that you're not checking. That's not great practice anyway, but in your particular case it's preventing you from seeing if there is indeed an error in the copying routine. Check it and deal with it appropriately. It also returns the number of bytes written, which might help you as well.

答案2

得分: 3

你的示例在使用WaitGroups方面没有明显的问题。只要你在启动goroutine时调用wg.Add()的次数与goroutine的数量相同,或者每次启动新的goroutine时将其递增1,那就是正确的。

然而,在goroutine中,你调用了os.Exitpanic来处理某些错误条件,所以如果有多个这样的goroutine在运行,其中任何一个失败都会终止所有的goroutine,而不管WaitGroups的使用情况。如果它在没有panic消息的情况下失败,我建议检查os.Exit(1)这一行。

在Go语言中,使用defer wg.Done()在函数开头是一个好的实践,这样即使发生错误,goroutine仍然会递减其计数器。这样,如果其中一个goroutine返回错误,主线程也不会在完成时挂起。

英文:

Your example doesn't have anything obviously wrong with its use of WaitGroups. As long as you are calling wg.Add() with the same number as the number of goroutines you launch, or incrementing it by 1 every time you start a new goroutine, that should be correct.

However you call os.Exit and panic for certain errors conditions in the goroutine, so if you have more than one of these running, a failure in any one of them will terminate all of them, regardless of the use of WaitGroups. If it's failing without a panic message, I would take a look at the os.Exit(1) line.

It would also, be good practice in go to use defer wg.Done() at the start of your function, so that even if an error occurs, the goroutine still decrements its counter. That way your main thread won't hang on completion if one of the goroutines returns an error.

答案3

得分: 0

我在你的示例中会做一个改变,就是在你完成时利用defer。我认为defer ws.Done()应该是你函数中的第一条语句。

我喜欢WaitGroup的简洁性。然而,我不喜欢我们需要传递对goroutine的引用,因为这意味着并发逻辑会与业务逻辑混合在一起。

所以我想出了这个通用函数来解决这个问题:

// Parallelize并行化函数调用
func Parallelize(functions ...func()) {
    var waitGroup sync.WaitGroup
    waitGroup.Add(len(functions))

    defer waitGroup.Wait()

    for _, function := range functions {
        go func(copy func()) {
            defer waitGroup.Done()
            copy()
        }(function)
    }
}

所以你的示例可以这样解决:

func imageDownloader(uri string, filename string) {
    ...
    io.Copy(outFile, rd)
}

func main() {
    functions := []func(){}
    list := make([]Object, 5)
    for _, object := range list {
        function := func(obj Object){ 
            imageDownloader(object.uri, object.filename) 
        }(object)
        functions = append(functions, function)
    }

    Parallelize(functions...)        

    fmt.Println("Done")
}

如果你想使用它,你可以在这里找到:https://github.com/shomali11/util

英文:

One change I would make in your example is leverage defer when you are Done. I think this defer ws.Done() should be the first statement in your function.

I like WaitGroup's simplicity. However, I do not like that we need to pass the reference to the goroutine because that would mean that the concurrency logic would be mixed with your business logic.

So I came up with this generic function to solve this problem for me:

// Parallelize parallelizes the function calls
func Parallelize(functions ...func()) {
    var waitGroup sync.WaitGroup
    waitGroup.Add(len(functions))

	defer waitGroup.Wait()

	for _, function := range functions {
    	go func(copy func()) {
	    	defer waitGroup.Done()
		    copy()
	    }(function)
    }
}

So your example could be solved this way:

func imageDownloader(uri string, filename string) {
    ...
    io.Copy(outFile, rd)
}

func main() {
    functions := []func(){}
    list := make([]Object, 5)
    for _, object := range list {
        function := func(obj Object){ 
            imageDownloader(object.uri, object.filename) 
        }(object)
        functions = append(functions, function)
    }

    Parallelize(functions...)        

    fmt.Println("Done")
}

If you would like to use it, you can find it here https://github.com/shomali11/util

huangapple
  • 本文由 发表于 2015年10月1日 12:28:23
  • 转载请务必保留本文链接:https://go.coder-hub.com/32879472.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定