Golang 并发下载死锁

huangapple go评论78阅读模式
英文:

Golang concurrent download deadlock

问题

我想在Go语言中并行下载文件,但我的代码从未退出:

package main

import (
	"fmt"
	"io"
	"net/http"
	"os"
	"path/filepath"
	"sync"
)

func download_file(file_path string, wg sync.WaitGroup) {
	defer wg.Done()
	resp, _ := http.Get(file_path)
	defer resp.Body.Close()
	filename := filepath.Base(file_path)
	file, _ := os.Create(filename)
	defer file.Close()

	size, _ := io.Copy(file, resp.Body)
	fmt.Println(filename, size, resp.Status)
}

func main() {
	var wg sync.WaitGroup

	file_list := []string{
		"http://i.imgur.com/dxGb2uZ.jpg",
		"http://i.imgur.com/RSU6NxX.jpg",
		"http://i.imgur.com/hUWgS2S.jpg",
		"http://i.imgur.com/U8kaix0.jpg",
		"http://i.imgur.com/w3cEYpY.jpg",
		"http://i.imgur.com/ooSCD9T.jpg"}
	fmt.Println(len(file_list))
	for _, url := range file_list {
		wg.Add(1)
		fmt.Println(wg)
		go download_file(url, wg)

	}
	wg.Wait()
}

原因是你在download_file函数中传递了sync.WaitGroup的值而不是指针。在Go语言中,如果你想在函数中修改一个变量的值,你需要传递指针而不是值。所以你需要将download_file函数的参数改为指针类型:

func download_file(file_path string, wg *sync.WaitGroup) {
	//...
}

此外,你还可以在http.Getos.Create等函数调用中检查错误并处理它们,以便更好地调试代码。你可以使用if err != nil语句来检查错误并采取适当的措施。

希望这可以帮助你解决问题!

英文:

I want to download files in parallel in go, but my code never exits:

package main

import (
    "fmt"
    "io"
    "net/http"
    "os"
    "path/filepath"
    "sync"
)

func download_file(file_path string, wg sync.WaitGroup) {
    defer wg.Done()
    resp, _ := http.Get(file_path)
    defer resp.Body.Close()
    filename := filepath.Base(file_path)
    file, _ := os.Create(filename)
    defer file.Close()

    size, _ := io.Copy(file, resp.Body)
    fmt.Println(filename, size, resp.Status)
}

func main() {
    var wg sync.WaitGroup

    file_list := []string{
	    "http://i.imgur.com/dxGb2uZ.jpg",
	    "http://i.imgur.com/RSU6NxX.jpg",
	    "http://i.imgur.com/hUWgS2S.jpg",
	    "http://i.imgur.com/U8kaix0.jpg",
	    "http://i.imgur.com/w3cEYpY.jpg",
	    "http://i.imgur.com/ooSCD9T.jpg"}
    fmt.Println(len(file_list))
    for _, url := range file_list {
	    wg.Add(1)
	    fmt.Println(wg)
	    go download_file(url, wg)

    }
    wg.Wait()
}

What's the reason? I've looked here: https://stackoverflow.com/questions/23635070/golang-download-multiple-files-in-parallel-using-goroutines but I found no solution.
What is the best way to debug such code?

答案1

得分: 1

根据Tim Cooper的说法,你需要将WaitGroup作为指针传递。如果在你的代码上运行go vet工具,它会给出以下警告:

$ go vet ex.go
ex.go:12: download_file passes Lock by value: sync.WaitGroup contains sync.Mutex
exit status 1

我建议你使用一个可以在保存文件时自动执行此操作的编辑器。例如,对于Atom编辑器,可以使用go-plus插件。

至于代码,我认为你应该按照以下方式重构它:

package main

import (
	"fmt"
	"io"
	"net/http"
	"os"
	"path/filepath"
	"sync"
)

func downloadFile(filePath string) error {
	resp, err := http.Get(filePath)
	if err != nil {
		return err
	}
	defer resp.Body.Close()

	name := filepath.Base(filePath)

	file, err := os.Create(name)
	if err != nil {
		return err
	}
	defer file.Close()

	size, err := io.Copy(file, resp.Body)
	if err != nil {
		return err
	}
	fmt.Println(name, size, resp.Status)
	return nil
}

func main() {
	var wg sync.WaitGroup

	fileList := []string{
		"http://i.imgur.com/dxGb2uZ.jpg",
		"http://i.imgur.com/RSU6NxX.jpg",
		"http://i.imgur.com/hUWgS2S.jpg",
		"http://i.imgur.com/U8kaix0.jpg",
		"http://i.imgur.com/w3cEYpY.jpg",
		"http://i.imgur.com/ooSCD9T.jpg"}
	fmt.Println("downloading", len(fileList), "files")
	for _, url := range fileList {
		wg.Add(1)
		go func(url string) {
			err := downloadFile(url)
			if err != nil {
				fmt.Println("[error]", url, err)
			}
			wg.Done()
		}(url)
	}
	wg.Wait()
}

我不喜欢在函数之间传递WaitGroup,而更喜欢保持函数简单、阻塞和顺序执行,然后在更高层次上组合并发。这样,你可以选择按顺序执行所有操作,而无需更改downloadFile函数。

我还添加了错误处理并修复了变量名,使其符合驼峰命名法。

英文:

As Tim Cooper said you need to pass the WaitGroup as a pointer. If you run the go vet tool on your code it will give you this warning:

$ go vet ex.go
ex.go:12: download_file passes Lock by value: sync.WaitGroup contains sync.Mutex
exit status 1

I recommend using an editor that can do this for you when you save a file. For example go-plus for Atom.

As for the code I think you should restructure it like this:

package main

import (
	"fmt"
	"io"
	"net/http"
	"os"
	"path/filepath"
	"sync"
)

func downloadFile(filePath string) error {
	resp, err := http.Get(filePath)
	if err != nil {
		return err
	}
	defer resp.Body.Close()

	name := filepath.Base(filePath)

	file, err := os.Create(name)
	if err != nil {
		return err
	}
	defer file.Close()

	size, err := io.Copy(file, resp.Body)
	if err != nil {
		return err
	}
	fmt.Println(name, size, resp.Status)
	return nil
}

func main() {
	var wg sync.WaitGroup

	fileList := []string{
		"http://i.imgur.com/dxGb2uZ.jpg",
		"http://i.imgur.com/RSU6NxX.jpg",
		"http://i.imgur.com/hUWgS2S.jpg",
		"http://i.imgur.com/U8kaix0.jpg",
		"http://i.imgur.com/w3cEYpY.jpg",
		"http://i.imgur.com/ooSCD9T.jpg"}
	fmt.Println("downloading", len(fileList), "files")
	for _, url := range fileList {
		wg.Add(1)
		go func(url string) {
			err := downloadFile(url)
			if err != nil {
				fmt.Println("[error]", url, err)
			}
			wg.Done()
		}(url)
	}
	wg.Wait()
}

I don't like passing WaitGroups around and prefer to keep functions simple, blocking and sequential and then stitch together the concurrency at a higher level. This gives you the option of doing it all sequentially without having to change downloadFile.

I also added error handling and fixed names so they are camelCase.

答案2

得分: 1

除了Calab的回答之外,你的方法没有任何问题,你只需要将指向sync.WaitGroup的指针传递给它即可。

func download_file(file_path string, wg *sync.WaitGroup) {
    defer wg.Done()
    ......
}
......
go download_file(url, &wg)
.....

playground

英文:

Adding to Calab's response, there's absolutely nothing wrong with your approach, all you had to do is to pass a pointer to the sync.WaitGroup.

func download_file(file_path string, wg *sync.WaitGroup) {
	defer wg.Done()
    ......
}
.....
        go download_file(url, &wg)
.....

<kbd>playground</kbd>

huangapple
  • 本文由 发表于 2015年5月9日 19:21:04
  • 转载请务必保留本文链接:https://go.coder-hub.com/30139279.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定