Golang 并发下载死锁

huangapple
go评论122阅读模式
英文:

Golang concurrent download deadlock

问题

我想在Go语言中并行下载文件，但我的代码从未退出：

  1. package main
  3. import (
  4. 	"fmt"
  5. 	"io"
  6. 	"net/http"
  7. 	"os"
  8. 	"path/filepath"
  9. 	"sync"
  10. )
  12. func download_file(file_path string, wg sync.WaitGroup) {
  13. 	defer wg.Done()
  14. 	resp, _ := http.Get(file_path)
  15. 	defer resp.Body.Close()
  16. 	filename := filepath.Base(file_path)
  17. 	file, _ := os.Create(filename)
  18. 	defer file.Close()
  20. 	size, _ := io.Copy(file, resp.Body)
  21. 	fmt.Println(filename, size, resp.Status)
  22. }
  24. func main() {
  25. 	var wg sync.WaitGroup
  27. 	file_list := []string{
  28. 		"http://i.imgur.com/dxGb2uZ.jpg",
  29. 		"http://i.imgur.com/RSU6NxX.jpg",
  30. 		"http://i.imgur.com/hUWgS2S.jpg",
  31. 		"http://i.imgur.com/U8kaix0.jpg",
  32. 		"http://i.imgur.com/w3cEYpY.jpg",
  33. 		"http://i.imgur.com/ooSCD9T.jpg"}
  34. 	fmt.Println(len(file_list))
  35. 	for _, url := range file_list {
  36. 		wg.Add(1)
  37. 		fmt.Println(wg)
  38. 		go download_file(url, wg)
  40. 	}
  41. 	wg.Wait()
  42. }

原因是你在download_file函数中传递了sync.WaitGroup的值而不是指针。在Go语言中，如果你想在函数中修改一个变量的值，你需要传递指针而不是值。所以你需要将download_file函数的参数改为指针类型：

  1. func download_file(file_path string, wg *sync.WaitGroup) {
  2. 	//...
  3. }

此外，你还可以在http.Get和os.Create等函数调用中检查错误并处理它们，以便更好地调试代码。你可以使用if err != nil语句来检查错误并采取适当的措施。

希望这可以帮助你解决问题！

英文:

I want to download files in parallel in go, but my code never exits:

  1. package main
  3. import (
  4.     "fmt"
  5.     "io"
  6.     "net/http"
  7.     "os"
  8.     "path/filepath"
  9.     "sync"
  10. )
  12. func download_file(file_path string, wg sync.WaitGroup) {
  13.     defer wg.Done()
  14.     resp, _ := http.Get(file_path)
  15.     defer resp.Body.Close()
  16.     filename := filepath.Base(file_path)
  17.     file, _ := os.Create(filename)
  18.     defer file.Close()
  20.     size, _ := io.Copy(file, resp.Body)
  21.     fmt.Println(filename, size, resp.Status)
  22. }
  24. func main() {
  25.     var wg sync.WaitGroup
  27.     file_list := []string{
  28. 	    "http://i.imgur.com/dxGb2uZ.jpg",
  29. 	    "http://i.imgur.com/RSU6NxX.jpg",
  30. 	    "http://i.imgur.com/hUWgS2S.jpg",
  31. 	    "http://i.imgur.com/U8kaix0.jpg",
  32. 	    "http://i.imgur.com/w3cEYpY.jpg",
  33. 	    "http://i.imgur.com/ooSCD9T.jpg"}
  34.     fmt.Println(len(file_list))
  35.     for _, url := range file_list {
  36. 	    wg.Add(1)
  37. 	    fmt.Println(wg)
  38. 	    go download_file(url, wg)
  40.     }
  41.     wg.Wait()
  42. }

What's the reason? I've looked here: https://stackoverflow.com/questions/23635070/golang-download-multiple-files-in-parallel-using-goroutines but I found no solution.
What is the best way to debug such code?

答案1

得分: 1

根据Tim Cooper的说法，你需要将WaitGroup作为指针传递。如果在你的代码上运行go vet工具，它会给出以下警告：

  1. $ go vet ex.go
  2. ex.go:12: download_file passes Lock by value: sync.WaitGroup contains sync.Mutex
  3. exit status 1

我建议你使用一个可以在保存文件时自动执行此操作的编辑器。例如，对于Atom编辑器，可以使用go-plus插件。

至于代码，我认为你应该按照以下方式重构它：

  1. package main
  3. import (
  4. 	"fmt"
  5. 	"io"
  6. 	"net/http"
  7. 	"os"
  8. 	"path/filepath"
  9. 	"sync"
  10. )
  12. func downloadFile(filePath string) error {
  13. 	resp, err := http.Get(filePath)
  14. 	if err != nil {
  15. 		return err
  16. 	}
  17. 	defer resp.Body.Close()
  19. 	name := filepath.Base(filePath)
  21. 	file, err := os.Create(name)
  22. 	if err != nil {
  23. 		return err
  24. 	}
  25. 	defer file.Close()
  27. 	size, err := io.Copy(file, resp.Body)
  28. 	if err != nil {
  29. 		return err
  30. 	}
  31. 	fmt.Println(name, size, resp.Status)
  32. 	return nil
  33. }
  35. func main() {
  36. 	var wg sync.WaitGroup
  38. 	fileList := []string{
  39. 		"http://i.imgur.com/dxGb2uZ.jpg",
  40. 		"http://i.imgur.com/RSU6NxX.jpg",
  41. 		"http://i.imgur.com/hUWgS2S.jpg",
  42. 		"http://i.imgur.com/U8kaix0.jpg",
  43. 		"http://i.imgur.com/w3cEYpY.jpg",
  44. 		"http://i.imgur.com/ooSCD9T.jpg"}
  45. 	fmt.Println("downloading", len(fileList), "files")
  46. 	for _, url := range fileList {
  47. 		wg.Add(1)
  48. 		go func(url string) {
  49. 			err := downloadFile(url)
  50. 			if err != nil {
  51. 				fmt.Println("[error]", url, err)
  52. 			}
  53. 			wg.Done()
  54. 		}(url)
  55. 	}
  56. 	wg.Wait()
  57. }

我不喜欢在函数之间传递WaitGroup，而更喜欢保持函数简单、阻塞和顺序执行，然后在更高层次上组合并发。这样，你可以选择按顺序执行所有操作，而无需更改downloadFile函数。

我还添加了错误处理并修复了变量名，使其符合驼峰命名法。

英文:

As Tim Cooper said you need to pass the WaitGroup as a pointer. If you run the go vet tool on your code it will give you this warning:

  1. $ go vet ex.go
  2. ex.go:12: download_file passes Lock by value: sync.WaitGroup contains sync.Mutex
  3. exit status 1

I recommend using an editor that can do this for you when you save a file. For example go-plus for Atom.

As for the code I think you should restructure it like this:

  1. package main
  3. import (
  4. 	"fmt"
  5. 	"io"
  6. 	"net/http"
  7. 	"os"
  8. 	"path/filepath"
  9. 	"sync"
  10. )
  12. func downloadFile(filePath string) error {
  13. 	resp, err := http.Get(filePath)
  14. 	if err != nil {
  15. 		return err
  16. 	}
  17. 	defer resp.Body.Close()
  19. 	name := filepath.Base(filePath)
  21. 	file, err := os.Create(name)
  22. 	if err != nil {
  23. 		return err
  24. 	}
  25. 	defer file.Close()
  27. 	size, err := io.Copy(file, resp.Body)
  28. 	if err != nil {
  29. 		return err
  30. 	}
  31. 	fmt.Println(name, size, resp.Status)
  32. 	return nil
  33. }
  35. func main() {
  36. 	var wg sync.WaitGroup
  38. 	fileList := []string{
  39. 		"http://i.imgur.com/dxGb2uZ.jpg",
  40. 		"http://i.imgur.com/RSU6NxX.jpg",
  41. 		"http://i.imgur.com/hUWgS2S.jpg",
  42. 		"http://i.imgur.com/U8kaix0.jpg",
  43. 		"http://i.imgur.com/w3cEYpY.jpg",
  44. 		"http://i.imgur.com/ooSCD9T.jpg"}
  45. 	fmt.Println("downloading", len(fileList), "files")
  46. 	for _, url := range fileList {
  47. 		wg.Add(1)
  48. 		go func(url string) {
  49. 			err := downloadFile(url)
  50. 			if err != nil {
  51. 				fmt.Println("[error]", url, err)
  52. 			}
  53. 			wg.Done()
  54. 		}(url)
  55. 	}
  56. 	wg.Wait()
  57. }

I don't like passing WaitGroups around and prefer to keep functions simple, blocking and sequential and then stitch together the concurrency at a higher level. This gives you the option of doing it all sequentially without having to change downloadFile.

I also added error handling and fixed names so they are camelCase.

答案2

得分: 1

除了Calab的回答之外，你的方法没有任何问题，你只需要将指向sync.WaitGroup的指针传递给它即可。

  1. func download_file(file_path string, wg *sync.WaitGroup) {
  2.     defer wg.Done()
  3.     ......
  4. }
  5. ......
  6. go download_file(url, &wg)
  7. .....

playground

英文:

Adding to Calab's response, there's absolutely nothing wrong with your approach, all you had to do is to pass a pointer to the sync.WaitGroup.

  1. func download_file(file_path string, wg *sync.WaitGroup) {
  2. 	defer wg.Done()
  3.     ......
  4. }
  5. .....
  6.         go download_file(url, &wg)
  7. .....

<kbd>playground</kbd>

huangapple
  • 本文由 发表于 2015年5月9日 19:21:04
  • 转载请务必保留本文链接：https://go.coder-hub.com/30139279.html
  • concurrency
  • go
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定