英文:
Golang concurrent download deadlock
问题
我想在Go语言中并行下载文件,但我的代码从未退出:
package main
import (
"fmt"
"io"
"net/http"
"os"
"path/filepath"
"sync"
)
func download_file(file_path string, wg sync.WaitGroup) {
defer wg.Done()
resp, _ := http.Get(file_path)
defer resp.Body.Close()
filename := filepath.Base(file_path)
file, _ := os.Create(filename)
defer file.Close()
size, _ := io.Copy(file, resp.Body)
fmt.Println(filename, size, resp.Status)
}
func main() {
var wg sync.WaitGroup
file_list := []string{
"http://i.imgur.com/dxGb2uZ.jpg",
"http://i.imgur.com/RSU6NxX.jpg",
"http://i.imgur.com/hUWgS2S.jpg",
"http://i.imgur.com/U8kaix0.jpg",
"http://i.imgur.com/w3cEYpY.jpg",
"http://i.imgur.com/ooSCD9T.jpg"}
fmt.Println(len(file_list))
for _, url := range file_list {
wg.Add(1)
fmt.Println(wg)
go download_file(url, wg)
}
wg.Wait()
}
原因是你在download_file
函数中传递了sync.WaitGroup
的值而不是指针。在Go语言中,如果你想在函数中修改一个变量的值,你需要传递指针而不是值。所以你需要将download_file
函数的参数改为指针类型:
func download_file(file_path string, wg *sync.WaitGroup) {
//...
}
此外,你还可以在http.Get
和os.Create
等函数调用中检查错误并处理它们,以便更好地调试代码。你可以使用if err != nil
语句来检查错误并采取适当的措施。
希望这可以帮助你解决问题!
英文:
I want to download files in parallel in go, but my code never exits:
package main
import (
"fmt"
"io"
"net/http"
"os"
"path/filepath"
"sync"
)
func download_file(file_path string, wg sync.WaitGroup) {
defer wg.Done()
resp, _ := http.Get(file_path)
defer resp.Body.Close()
filename := filepath.Base(file_path)
file, _ := os.Create(filename)
defer file.Close()
size, _ := io.Copy(file, resp.Body)
fmt.Println(filename, size, resp.Status)
}
func main() {
var wg sync.WaitGroup
file_list := []string{
"http://i.imgur.com/dxGb2uZ.jpg",
"http://i.imgur.com/RSU6NxX.jpg",
"http://i.imgur.com/hUWgS2S.jpg",
"http://i.imgur.com/U8kaix0.jpg",
"http://i.imgur.com/w3cEYpY.jpg",
"http://i.imgur.com/ooSCD9T.jpg"}
fmt.Println(len(file_list))
for _, url := range file_list {
wg.Add(1)
fmt.Println(wg)
go download_file(url, wg)
}
wg.Wait()
}
What's the reason? I've looked here: https://stackoverflow.com/questions/23635070/golang-download-multiple-files-in-parallel-using-goroutines but I found no solution.
What is the best way to debug such code?
答案1
得分: 1
根据Tim Cooper的说法,你需要将WaitGroup
作为指针传递。如果在你的代码上运行go vet
工具,它会给出以下警告:
$ go vet ex.go
ex.go:12: download_file passes Lock by value: sync.WaitGroup contains sync.Mutex
exit status 1
我建议你使用一个可以在保存文件时自动执行此操作的编辑器。例如,对于Atom编辑器,可以使用go-plus插件。
至于代码,我认为你应该按照以下方式重构它:
package main
import (
"fmt"
"io"
"net/http"
"os"
"path/filepath"
"sync"
)
func downloadFile(filePath string) error {
resp, err := http.Get(filePath)
if err != nil {
return err
}
defer resp.Body.Close()
name := filepath.Base(filePath)
file, err := os.Create(name)
if err != nil {
return err
}
defer file.Close()
size, err := io.Copy(file, resp.Body)
if err != nil {
return err
}
fmt.Println(name, size, resp.Status)
return nil
}
func main() {
var wg sync.WaitGroup
fileList := []string{
"http://i.imgur.com/dxGb2uZ.jpg",
"http://i.imgur.com/RSU6NxX.jpg",
"http://i.imgur.com/hUWgS2S.jpg",
"http://i.imgur.com/U8kaix0.jpg",
"http://i.imgur.com/w3cEYpY.jpg",
"http://i.imgur.com/ooSCD9T.jpg"}
fmt.Println("downloading", len(fileList), "files")
for _, url := range fileList {
wg.Add(1)
go func(url string) {
err := downloadFile(url)
if err != nil {
fmt.Println("[error]", url, err)
}
wg.Done()
}(url)
}
wg.Wait()
}
我不喜欢在函数之间传递WaitGroup
,而更喜欢保持函数简单、阻塞和顺序执行,然后在更高层次上组合并发。这样,你可以选择按顺序执行所有操作,而无需更改downloadFile
函数。
我还添加了错误处理并修复了变量名,使其符合驼峰命名法。
英文:
As Tim Cooper said you need to pass the WaitGroup
as a pointer. If you run the go vet
tool on your code it will give you this warning:
$ go vet ex.go
ex.go:12: download_file passes Lock by value: sync.WaitGroup contains sync.Mutex
exit status 1
I recommend using an editor that can do this for you when you save a file. For example go-plus for Atom.
As for the code I think you should restructure it like this:
package main
import (
"fmt"
"io"
"net/http"
"os"
"path/filepath"
"sync"
)
func downloadFile(filePath string) error {
resp, err := http.Get(filePath)
if err != nil {
return err
}
defer resp.Body.Close()
name := filepath.Base(filePath)
file, err := os.Create(name)
if err != nil {
return err
}
defer file.Close()
size, err := io.Copy(file, resp.Body)
if err != nil {
return err
}
fmt.Println(name, size, resp.Status)
return nil
}
func main() {
var wg sync.WaitGroup
fileList := []string{
"http://i.imgur.com/dxGb2uZ.jpg",
"http://i.imgur.com/RSU6NxX.jpg",
"http://i.imgur.com/hUWgS2S.jpg",
"http://i.imgur.com/U8kaix0.jpg",
"http://i.imgur.com/w3cEYpY.jpg",
"http://i.imgur.com/ooSCD9T.jpg"}
fmt.Println("downloading", len(fileList), "files")
for _, url := range fileList {
wg.Add(1)
go func(url string) {
err := downloadFile(url)
if err != nil {
fmt.Println("[error]", url, err)
}
wg.Done()
}(url)
}
wg.Wait()
}
I don't like passing WaitGroup
s around and prefer to keep functions simple, blocking and sequential and then stitch together the concurrency at a higher level. This gives you the option of doing it all sequentially without having to change downloadFile
.
I also added error handling and fixed names so they are camelCase.
答案2
得分: 1
除了Calab的回答之外,你的方法没有任何问题,你只需要将指向sync.WaitGroup
的指针传递给它即可。
func download_file(file_path string, wg *sync.WaitGroup) {
defer wg.Done()
......
}
......
go download_file(url, &wg)
.....
英文:
Adding to Calab's response, there's absolutely nothing wrong with your approach, all you had to do is to pass a pointer to the sync.WaitGroup
.
func download_file(file_path string, wg *sync.WaitGroup) {
defer wg.Done()
......
}
.....
go download_file(url, &wg)
.....
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论