在for循环中使用goroutine导致了意外的行为。

huangapple
go评论111阅读模式
英文:

Goroutine in for loop causes unexpected behavior

问题

我正在为你翻译以下内容：

我正在完成《Go之旅》中的Web爬虫练习。

我尝试使用并发的互斥锁（Mutex）来解决问题，参考了这里找到的一个解决方案。我对其进行了修改，以适应原始问题中的预定义签名。然而，在URL树的第二层时，爬虫停止了。在调试过程中，打印语句的不同行为完全让我困惑了：

  1. 	var done sync.WaitGroup
  2. 	for _, u := range urls {
  3. 		done.Add(1)
  4. 		fmt.Printf("enter: %s\n", u) // 这里
  5. 		go func(url string) {
  6. 			defer done.Done()
  7. 			Crawl(u, depth-1, fetcher, f)
  8. 		}(u)
  9. 	}
  10. 	done.Wait()

如果我将打印语句放在goroutine之外，输出是符合预期的。但我不知道为什么会停在那里。

  1. enter: https://golang.org/pkg/
  2. enter: https://golang.org/cmd/

但是，如果我将打印语句放在goroutine内部，即

  1. 	var done sync.WaitGroup
  2. 	for _, u := range urls {
  3. 		done.Add(1)
  4. 		go func(url string) {
  5. 			defer done.Done()
  6. 			fmt.Printf("enter: %s\n", u) // 这里
  7. 			Crawl(u, depth-1, fetcher, f)
  8. 		}(u)
  9. 	}
  10. 	done.Wait()

输出变成了

  1. enter: https://golang.org/cmd/
  2. enter: https://golang.org/cmd/

我有两个问题：

  1. 在第二种情况下，为什么会打印两次 enter: https://golang.org/cmd/？
  2. 为什么Crawl函数会在出现错误时停止，而不是继续遍历URL树？

PS：第二个问题可能与第一个问题有关。我故意在goroutine内部将 u 改为 url，以重现困扰我的错误。

以下是我修改后的解决方案：

  1. package main
  3. import (
  4. 	"fmt"
  5. 	"sync"
  6. )
  8. type Fetcher interface {
  9. 	// Fetch返回URL的内容和在该页面上找到的URL切片。
  10. 	Fetch(url string) (body string, urls []string, err error)
  11. }
  13. type fetchState struct {
  14. 	mu      sync.Mutex
  15. 	fetched map[string]bool
  16. }
  18. // Crawl使用fetcher递归地爬取以url为起点的页面，最大深度为depth。
  19. func Crawl(url string, depth int, fetcher Fetcher, f *fetchState) {
  20. 	// TODO：并行获取URL。
  21. 	// TODO：不要重复获取相同的URL。
  22. 	// 这个实现两者都没有做到：
  23. 	f.mu.Lock()
  24. 	already := f.fetched[url]
  25. 	f.fetched[url] = true
  26. 	f.mu.Unlock()
  28. 	if already {
  29. 		return
  30. 	}
  32. 	if depth <= 0 {
  33. 		return
  34. 	}
  35. 	body, urls, err := fetcher.Fetch(url)
  36. 	if err != nil {
  37. 		fmt.Println(err)
  38. 		return
  39. 	}
  40. 	fmt.Printf("found: %s %q\n", url, body)
  42. 	var done sync.WaitGroup
  43. 	for _, u := range urls {
  44. 		done.Add(1)
  45. 		go func(url string) {
  46. 			defer done.Done()
  47. 			fmt.Printf("enter: %s\n", u)
  48. 			Crawl(u, depth-1, fetcher, f)
  49. 		}(u)
  50. 	}
  51. 	done.Wait()
  53. 	return
  54. }
  56. func makeState() *fetchState {
  57. 	f := &fetchState{}
  58. 	f.fetched = make(map[string]bool)
  59. 	return f
  60. }
  62. func main() {
  63. 	Crawl("https://golang.org/", 4, fetcher, makeState())
  64. }
  66. // fakeFetcher是一个返回预定义结果的Fetcher。
  67. type fakeFetcher map[string]*fakeResult
  69. type fakeResult struct {
  70. 	body string
  71. 	urls []string
  72. }
  74. func (f fakeFetcher) Fetch(url string) (string, []string, error) {
  75. 	if res, ok := f[url]; ok {
  76. 		return res.body, res.urls, nil
  77. 	}
  78. 	return "", nil, fmt.Errorf("not found: %s", url)
  79. }
  81. // fetcher是一个填充了预定义结果的fakeFetcher。
  82. var fetcher = fakeFetcher{
  83. 	"https://golang.org/": &fakeResult{
  84. 		"The Go Programming Language",
  85. 		[]string{
  86. 			"https://golang.org/pkg/",
  87. 			"https://golang.org/cmd/",
  88. 		},
  89. 	},
  90. 	"https://golang.org/pkg/": &fakeResult{
  91. 		"Packages",
  92. 		[]string{
  93. 			"https://golang.org/",
  94. 			"https://golang.org/cmd/",
  95. 			"https://golang.org/pkg/fmt/",
  96. 			"https://golang.org/pkg/os/",
  97. 		},
  98. 	},
  99. 	"https://golang.org/pkg/fmt/": &fakeResult{
  100. 		"Package fmt",
  101. 		[]string{
  102. 			"https://golang.org/",
  103. 			"https://golang.org/pkg/",
  104. 		},
  105. 	},
  106. 	"https://golang.org/pkg/os/": &fakeResult{
  107. 		"Package os",
  108. 		[]string{
  109. 			"https://golang.org/",
  110. 			"https://golang.org/pkg/",
  111. 		},
  112. 	},
  113. }
英文:

I was doing the Web Crawler Exercise in A Tour of Go.

I was trying to use concurrent Mutex to solve the question, based on a solution found here. I modified it to fit the pre-defined signatures in the original question. However, the crawler stops at the second level of the URL tree. During debugging, the different behaviors of the print statements completely confused me:

  1. 	var done sync.WaitGroup
  2. 	for _, u := range urls {
  3. 		done.Add(1)
  4. 		fmt.Printf(&quot;enter: %s\n&quot;, u) // here
  5. 		go func(url string) {
  6. 			defer done.Done()
  7. 			Crawl(u, depth-1, fetcher, f)
  8. 		}(u)
  9. 	}
  10. 	done.Wait()

If I put the print statement outside the goroutine, outputs are expected. But I didn't know why it stops there.

  1. enter: https://golang.org/pkg/
  2. enter: https://golang.org/cmd/

But if I put the print statement inside the goroutine, that is

  1. 	var done sync.WaitGroup
  2. for _, u := range urls {
  3. done.Add(1)
  4. go func(url string) {
  5. defer done.Done()
  6. fmt.Printf(&quot;enter: %s\n&quot;, u) // here
  7. Crawl(u, depth-1, fetcher, f)
  8. }(u)
  9. }
  10. done.Wait()

The output becomes

  1. enter: https://golang.org/cmd/
  2. enter: https://golang.org/cmd/

I have two questions:

  1. In the second case, why enter: https://golang.org/cmd/ gets printed twice?
  2. Why does the Crawl function stop at an error, instead of keeping traversing the URL tree?

PS: the second question might be related to the first one. I intentionally made u instead of url inside the goroutine to reproduce the bug that confused me.

Below is my modified solution

  1. package main
  3. import (
  4. 	&quot;fmt&quot;
  5. 	&quot;sync&quot;
  6. )
  8. type Fetcher interface {
  9. 	// Fetch returns the body of URL and
  10. 	// a slice of URLs found on that page.
  11. 	Fetch(url string) (body string, urls []string, err error)
  12. }
  14. type fetchState struct {
  15. 	mu sync.Mutex
  16. 	fetched map[string]bool
  17. }
  19. // Crawl uses fetcher to recursively crawl
  20. // pages starting with url, to a maximum of depth.
  21. func Crawl(url string, depth int, fetcher Fetcher, f *fetchState) {
  22. 	// TODO: Fetch URLs in parallel.
  23. 	// TODO: Don&#39;t fetch the same URL twice.
  24. 	// This implementation doesn&#39;t do either:
  25. 	f.mu.Lock()
  26. 	already := f.fetched[url]
  27. 	f.fetched[url] = true
  28. 	f.mu.Unlock()
  29. 	
  30. 	if already {
  31. 		return
  32. 	}
  33. 	
  34. 	if depth &lt;= 0 {
  35. 		return
  36. 	}
  37. 	body, urls, err := fetcher.Fetch(url)
  38. 	if err != nil {
  39. 		fmt.Println(err)
  40. 		return
  41. 	}
  42. 	fmt.Printf(&quot;found: %s %q\n&quot;, url, body)
  43. 	
  44. 	var done sync.WaitGroup
  45. 	for _, u := range urls {
  46. 		done.Add(1)
  47. 		go func(url string) {
  48. 			defer done.Done()
  49. 			fmt.Printf(&quot;enter: %s\n&quot;, u)
  50. 			Crawl(u, depth-1, fetcher, f)
  51. 		}(u)
  52. 	}
  53. 	done.Wait()
  54. 	
  55. 	return
  56. }
  58. func makeState() *fetchState{
  59. 	f := &amp;fetchState{}
  60. 	f.fetched = make(map[string]bool)
  61. 	return f
  62. }
  64. func main() {
  65. 	Crawl(&quot;https://golang.org/&quot;, 4, fetcher, makeState())
  66. }
  68. // fakeFetcher is Fetcher that returns canned results.
  69. type fakeFetcher map[string]*fakeResult
  71. type fakeResult struct {
  72. 	body string
  73. 	urls []string
  74. }
  76. func (f fakeFetcher) Fetch(url string) (string, []string, error) {
  77. 	if res, ok := f[url]; ok {
  78. 		return res.body, res.urls, nil
  79. 	}
  80. 	return &quot;&quot;, nil, fmt.Errorf(&quot;not found: %s&quot;, url)
  81. }
  83. // fetcher is a populated fakeFetcher.
  84. var fetcher = fakeFetcher{
  85. 	&quot;https://golang.org/&quot;: &amp;fakeResult{
  86. 		&quot;The Go Programming Language&quot;,
  87. 		[]string{
  88. 			&quot;https://golang.org/pkg/&quot;,
  89. 			&quot;https://golang.org/cmd/&quot;,
  90. 		},
  91. 	},
  92. 	&quot;https://golang.org/pkg/&quot;: &amp;fakeResult{
  93. 		&quot;Packages&quot;,
  94. 		[]string{
  95. 			&quot;https://golang.org/&quot;,
  96. 			&quot;https://golang.org/cmd/&quot;,
  97. 			&quot;https://golang.org/pkg/fmt/&quot;,
  98. 			&quot;https://golang.org/pkg/os/&quot;,
  99. 		},
  100. 	},
  101. 	&quot;https://golang.org/pkg/fmt/&quot;: &amp;fakeResult{
  102. 		&quot;Package fmt&quot;,
  103. 		[]string{
  104. 			&quot;https://golang.org/&quot;,
  105. 			&quot;https://golang.org/pkg/&quot;,
  106. 		},
  107. 	},
  108. 	&quot;https://golang.org/pkg/os/&quot;: &amp;fakeResult{
  109. 		&quot;Package os&quot;,
  110. 		[]string{
  111. 			&quot;https://golang.org/&quot;,
  112. 			&quot;https://golang.org/pkg/&quot;,
  113. 		},
  114. 	},
  115. }

答案1

得分: 2

欢迎来到Stack Overflow！

在你的函数中，你将url定义为参数，但在其中一直使用u。循环变量u被函数字面量捕获。

尝试这样做：

  1. var done sync.WaitGroup
  2. for _, u := range urls {
  3.     done.Add(1)
  4.     go func(url string) {
  5.         defer done.Done()
  6.         fmt.Printf("enter: %s\n", url)  // <- 检查区别
  7.         Crawl(url, depth-1, fetcher, f) // <- 检查区别
  8.     }(u)
  9. }
  10. done.Wait()

关于为什么u变量打印相同的值，这是一个非常常见的错误：https://github.com/golang/go/wiki/CommonMistakes#using-goroutines-on-loop-iterator-variables

简而言之，Go语言通过引用将单个变量传递给goroutine。当它们执行时，它们可能会在其中找到迭代的最后一个值。

我找到了一篇详细解释的好文章：https://eli.thegreenplace.net/2019/go-internals-capturing-loop-variables-in-closures/

英文:

Welcome to Stack Overflow!

In you function, you defined url as the parameter, but kept using u inside of it.
The loop variable u captured by func literal.

Try doing this:

  1. 	var done sync.WaitGroup
  2. for _, u := range urls {
  3. done.Add(1)
  4. go func(url string) {
  5. defer done.Done()
  6. fmt.Printf(&quot;enter: %s\n&quot;, url)  // &lt;- check the difference
  7. Crawl(url, depth-1, fetcher, f) // &lt;- check the difference
  8. }(u)
  9. }
  10. done.Wait()

For why the same value was being printed with the u variable, this is a very common mistake: https://github.com/golang/go/wiki/CommonMistakes#using-goroutines-on-loop-iterator-variables

In short, the go is passing a single variable by reference to the goroutines. When they execute, they are probably going to find the last value of the iteration in it.

I found this neat article that explains it in detail: https://eli.thegreenplace.net/2019/go-internals-capturing-loop-variables-in-closures/

huangapple
  • 本文由 发表于 2022年8月26日 05:47:39
  • 转载请务必保留本文链接：https://go.coder-hub.com/73493998.html
  • concurrency
  • go
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定