将WaitGroup传递给函数会改变其行为,为什么?

huangapple go评论65阅读模式
英文:

Passing a WaitGroup to a function changes behavior, why?

问题

我有3个归并排序的实现:

MergeSort:简单的实现,没有并发;

MergeSortSmart:通过有限的缓冲通道大小限制并发。如果缓冲区已满,则调用简单的实现;

MergeSortSmartBug:与前一个实现相同的策略,但通过将wg指针传递给一个函数来减少代码重复。

前两个按预期工作,但第三个返回一个空切片而不是排序后的输入。我无法理解发生了什么,并且也找不到答案。

这是代码的playground链接:https://play.golang.org/p/DU1ypbanpVi

为什么Bug版本返回一个空切片?如何重构Smart版本而不是连续进行两次选择?

很抱歉我无法在一个更小的示例中重现这种行为。

英文:

I have 3 merge sort implementations:

MergeSort: simple one without concurrency;

MergeSortSmart: with concurrency limited by buffered channel size limit. If buffer is full, calls the simple implementation;

MergeSortSmartBug: same strategy as the previous one, but with a small "refactor", passing wg pointer to a function reducing code duplication.

The first two works as expected, but the third one returns an empty slice instead of the sorted input. I couldn't understand what happened and found no answers as well.

Here is the playground link for the code: https://play.golang.org/p/DU1ypbanpVi

package main

import (
	"fmt"
	"math/rand"
	"runtime"
	"sync"
)

type pass struct{}

var semaphore = make(chan pass, runtime.NumCPU())

func main() {
	rand.Seed(10)
	s := make([]int, 16)
	for i := 0; i < 16; i++ {
		s[i] = int(rand.Int31n(1000))
	}

	fmt.Println(s)
	fmt.Println(MergeSort(s))
	fmt.Println(MergeSortSmart(s))
	fmt.Println(MergeSortSmartBug(s))
}

func merge(l, r []int) []int {
	tmp := make([]int, 0, len(l)+len(r))
	for len(l) > 0 || len(r) > 0 {
		if len(l) == 0 {
			return append(tmp, r...)
		}
		if len(r) == 0 {
			return append(tmp, l...)
		}
		if l[0] <= r[0] {
			tmp = append(tmp, l[0])
			l = l[1:]
		} else {
			tmp = append(tmp, r[0])
			r = r[1:]
		}
	}
	return tmp
}

func MergeSort(s []int) []int {
	if len(s) <= 1 {
		return s
	}

	n := len(s) / 2

	l := MergeSort(s[:n])
	r := MergeSort(s[n:])

	return merge(l, r)
}

func MergeSortSmart(s []int) []int {
	if len(s) <= 1 {
		return s
	}

	n := len(s) / 2

	var wg sync.WaitGroup
	wg.Add(2)

	var l, r []int
	select {
	case semaphore <- pass{}:
		go func() {
			l = MergeSortSmart(s[:n])
			<-semaphore
			wg.Done()
		}()
	default:
		l = MergeSort(s[:n])
		wg.Done()
	}

	select {
	case semaphore <- pass{}:
		go func() {
			r = MergeSortSmart(s[n:])
			<-semaphore
			wg.Done()
		}()
	default:
		r = MergeSort(s[n:])
		wg.Done()
	}

	wg.Wait()
	return merge(l, r)
}

func MergeSortSmartBug(s []int) []int {
	if len(s) <= 1 {
		return s
	}

	n := len(s) / 2

	var wg sync.WaitGroup
	wg.Add(2)

	l := mergeSmart(s[:n], &wg)
	r := mergeSmart(s[n:], &wg)

	wg.Wait()
	return merge(l, r)
}

func mergeSmart(s []int, wg *sync.WaitGroup) []int {
	var tmp []int
	select {
	case semaphore <- pass{}:
		go func() {
			tmp = MergeSortSmartBug(s)
			<-semaphore
			wg.Done()
		}()
	default:
		tmp = MergeSort(s)
		wg.Done()
	}
	return tmp
}

Why does the Bug version returns an empty slice? How can I refactor the Smart version without doing two selects one after the other?

Sorry for I couldn't reproduce this behavior in a smaller example.

答案1

得分: 2

问题不在于WaitGroup本身,而在于你的并发处理方式。你的mergeSmart函数启动了一个go例程,并在等待go例程完成之前就返回了tmp变量。
你可以尝试使用以下更类似的模式:

leftchan := make(chan []int)
rightchan := make(chan []int)
go mergeSmart(s[:n], leftchan)
go mergeSmart(s[n:], rightchan)
l := <-leftchan
r := <-rightchan

或者如果顺序不重要的话,你可以使用一个单一的通道。

英文:

The problem is not with the WaitGroup itself. It's with your concurrency handling. Your mergeSmart function lunches a go routine and returns the tmp variable without waiting for the go routine to finish.
You might want to try a pattern more like this:

leftchan := make(chan []int)
rightchan := make(chan []int)
go mergeSmart(s[:n], leftchan)
go mergeSmart(s[n:], rightchan)
l := &lt;-leftchan
r := &lt;-rightchan

Or you can use a single channel if order doesn't matter.

答案2

得分: 1

mergeSmart不等待wg,所以它返回一个尚未接收到值的tmp。你可以通过将目标切片的引用传递给函数,而不是返回一个切片来修复它。

英文:

mergeSmart doesn't wait on the wg, so it returns a tmp that hasn't received a value yet. You could probably repair it by passing a reference to the destination slice in to the function, instead of returning a slice.

答案3

得分: 1

看一下mergeSmart函数。当选择进入第一个情况时,启动了一个goroutine并立即返回了tmp(一个空数组)。在这种情况下,无法获取正确的值。(在这里可以看到高级调试打印信息https://play.golang.org/p/IedaY3muso2)
也许可以通过引用传递预先分配的数组?

英文:

Look at the mergeSmart function. When the select enter into the first case, the goroutine is launched and imediatly returns tmp (which is an empty array). In that case there is no way to get the right value. (See advanced debugging prints here https://play.golang.org/p/IedaY3muso2)
Maybe passing arrays preallocated by reference?

答案4

得分: 0

我实现了两个建议(通过引用传递切片和使用通道),结果(工作中!)在这里:https://play.golang.org/p/DcDC_-NjjAH

package main

import (
	"fmt"
	"math/rand"
	"runtime"
	"sync"
)

type pass struct{}

var semaphore = make(chan pass, runtime.NumCPU())

func main() {
	rand.Seed(10)
	s := make([]int, 16)
	for i := 0; i < 16; i++ {
		s[i] = int(rand.Int31n(1000))
	}

	fmt.Println(s)
	fmt.Println(MergeSort(s))
	fmt.Println(MergeSortSmart(s))
	fmt.Println(MergeSortSmartPointer(s))
	fmt.Println(MergeSortSmartChan(s))
}

func merge(l, r []int) []int {
	tmp := make([]int, 0, len(l)+len(r))
	for len(l) > 0 || len(r) > 0 {
		if len(l) == 0 {
			return append(tmp, r...)
		}
		if len(r) == 0 {
			return append(tmp, l...)
		}
		if l[0] <= r[0] {
			tmp = append(tmp, l[0])
			l = l[1:]
		} else {
			tmp = append(tmp, r[0])
			r = r[1:]
		}
	}
	return tmp
}

func MergeSort(s []int) []int {
	if len(s) <= 1 {
		return s
	}

	n := len(s) / 2

	l := MergeSort(s[:n])
	r := MergeSort(s[n:])

	return merge(l, r)
}

func MergeSortSmart(s []int) []int {
	if len(s) <= 1 {
		return s
	}

	n := len(s) / 2

	var wg sync.WaitGroup
	wg.Add(2)

	var l, r []int
	select {
	case semaphore <- pass{}:
		go func() {
			l = MergeSortSmart(s[:n])
			<-semaphore
			wg.Done()
		}()
	default:
		l = MergeSort(s[:n])
		wg.Done()
	}

	select {
	case semaphore <- pass{}:
		go func() {
			r = MergeSortSmart(s[n:])
			<-semaphore
			wg.Done()
		}()
	default:
		r = MergeSort(s[n:])
		wg.Done()
	}

	wg.Wait()
	return merge(l, r)
}

func MergeSortSmartPointer(s []int) []int {
	if len(s) <= 1 {
		return s
	}
	n := len(s) / 2
	var l, r []int

	var wg sync.WaitGroup
	wg.Add(2)

	mergeSmartPointer(&l, s[:n], &wg)
	mergeSmartPointer(&r, s[n:], &wg)

	wg.Wait()
	return merge(l, r)
}

func mergeSmartPointer(tmp *[]int, s []int, wg *sync.WaitGroup) {
	select {
	case semaphore <- pass{}:
		go func() {
			*tmp = MergeSortSmartPointer(s)
			<-semaphore
			wg.Done()
		}()
	default:
		*tmp = MergeSort(s)
		wg.Done()
	}
}

func MergeSortSmartChan(s []int) []int {
	if len(s) <= 1 {
		return s
	}
	n := len(s) / 2

	lchan := make(chan []int)
	rchan := make(chan []int)

	go mergeSmartChan(s[:n], lchan)
	go mergeSmartChan(s[n:], rchan)

	l := <-lchan
	r := <-rchan

	return merge(l, r)
}

func mergeSmartChan(s []int, c chan []int) {
	select {
	case semaphore <- pass{}:
		go func() {
			c <- MergeSortSmartChan(s)
			<-semaphore
		}()
	default:
		c <- MergeSort(s)
	}
}

我完全理解了我做错了什么,谢谢!以后参考一下,这是对排序一个包含100,000个元素的切片的基准测试结果:

$ go test -bench=.
goos: linux
goarch: amd64
cpu: Intel(R) Core(TM) i5-9300H CPU @ 2.40GHz
BenchmarkMergeSort-8               	      97	  12230309 ns/op
BenchmarkMergeSortSmart-8          	     181	   7209844 ns/op
BenchmarkMergeSortSmartPointer-8   	     163	   7483136 ns/op
BenchmarkMergeSortSmartChan-8      	     156	   8149585 ns/op
英文:

I implemented both suggestions (passing slice by reference and using channels) and the (working!) result is here: https://play.golang.org/p/DcDC_-NjjAH

package main
import (
&quot;fmt&quot;
&quot;math/rand&quot;
&quot;runtime&quot;
&quot;sync&quot;
)
type pass struct{}
var semaphore = make(chan pass, runtime.NumCPU())
func main() {
rand.Seed(10)
s := make([]int, 16)
for i := 0; i &lt; 16; i++ {
s[i] = int(rand.Int31n(1000))
}
fmt.Println(s)
fmt.Println(MergeSort(s))
fmt.Println(MergeSortSmart(s))
fmt.Println(MergeSortSmartPointer(s))
fmt.Println(MergeSortSmartChan(s))
}
func merge(l, r []int) []int {
tmp := make([]int, 0, len(l)+len(r))
for len(l) &gt; 0 || len(r) &gt; 0 {
if len(l) == 0 {
return append(tmp, r...)
}
if len(r) == 0 {
return append(tmp, l...)
}
if l[0] &lt;= r[0] {
tmp = append(tmp, l[0])
l = l[1:]
} else {
tmp = append(tmp, r[0])
r = r[1:]
}
}
return tmp
}
func MergeSort(s []int) []int {
if len(s) &lt;= 1 {
return s
}
n := len(s) / 2
l := MergeSort(s[:n])
r := MergeSort(s[n:])
return merge(l, r)
}
func MergeSortSmart(s []int) []int {
if len(s) &lt;= 1 {
return s
}
n := len(s) / 2
var wg sync.WaitGroup
wg.Add(2)
var l, r []int
select {
case semaphore &lt;- pass{}:
go func() {
l = MergeSortSmart(s[:n])
&lt;-semaphore
wg.Done()
}()
default:
l = MergeSort(s[:n])
wg.Done()
}
select {
case semaphore &lt;- pass{}:
go func() {
r = MergeSortSmart(s[n:])
&lt;-semaphore
wg.Done()
}()
default:
r = MergeSort(s[n:])
wg.Done()
}
wg.Wait()
return merge(l, r)
}
func MergeSortSmartPointer(s []int) []int {
if len(s) &lt;= 1 {
return s
}
n := len(s) / 2
var l, r []int
var wg sync.WaitGroup
wg.Add(2)
mergeSmartPointer(&amp;l, s[:n], &amp;wg)
mergeSmartPointer(&amp;r, s[n:], &amp;wg)
wg.Wait()
return merge(l, r)
}
func mergeSmartPointer(tmp *[]int, s []int, wg *sync.WaitGroup) {
select {
case semaphore &lt;- pass{}:
go func() {
*tmp = MergeSortSmartPointer(s)
&lt;-semaphore
wg.Done()
}()
default:
*tmp = MergeSort(s)
wg.Done()
}
}
func MergeSortSmartChan(s []int) []int {
if len(s) &lt;= 1 {
return s
}
n := len(s) / 2
lchan := make(chan []int)
rchan := make(chan []int)
go mergeSmartChan(s[:n], lchan)
go mergeSmartChan(s[n:], rchan)
l := &lt;-lchan
r := &lt;-rchan
return merge(l, r)
}
func mergeSmartChan(s []int, c chan []int) {
select {
case semaphore &lt;- pass{}:
go func() {
c &lt;- MergeSortSmartChan(s)
&lt;-semaphore
}()
default:
c &lt;- MergeSort(s)
}
}

I understood 100% what I was doing wrong, thanks!
And for future references, here's the benchmark of sorting a slice of 100,000 elems:

$ go test -bench=.
goos: linux
goarch: amd64
cpu: Intel(R) Core(TM) i5-9300H CPU @ 2.40GHz
BenchmarkMergeSort-8               	      97	  12230309 ns/op
BenchmarkMergeSortSmart-8          	     181	   7209844 ns/op
BenchmarkMergeSortSmartPointer-8   	     163	   7483136 ns/op
BenchmarkMergeSortSmartChan-8      	     156	   8149585 ns/op

huangapple
  • 本文由 发表于 2021年7月18日 04:37:26
  • 转载请务必保留本文链接:https://go.coder-hub.com/68424103.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定