英文:
Append not thread-safe?
问题
我注意到,如果我尝试在for
循环内使用goroutine来追加到一个切片中,会出现丢失/空白数据的情况:
destSlice := make([]myClass, 0)
var wg sync.WaitGroup
for _, myObject := range sourceSlice {
wg.Add(1)
go func(closureMyObject myClass) {
defer wg.Done()
var tmpObj myClass
tmpObj.AttributeName = closureMyObject.AttributeName
destSlice = append(destSlice, tmpObj)
}(myObject)
}
wg.Wait()
有时,当我打印destSlice
中的所有AttributeName
时,有些元素是空字符串(""
),而其他时候,sourceSlice
中的一些元素在destSlice
中不存在。
我的代码是否存在数据竞争问题?这是否意味着append
在多个goroutine并发使用时不是线程安全的?
英文:
I noticed that if I tried appending to a slice using goroutines inside a for
loop, there would be instances where I would get missing/blank data:
destSlice := make([]myClass, 0)
var wg sync.WaitGroup
for _, myObject := range sourceSlice {
wg.Add(1)
go func(closureMyObject myClass) {
defer wg.Done()
var tmpObj myClass
tmpObj.AttributeName = closureMyObject.AttributeName
destSlice = append(destSlice, tmpObj)
}(myObject)
}
wg.Wait()
Sometimes, when I print all AttributeName
s from destSlice
, some elements are empty strings (""
), and other times, some elements from sourceSlice
are not present in destSlice
.
Does my code have a data race, and does this mean that append
is not thread-safe for concurrent use by multiple goroutines?
答案1
得分: 55
在Go语言中,没有任何值可以安全地进行并发读写,切片(即slice headers)也不例外。
是的,你的代码存在数据竞争。可以使用-race
选项运行代码进行验证。
示例:
type myClass struct {
AttributeName string
}
sourceSlice := make([]myClass, 100)
destSlice := make([]myClass, 0)
var wg sync.WaitGroup
for _, myObject := range sourceSlice {
wg.Add(1)
go func(closureMyObject myClass) {
defer wg.Done()
var tmpObj myClass
tmpObj.AttributeName = closureMyObject.AttributeName
destSlice = append(destSlice, tmpObj)
}(myObject)
}
wg.Wait()
使用以下命令运行代码:
go run -race play.go
输出结果为:
==================
WARNING: DATA RACE
Read at 0x00c420074000 by goroutine 6:
main.main.func1()
/home/icza/gows/src/play/play.go:20 +0x69
Previous write at 0x00c420074000 by goroutine 5:
main.main.func1()
/home/icza/gows/src/play/play.go:20 +0x106
Goroutine 6 (running) created at:
main.main()
/home/icza/gows/src/play/play.go:21 +0x1cb
Goroutine 5 (running) created at:
main.main()
/home/icza/gows/src/play/play.go:21 +0x1cb
==================
==================
WARNING: DATA RACE
Read at 0x00c42007e000 by goroutine 6:
runtime.growslice()
/usr/local/go/src/runtime/slice.go:82 +0x0
main.main.func1()
/home/icza/gows/src/play/play.go:20 +0x1a7
Previous write at 0x00c42007e000 by goroutine 5:
main.main.func1()
/home/icza/gows/src/play/play.go:20 +0xc4
Goroutine 6 (running) created at:
main.main()
/home/icza/gows/src/play/play.go:21 +0x1cb
Goroutine 5 (running) created at:
main.main()
/home/icza/gows/src/play/play.go:21 +0x1cb
==================
==================
WARNING: DATA RACE
Write at 0x00c420098120 by goroutine 80:
main.main.func1()
/home/icza/gows/src/play/play.go:20 +0xc4
Previous write at 0x00c420098120 by goroutine 70:
main.main.func1()
/home/icza/gows/src/play/play.go:20 +0xc4
Goroutine 80 (running) created at:
main.main()
/home/icza/gows/src/play/play.go:21 +0x1cb
Goroutine 70 (running) created at:
main.main()
/home/icza/gows/src/play/play.go:21 +0x1cb
==================
Found 3 data race(s)
exit status 66
解决方法很简单,使用sync.Mutex
来保护对destSlice
值的写入:
var (
mu = &sync.Mutex{}
destSlice = make([]myClass, 0)
)
var wg sync.WaitGroup
for _, myObject := range sourceSlice {
wg.Add(1)
go func(closureMyObject myClass) {
defer wg.Done()
var tmpObj myClass
tmpObj.AttributeName = closureMyObject.AttributeName
mu.Lock()
destSlice = append(destSlice, tmpObj)
mu.Unlock()
}(myObject)
}
wg.Wait()
你也可以用其他方法解决这个问题,例如可以使用一个通道,在通道上发送要追加的值,并有一个指定的goroutine从该通道接收并进行追加操作。
还要注意,虽然切片头部不安全,但切片元素作为不同的变量,可以在没有同步的情况下并发写入(因为它们是不同的变量)。参见这里。
英文:
In Go no value is safe for concurrent read/write, slices (which are slice headers) are no exception.
Yes, your code has data races. Run with the -race
option to verify.
Example:
type myClass struct {
AttributeName string
}
sourceSlice := make([]myClass, 100)
destSlice := make([]myClass, 0)
var wg sync.WaitGroup
for _, myObject := range sourceSlice {
wg.Add(1)
go func(closureMyObject myClass) {
defer wg.Done()
var tmpObj myClass
tmpObj.AttributeName = closureMyObject.AttributeName
destSlice = append(destSlice, tmpObj)
}(myObject)
}
wg.Wait()
Running it with
go run -race play.go
Output is:
==================
WARNING: DATA RACE
Read at 0x00c420074000 by goroutine 6:
main.main.func1()
/home/icza/gows/src/play/play.go:20 +0x69
Previous write at 0x00c420074000 by goroutine 5:
main.main.func1()
/home/icza/gows/src/play/play.go:20 +0x106
Goroutine 6 (running) created at:
main.main()
/home/icza/gows/src/play/play.go:21 +0x1cb
Goroutine 5 (running) created at:
main.main()
/home/icza/gows/src/play/play.go:21 +0x1cb
==================
==================
WARNING: DATA RACE
Read at 0x00c42007e000 by goroutine 6:
runtime.growslice()
/usr/local/go/src/runtime/slice.go:82 +0x0
main.main.func1()
/home/icza/gows/src/play/play.go:20 +0x1a7
Previous write at 0x00c42007e000 by goroutine 5:
main.main.func1()
/home/icza/gows/src/play/play.go:20 +0xc4
Goroutine 6 (running) created at:
main.main()
/home/icza/gows/src/play/play.go:21 +0x1cb
Goroutine 5 (running) created at:
main.main()
/home/icza/gows/src/play/play.go:21 +0x1cb
==================
==================
WARNING: DATA RACE
Write at 0x00c420098120 by goroutine 80:
main.main.func1()
/home/icza/gows/src/play/play.go:20 +0xc4
Previous write at 0x00c420098120 by goroutine 70:
main.main.func1()
/home/icza/gows/src/play/play.go:20 +0xc4
Goroutine 80 (running) created at:
main.main()
/home/icza/gows/src/play/play.go:21 +0x1cb
Goroutine 70 (running) created at:
main.main()
/home/icza/gows/src/play/play.go:21 +0x1cb
==================
Found 3 data race(s)
exit status 66
Solution is simple, use a sync.Mutex
to protect writing the destSlice
value:
var (
mu = &sync.Mutex{}
destSlice = make([]myClass, 0)
)
var wg sync.WaitGroup
for _, myObject := range sourceSlice {
wg.Add(1)
go func(closureMyObject myClass) {
defer wg.Done()
var tmpObj myClass
tmpObj.AttributeName = closureMyObject.AttributeName
mu.Lock()
destSlice = append(destSlice, tmpObj)
mu.Unlock()
}(myObject)
}
wg.Wait()
You could also solve it in other ways, e.g. you could use a channel on which you'd send the value to be appended, and have a designated goroutine receiving from this channel and do the append.
Also note that while slice headers are not safe, slice elements act as different variables and different slice elements can be written concurrently without synchronization (because they are distinct variables). See https://stackoverflow.com/questions/49879322/can-i-concurrently-write-different-slice-elements/49879469#49879469
答案2
得分: 17
这是一个相当古老的问题,但有另一个小的改进可以帮助摆脱互斥锁。你可以使用索引来添加到数组中,每个Go协程将使用自己的索引。在这种情况下,不需要同步。
destSlice := make([]myClass, len(sourceSlice))
var wg sync.WaitGroup
for i, myObject := range sourceSlice {
wg.Add(1)
go func(idx int, closureMyObject myClass) {
defer wg.Done()
var tmpObj myClass
tmpObj.AttributeName = closureMyObject.AttributeName
destSlice[idx] = tmpObj
}(i, myObject)
}
wg.Wait()
这段代码中,我们创建了一个目标切片 destSlice
,长度与源切片 sourceSlice
相同。然后,我们使用 sync.WaitGroup
来等待所有的协程完成。
在每个循环迭代中,我们使用 go
关键字启动一个匿名函数作为一个新的协程。这个匿名函数接受两个参数:idx
表示索引,closureMyObject
表示当前迭代的对象。在匿名函数中,我们创建了一个临时对象 tmpObj
,并将闭包对象的 AttributeName
赋值给它。最后,我们将临时对象赋值给目标切片的对应索引位置。
通过使用每个协程自己的索引,我们避免了对目标切片的并发访问,从而避免了同步操作的需要。最后,我们使用 wg.Wait()
等待所有的协程完成。
英文:
It's quite an old question but there is another small improvement that helps to get rid of mutex. You can use index to add to array. Each go routine will use it's own index. In this case synchronization is not necessary.
destSlice := make([]myClass, len(sourceSlice))
var wg sync.WaitGroup
for i, myObject := range sourceSlice {
wg.Add(1)
go func(idx int, closureMyObject myClass) {
defer wg.Done()
var tmpObj myClass
tmpObj.AttributeName = closureMyObject.AttributeName
destSlice[idx] = tmpObj
}(i, myObject)
}
wg.Wait()
答案3
得分: 6
为了给这个问题提供一个更近期的解决方案,看起来Go语言发布了一个用于同步目的的新地图:
https://godoc.org/golang.org/x/sync/syncmap
英文:
To give a more recent solution to this problem, it looks like Go has released a new map for sync purposes:
答案4
得分: 6
问题已经回答了,但我最喜欢解决这个问题的方式是使用errgroup。文档中的一个示例正是这个问题,还增加了错误处理的部分。
下面是文档中示例的核心代码:
g, ctx := errgroup.WithContext(ctx)
searches := []Search{Web, Image, Video}
results := make([]Result, len(searches))
for i, search := range searches {
i, search := i, search // https://golang.org/doc/faq#closures_and_goroutines
g.Go(func() error {
result, err := search(ctx, query)
if err == nil {
results[i] = result
}
return err
})
}
if err := g.Wait(); err != nil {
return nil, err
}
return results, nil
希望对那些不熟悉errgroup包的人有所帮助。
英文:
Question has been answered, but my favorite way to solve this problem is with errgroup. One of the examples in the docs is this exact problem plus one nice addition the handling of errors.
Below is the meat of the example from the docs:
g, ctx := errgroup.WithContext(ctx)
searches := []Search{Web, Image, Video}
results := make([]Result, len(searches))
for i, search := range searches {
i, search := i, search // https://golang.org/doc/faq#closures_and_goroutines
g.Go(func() error {
result, err := search(ctx, query)
if err == nil {
results[i] = result
}
return err
})
}
if err := g.Wait(); err != nil {
return nil, err
}
return results, nil
Hope this is helpful to those who are not aware of the errgroup package.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论