Go协程返回的结果比实际结果少。

huangapple go评论84阅读模式
英文:

Go routine returns less results then the actual result

问题

我有一个循环,用于对给定的键进行哈希处理并返回结果,但是在结果方面,如果我有1500个URL列表进入循环,它永远不会返回1500个结果,而是始终返回少于1500个。

在下面的代码中,我做错了什么:

if len(URLLists) > 0 {
    var base = "https://example.com/query?="
    var wg sync.WaitGroup
    var mutex = sync.Mutex{}
    wg.Add(len(URLLists))
    for _, url := range URLLists {
        // wg.Add(1)  OR above for loop
        go func() {
            defer wg.Done()
            hmac := "HMAX_123"
            out := encoding.HexEncodeURL(hmac, url)
            final := base + out
            list := Lists{
                Old: url,
                New: final,
            }
            mutex.Lock()
            response.URL = append(response.URL, list)
            mutex.Unlock()
        }()
    }
    wg.Wait()
    jR, err := json.Marshal(response)
    if err != nil {
        w.Write([]byte(`{"success": false, "url" : ""}`))
    } else {
        w.Write(jR)
    }
    return
}

我尝试了两种方法来使用Add函数 - 一种是在循环内部逐个添加,另一种是在循环外部一次性添加。

我希望函数返回所有1500个URL列表,而不仅仅是随机的"700, 977, 1123"列表。

看起来,wg.Wait()没有等待所有的wg.Add被添加完成。

英文:

I have a LOOP that does the hashing inside for a given key and returns the result, but on Result, If I have 1500 list of URL that goes inside LOOP, it never returns the result of 1500, it returns always lesser then 1500.

What am I doing wrong in below:

if len(URLLists) > 0 {
	var base = "https://example.com/query?="
	var wg sync.WaitGroup
    var mutex = sync.Mutex{}
	wg.Add(len(URLLists))
	for _, url := range URLLists {
		// wg.Add(1)  OR above for loop
		go func() {
			defer wg.Done()
			hmac := "HMAX_123"
			out := encoding.HexEncodeURL(hmac, url)
			final := base + out
			list := Lists{
				Old: url,
				New: final,
			}
            mutex.Lock()
			response.URL = append(response.URL, list)
            mutex.Unlock()
		}()
	}
	wg.Wait()
	jR, err := json.Marshal(response)
	if err != nil {
		w.Write([]byte(`{"success": false, "url" : ""}`))
	} else {
		w.Write(jR)
	}
	return
}

I tried both method for Add - one inside loop by 1 and one outside loop by total length.

I want function to return all 1500 URL list and not just "700, 977, 1123" random list.

It looks like - wg.Wait() is not waiting for all the wg.Add - added

答案1

得分: 2

你这里有一个相当严重的竞态条件:

response.URL = append(response.URL, list)

如果你同时启动了多达1500个并发的Go协程,那么其中几百个协程将会同时执行这行代码。它们将不断地覆盖对数组的更改。

你需要使用sync.Mutex来保护将新数据插入到这个切片中,或者通过通道发送结果,并有一个单独的Go协程从该通道读取并追加到列表中。

英文:

You have a pretty serious race condition here:

response.URL = append(response.URL, list)

If you're starting as many as 1500 concurrent Go routines, you will have many hundreds of them all attempting to execute this line at once. They will be constantly overwriting changes to the array.

You need to protect the insertion of new data into this slice with a sync.Mutex, or be sending results over a channel and having a single Go routine reading from that channel and appending to the list.

答案2

得分: 2

这个程序中有几个错误:

1)你在goroutine中使用了循环变量。循环变量在每次迭代时都会被重写,所以当goroutine使用url时,它可能已经移动到下一个URL,因此你最终会得到多个goroutine对同一个URL进行哈希。修复方法如下:

for _, url := range URLLists {
    urlCopy := url // 创建url的副本
    // wg.Add(1) 或者放在上面的for循环之前

2)你有一个竞争条件。你必须保护对response.URL的访问,因为它被多个goroutine写入。你可以使用互斥锁:

lock := sync.Mutex{}
for _, url := ...
  ...
  lock.Lock()
  response.URL = append(response.URL, list)
  lock.Unlock()

更好的方法是通过通道发送它们。

英文:

There are a couple of errors in this program:

  1. You are using the loop variable inside the goroutine. The loop variable is rewritten at each iteration, so when the goroutine uses the url, it might have already moved on to the next URL, thus you end up with multiple goroutines hashing the same URL. To fix:
    for _, url := range URLLists {
        url:=url // Create a copy of the url
        // wg.Add(1)  OR above for loop
  1. You have a race condition. You have to protect access to response.URL because it is being written by multiple goroutines. You can use a mutex:
lock:=sync.Mutex{}
for _,url:=...
  ...
  lock.Lock()
  response.URL = append(response.URL, list)
  lock.Unlock()

A better way would be to send these over a channel.

huangapple
  • 本文由 发表于 2021年10月28日 09:22:05
  • 转载请务必保留本文链接:https://go.coder-hub.com/69747213.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定