Golang的WaitGroup导致内存泄漏,我该如何改进这个函数?

huangapple go评论79阅读模式
英文:

Golang WaitGroup causing memory leak, what can I do to improve this function

问题

我一直在努力寻找我们应用程序中的内存泄漏问题,并使用pprof工具来了解发生了什么。

当我查看堆时,我经常看到以下函数,但我不明白为什么(或者是否)它实际上是一个问题。

func CreateClients(raw []byte) bool {

    macs := []string{}
    conn := FormatConn(raw)

    if conn.Ap_Mac != "" {

        var wg sync.WaitGroup
        var array []Client

        c1 := make(chan Client)

        clients := FormatClients(conn)

        wg.Add(len(clients))

        for _, c := range clients {
            go func(d Client) {
                defer wg.Done()
                c1 <- UpdateClients(d)
            }(c)
        }

        go func() {
            defer wg.Done()
            for {
                select {
                case client := <-c1:
                    array = append(array, client)
                    macs = append(macs, client.Client_Mac)
                }
            }
        }()

        wg.Wait()
        // Do some other stuff
        ...
    }
}

UpdateClients函数在Mongo中更新客户端模型。当它返回时,我需要每个客户端 - 这样我就可以用ES索引它,还需要一个macs数组来做其他一些事情。

我已经查看了在线示例,并认为这是循环遍历通道的推荐方式。

我的pprof堆看起来像这样,并在几天内稳定增长:

7.53MB of 9.53MB total (79.00%)
Dropped 234 nodes (cum <= 0.05MB)
Showing top 5 nodes out of 28 (cum >= 1MB)
      flat  flat%   sum%        cum   cum%
       2MB 21.00% 21.00%        2MB 21.00%  strings.Replace
    1.51MB 15.89% 36.89%     1.51MB 15.89%  github.com/PolkaSpots/worker/worker.func·006
    1.51MB 15.87% 52.76%     1.51MB 15.87%  github.com/PolkaSpots/worker/worker.func·008
    1.50MB 15.75% 68.51%     1.50MB 15.75%  newproc_m
       1MB 10.50% 79.00%        1MB 10.50%  gopkg.in/mgo.v2/bson.(*decoder).readStr

有没有更高效/推荐的方法来实现这个?

英文:

I've been struggling to find a memory leak in our application and have been using the pprof tool to understand what's going on.

When I look at the heap, I constantly see the following function and I don't understand why (or if) it's actually a problem.

func CreateClients(raw []byte) bool {

	macs := []string{}
	conn := FormatConn(raw)

	if conn.Ap_Mac != &quot;&quot; {

		var wg sync.WaitGroup
		var array []Client

		c1 := make(chan Client)

		clients := FormatClients(conn)

		wg.Add(len(clients))

		for _, c := range clients {
			go func(d Client) {
				defer wg.Done()
				c1 &lt;- UpdateClients(d)
			}(c)
		}

		go func() {
            defer wg.Done()
			for {
				select {
				case client := &lt;-c1:
					array = append(array, client)
					macs = append(macs, client.Client_Mac)
				}
			}
		}()

		wg.Wait()
        // Do some other stuff
    ...
}

The UpdateClients function updates the client model in Mongo. When it returns, I need each client - so I can index it with ES plus I need an array of the macs to do some other stuff.

I've gone through the online examples and thought this was the recommended way to loop through a channel.

My pprof heap looks like this, and grows steadily over a few days:

7.53MB of 9.53MB total (79.00%)
Dropped 234 nodes (cum &lt;= 0.05MB)
Showing top 5 nodes out of 28 (cum &gt;= 1MB)
      flat  flat%   sum%        cum   cum%
       2MB 21.00% 21.00%        2MB 21.00%  strings.Replace
    1.51MB 15.89% 36.89%     1.51MB 15.89%  github.com/PolkaSpots/worker/worker.func&#183;006
    1.51MB 15.87% 52.76%     1.51MB 15.87%  github.com/PolkaSpots/worker/worker.func&#183;008
    1.50MB 15.75% 68.51%     1.50MB 15.75%  newproc_m
       1MB 10.50% 79.00%        1MB 10.50%  gopkg.in/mgo.v2/bson.(*decoder).readStr

Is there a more efficient / recommended way to achieve this?

EDIT_

As suggested, I've altered the loop as so

	done := make(chan bool)

	go func() {
		for {
			select {
			case client := &lt;-c1:
				array = append(array, client)
				macs = append(macs, client.Client_Mac)
			case &lt;-done:
				return
			}
		}
	}()

	wg.Wait()
	close(done)

答案1

得分: 4

接收循环永远不会中断:

for {
    select {
    case client := <-c1:
        ...
    }
}

它没有停止条件,没有超时,什么都没有 - 所以它将永远挂在那里 - 即使整个函数退出。它将泄漏goroutine和通道。

此外,当此循环退出时,您延迟了wg.Done,但您没有进行wg.Add以匹配它。因此,如果此循环退出,将会引发panic。

您需要做的是找到一种方法来停止for/select循环。我认为最简单的方法是在wg.Wait()之后添加一个接收数据的第二个通道,但在该goroutine中不要执行wg.Done()

英文:

The receive loop never breaks:

  for {
            select {
            case client := &lt;-c1:
            ...
   }

It has no stop condition, no timeout, nothing - so it will just hang there forever - even if your whole function exits. and it will leak both the goroutine and the channel.

On top of that, you're deferring a wg.Done when this loop exits, but you're not doing wg.Add to match it. So if this loop ever exits, you will panic.

What you need to do is find some way to stop the for/select loop. Simplest way IMO - add a second channel that receives data after wg.Wait(), but do not do wg.Done() in that goroutine.

huangapple
  • 本文由 发表于 2015年8月11日 21:11:08
  • 转载请务必保留本文链接:https://go.coder-hub.com/31942770.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定