使用多个goroutine时的Go内存消耗

huangapple go评论89阅读模式
英文:

Go memory consumption with many goroutines

问题

我正在尝试检查使用100,000个goroutine的情况下Go的性能。我编写了一个简单的程序来生成这么多的例程,它们只是打印一些公告。我将MaxStack大小限制为仅为512字节。但我注意到程序的大小并没有减小。它消耗了大约460 MB的内存,因此每个goroutine大约为4 KB。我的问题是,我们是否可以将最大堆栈大小设置为低于goroutine的“最小”堆栈大小(可能为4 KB)。我们如何设置Goroutine启动时的最小堆栈大小?
以下是我用于测试的示例代码:

package main

import "fmt"
import "time"

import "runtime/debug"

func main() {
	fmt.Printf("%v\n", debug.SetMaxStack(512))
	var i int
	for i = 0; i < 100000; i++ {
		go func(x int) {
			for {
				time.Sleep(10 * time.Millisecond)
				//fmt.Printf("I am %v\n", x)
			}
		}(i)
	}
	fmt.Println("Done")
	time.Sleep(999999999999)
}
英文:

I was trying to check how Go will perform with 100,000 goroutines. I wrote a simple program to spawn that many routines which does nothing but print some announcements. I restricted the MaxStack size to just 512 bytes. But what I noticed was the program size doesn't decrease with that. It was consuming around 460 MB of memory and hence around 4 KB per goroutine. My question is, can we set max stack size lower than the "minimum" stack size (which may be 4 KB) for the goroutines. How can we set the minimum Stack size that Goroutine starts with ?
Below is sample code I used for the test:

package main

import &quot;fmt&quot;
import &quot;time&quot;

import &quot;runtime/debug&quot;

func main() {
	fmt.Printf(&quot;%v\n&quot;, debug.SetMaxStack(512))
	var i int
	for i = 0; i &lt; 100000; i++ {
		go func(x int) {
			for {
				time.Sleep(10 * time.Millisecond)
				//fmt.Printf(&quot;I am %v\n&quot;, x)
			}
		}(i)
	}
	fmt.Println(&quot;Done&quot;)
	time.Sleep(999999999999)
}

答案1

得分: 12

目前还没有办法设置goroutine的最小堆栈大小。

Go 1.2将最小大小从4KB增加到8KB。

文档中提到:
“在Go 1.2中,创建goroutine时的最小堆栈大小从4KB增加到8KB。许多程序在旧的大小下遇到性能问题,这往往会在性能关键的部分引入昂贵的堆栈段切换。新的数字是通过经验测试确定的。”

但他们接着说:

“更新:增加的最小堆栈大小可能导致具有许多goroutine的程序使用更多内存。目前没有解决方法,但未来的版本计划包括新的堆栈管理技术,应该能更好地解决这个问题。”

所以你将来可能会有更好的运气。

更多信息请参见http://golang.org/doc/go1.2#stack_size。

英文:

There's currently no way to set the minimum stack size for goroutines.

Go 1.2 increased the minimum size from 4KB to 8KB

The docs say:
>"In Go 1.2, the minimum size of the stack when a goroutine is created has been lifted from 4KB to 8KB. Many programs were suffering performance problems with the old size, which had a tendency to introduce expensive stack-segment switching in performance-critical sections. The new number was determined by empirical testing."

But they go on to say:

>"Updating: The increased minimum stack size may cause programs with many goroutines to use more memory. There is no workaround, but plans for future releases include new stack management technology that should address the problem better."

So you may have more luck in the future.

See http://golang.org/doc/go1.2#stack_size for more info.

答案2

得分: 10

runtime/debug.SetMaxStack函数只确定了Go语言在何时认为程序无限递归并终止它。设置它的值非常低并不会改变栈的最小大小,只会通过在任何一个栈的使用大小超过限制时导致程序崩溃来限制栈的最大大小。

从技术上讲,只有在需要扩展栈时才会发生崩溃,因此当一个栈需要超过8KB(或者在Go 1.2之前是4KB)时,你的程序将会终止。

你的程序之所以使用至少4KB * nGoroutines的最小大小是因为栈是按页对齐的,所以在一个虚拟内存页上永远只能有一个栈。因此,你的程序将至少使用nGoroutines个页面,并且操作系统通常只以页面大小的增量来测量和分配内存。

改变栈的起始(最小)大小的唯一方法是修改并重新编译Go运行时(可能还包括编译器)。

Go 1.3将包括连续栈,这通常比Go 1.2及之前的分割栈更快,并且在未来可能还会导致更小的初始栈。

英文:

The runtime/debug.SetMaxStack function only determines a what point does go consider a program infinitely recursive, and terminate it. http://golang.org/pkg/runtime/debug/#SetMaxStack

Setting it absurdly low does nothing to the minimum size of stacks, and only limits the maximum size by virtue of your program crashing when any stack's in-use size exceeds the limit.

Technically the crash only happens when the stack must be grown, so your program will die when a stack needs more than 8KB (or 4KB prior to go 1.2).

The reason why your program uses a minimum of 4KB * nGoroutines is because stacks are page-aligned, so there can never be more than one stack on a VM page. Therefore your program will use at least nGoroutines worth of pages, and OSes usually only measure and allocate memory in page-sized increments.

The only way to change the starting (minimum) size of a stack is to modify and recompile the go runtime (and possibly the compiler too).

Go 1.3 will include contiguous stacks, which are generally faster than the split stacks in Go 1.2 and earlier, and which may also lead to smaller initial stacks in the future.

答案3

得分: 5

只是一个注意事项:在Go 1.4中,goroutine堆栈的最小大小从8Kb减少到2Kb。

而根据Go 1.13的说明,它仍然是相同的-https://github.com/golang/go/blob/bbd25d26c0a86660fb3968137f16e74837b7a9c6/src/runtime/stack.go#L72:

// Go代码使用的堆栈的最小大小
_StackMin = 2048

英文:

Just a note: in Go 1.4: the minimum size of the goroutine stack has decreased from 8Kb to 2Kb.

And as per Go 1.13, it's still same -https://github.com/golang/go/blob/bbd25d26c0a86660fb3968137f16e74837b7a9c6/src/runtime/stack.go#L72:

// The minimum size of stack used by Go code
_StackMin = 2048

huangapple
  • 本文由 发表于 2014年3月11日 21:12:02
  • 转载请务必保留本文链接:https://go.coder-hub.com/22326765.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定