Go内存消耗管理

huangapple go评论99阅读模式
英文:

Go memory consumption management

问题

我是Go的新手,正在努力弄清楚它如何管理内存消耗。

我在一个测试项目中遇到了内存问题。我不明白为什么当我的程序运行很长时间时,Go会越来越多地使用内存(从不释放)。

我正在运行下面提供的测试用例。在第一次分配之后,程序使用了近350MB的内存(根据ActivityMonitor)。然后我尝试释放它,ActivityMonitor显示内存消耗翻倍。为什么会这样?

我在OS X上使用Go 1.0.3运行此代码。

这段代码有什么问题?在Go程序中管理大型变量的正确方法是什么?

当我实现一个使用大量时间和内存的算法时,我遇到了另一个与内存管理相关的问题;在运行一段时间后,它会抛出一个“内存不足”的异常。

package main

import (
    "fmt"
    "time"
)

func main() {
    fmt.Println("获取内存")
    tmp := make([]uint32, 100000000)
    for kk, _ := range tmp {
        tmp[kk] = 0
    }
    time.Sleep(5 * time.Second)
    fmt.Println("释放内存")
    tmp = make([]uint32, 1)
    tmp = nil
    time.Sleep(5 * time.Second)
    fmt.Println("获取内存")
    tmp = make([]uint32, 100000000)
    for kk, _ := range tmp {
        tmp[kk] = 0
    }
    time.Sleep(5 * time.Second)
    fmt.Println("释放内存")
    tmp = make([]uint32, 1)
    tmp = nil
    time.Sleep(5 * time.Second)
    return
}
英文:

I am new to Go and trying to figure out how it manages memory consumption.

I have trouble with memory in one of my test projects. I don't understand why Go uses more and more memory (never freeing it) when my program runs for a long time.

I am running the test case provided below. After the first allocation, program uses nearly 350 MB of memory (according to ActivityMonitor). Then I try to free it and ActivityMonitor shows that memory consumption doubles. Why?

I am running this code on OS X using Go 1.0.3.

What is wrong with this code? And what is the right way to manage large variables in Go programs?

I had another memory-management-related problem when implementing an algorithm that uses a lot of time and memory; after running it for some time it throws an "out of memory" exception.

package main

import ("fmt" 
"time"
)

func main() {
  fmt.Println("getting memory")
  tmp := make([]uint32, 100000000)
  for kk, _ := range tmp {
    tmp[kk] = 0
  }
  time.Sleep(5 * time.Second)
  fmt.Println("returning memory")
  tmp = make([]uint32, 1)
  tmp = nil
  time.Sleep(5 * time.Second)
  fmt.Println("getting memory")
  tmp = make([]uint32, 100000000)
  for kk, _ := range tmp {
    tmp[kk] = 0
  }
  time.Sleep(5 * time.Second)
  fmt.Println("returning memory")
  tmp = make([]uint32, 1)
  tmp = nil
  time.Sleep(5 * time.Second)  
  return
}

答案1

得分: 36

目前,Go使用了一种标记-清除垃圾收集器,一般情况下不会定义对象何时被丢弃。

然而,如果你仔细观察,会发现有一个名为sysmon的Go例程,它基本上会在程序运行期间一直运行,并定期调用垃圾收集器:

// forcegcperiod是两次垃圾收集之间的最大时间间隔(以纳秒为单位)。
// 如果超过这个时间没有进行垃圾收集,就会强制运行一次。
//
// 这是一个用于测试目的的变量。通常不会更改它。
var forcegcperiod int64 = 2 * 60 * 1e9

(...)

// 如果一个堆段在垃圾收集后5分钟没有使用,我们将其归还给操作系统。
scavengelimit := int64(5 * 60 * 1e9)

forcegcperiod决定了强制调用垃圾收集器的时间间隔。scavengelimit决定了何时将堆段返回给操作系统。堆段是一些内存页,可以容纳多个对象。它们会保留scavengelimit时间,如果没有对象在其上并且超过了scavengelimit,则会被释放。

在代码的后面部分,你可以看到有一个跟踪选项。你可以使用它来查看,每当清理程序认为需要清理时:

$ GOGCTRACE=1 go run gc.go
gc1(1): 0+0+0 ms 0 -> 0 MB 423 -> 350 (424-74) objects 0 handoff
gc2(1): 0+0+0 ms 1 -> 0 MB 2664 -> 1437 (2880-1443) objects 0 handoff
gc3(1): 0+0+0 ms 1 -> 0 MB 4117 -> 2213 (5712-3499) objects 0 handoff
gc4(1): 0+0+0 ms 2 -> 1 MB 3128 -> 2257 (6761-4504) objects 0 handoff
gc5(1): 0+0+0 ms 2 -> 0 MB 8892 -> 2531 (13734-11203) objects 0 handoff
gc6(1): 0+0+0 ms 1 -> 1 MB 8715 -> 2689 (20173-17484) objects 0 handoff
gc7(1): 0+0+0 ms 2 -> 1 MB 5231 -> 2406 (22878-20472) objects 0 handoff
gc1(1): 0+0+0 ms 0 -> 0 MB 172 -> 137 (173-36) objects 0 handoff
getting memory
gc2(1): 0+0+0 ms 381 -> 381 MB 203 -> 202 (248-46) objects 0 handoff
returning memory
getting memory
returning memory

如你所见,在获取和返回之间没有进行垃圾收集。然而,如果你将延迟从5秒更改为3分钟(超过了forcegcperiod的2分钟),则对象将被垃圾收集器移除:

returning memory
scvg0: inuse: 1, idle: 1, sys: 3, released: 0, consumed: 3 (MB)
scvg0: inuse: 381, idle: 0, sys: 382, released: 0, consumed: 382 (MB)
scvg1: inuse: 1, idle: 1, sys: 3, released: 0, consumed: 3 (MB)
scvg1: inuse: 381, idle: 0, sys: 382, released: 0, consumed: 382 (MB)
gc9(1): 1+0+0 ms 1 -> 1 MB 4485 -> 2562 (26531-23969) objects 0 handoff
gc10(1): 1+0+0 ms 1 -> 1 MB 2563 -> 2561 (26532-23971) objects 0 handoff
scvg2: GC forced // forcegc(2分钟)超过了
scvg2: inuse: 1, idle: 1, sys: 3, released: 0, consumed: 3 (MB)
gc3(1): 0+0+0 ms 381 -> 381 MB 206 -> 206 (252-46) objects 0 handoff
scvg2: GC forced
scvg2: inuse: 381, idle: 0, sys: 382, released: 0, consumed: 382 (MB)
getting memory

内存仍然没有被释放,但是垃圾收集器将内存区域标记为未使用。当使用的堆段未使用且年龄超过limit时,释放将开始。从清理程序代码中可以看到:

if(s->unusedsince != 0 && (now - s->unusedsince) > limit) {
	// ...
	runtime·SysUnused((void*)(s->start << PageShift), s->npages << PageShift);
}

当然,这种行为可能会随着时间的推移而改变,但我希望你现在对于何时强制丢弃对象和何时不强制有了一些了解。

正如zupa指出的,释放对象可能不会将内存返回给操作系统,因此在某些系统上,你可能看不到内存使用情况的变化。根据golang-nuts上的这个帖子,这似乎是Plan 9和Windows的情况。

英文:

Currently, go uses a mark-and-sweep garbage collector, which in general does not define when the object is thrown away.

However, if you look closely, there is a go routine called sysmon which essentially runs as long as your program does and calls the GC periodically:

// forcegcperiod is the maximum time in nanoseconds between garbage
// collections. If we go this long without a garbage collection, one
// is forced to run.
//
// This is a variable for testing purposes. It normally doesn't change.
var forcegcperiod int64 = 2 * 60 * 1e9

(...)

// If a heap span goes unused for 5 minutes after a garbage collection,
// we hand it back to the operating system.
scavengelimit := int64(5 * 60 * 1e9)

forcegcperiod determines the period after which the GC is called by force. scavengelimit determines when spans are returned to the operating system. Spans are a number of memory pages which can hold several objects. They're kept for scavengelimit time and are freed if no object is on them and scavengelimit is exceeded.

Further down in the code you can see that there is a trace option. You can use this to see, whenever the
scavenger thinks he needs to clean up:

$ GOGCTRACE=1 go run gc.go
gc1(1): 0+0+0 ms 0 -> 0 MB 423 -> 350 (424-74) objects 0 handoff
gc2(1): 0+0+0 ms 1 -> 0 MB 2664 -> 1437 (2880-1443) objects 0 handoff
gc3(1): 0+0+0 ms 1 -> 0 MB 4117 -> 2213 (5712-3499) objects 0 handoff
gc4(1): 0+0+0 ms 2 -> 1 MB 3128 -> 2257 (6761-4504) objects 0 handoff
gc5(1): 0+0+0 ms 2 -> 0 MB 8892 -> 2531 (13734-11203) objects 0 handoff
gc6(1): 0+0+0 ms 1 -> 1 MB 8715 -> 2689 (20173-17484) objects 0 handoff
gc7(1): 0+0+0 ms 2 -> 1 MB 5231 -> 2406 (22878-20472) objects 0 handoff
gc1(1): 0+0+0 ms 0 -> 0 MB 172 -> 137 (173-36) objects 0 handoff
getting memory
gc2(1): 0+0+0 ms 381 -> 381 MB 203 -> 202 (248-46) objects 0 handoff
returning memory
getting memory
returning memory

As you can see, no gc invoke is done between getting and returning. However, if you change
the delay from 5 seconds to 3 minutes (more than the 2 minutes from forcegcperiod),
the objects are removed by the gc:

returning memory
scvg0: inuse: 1, idle: 1, sys: 3, released: 0, consumed: 3 (MB)
scvg0: inuse: 381, idle: 0, sys: 382, released: 0, consumed: 382 (MB)
scvg1: inuse: 1, idle: 1, sys: 3, released: 0, consumed: 3 (MB)
scvg1: inuse: 381, idle: 0, sys: 382, released: 0, consumed: 382 (MB)
gc9(1): 1+0+0 ms 1 -> 1 MB 4485 -> 2562 (26531-23969) objects 0 handoff
gc10(1): 1+0+0 ms 1 -> 1 MB 2563 -> 2561 (26532-23971) objects 0 handoff
scvg2: GC forced // forcegc (2 minutes) exceeded
scvg2: inuse: 1, idle: 1, sys: 3, released: 0, consumed: 3 (MB)
gc3(1): 0+0+0 ms 381 -> 381 MB 206 -> 206 (252-46) objects 0 handoff
scvg2: GC forced
scvg2: inuse: 381, idle: 0, sys: 382, released: 0, consumed: 382 (MB)
getting memory

The memory is still not freed, but the GC marked the memory region as unused. Freeing will begin when
the used span is unused and older than limit. From scavenger code:

if(s->unusedsince != 0 && (now - s->unusedsince) > limit) {
	// ...
	runtime·SysUnused((void*)(s->start << PageShift), s->npages << PageShift);
}

This behavior may of course change over time, but I hope you now get a bit of a feel when objects
are thrown away by force and when not.

As pointed out by zupa, releasing objects may not return the memory to the operating system, so on
certain systems you may not see a change in memory usage. This seems to be the case for Plan 9
and Windows according to this thread on golang-nuts.

答案2

得分: 15

要最终(强制)回收未使用的内存,您必须调用runtime.GC()

variable = nil可能会使某些对象变得不可访问,从而有资格进行回收,但它本身并不释放任何东西。

英文:

To eventually (force) collect unused memory you must call runtime.GC().

variable = nil may make things unreachable and thus eligible for collection, but it per se doesn't free anything.

huangapple
  • 本文由 发表于 2013年1月29日 20:05:04
  • 转载请务必保留本文链接:https://go.coder-hub.com/14582471.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定