Why does my Go program's memory fluctuate so much?

I have a Go program that allocates lots of maps and slices. Generally a lot of usage, allocation overhead etc. I run it, it loads a lot of data in, and then I query it with a web service.

After I leave it running, when it's read in all its data and isn't doing any queries (i.e should be stable) I see memory fluctuations. Recently it's reported: 5.42 GB, 5.01 GB and 4.3 GB of real memory. That's a massive fluctuation.

I have about 150 million objects (slices hanging off the main hashtable). That's a lot of little objects. I expect a little fluctuation (although I would never expect memory to increase when no new objects are being allocated and the main thread/s is blocking on a socket).

Possible explanations are

• the overhead of lots of small allocations just multiplies any natural fluctuation
• some code is allocating objects (although I can't see how)
• the Go GC is doing its own paging (?)
• I'm using Mac OS, and it's at fault somehow

Is this amount of fluctuation normal / expected?

The go-runtime doesn't immediately release unused memory to the OS (it might be needed again soon).
So looking at the OS-level, you see only a part of the overall picture.
Using http://golang.org/pkg/runtime/#ReadMemStats you can see another part of the picture.

pkg/runtime/malloc.goc shows the freelist, and pkg/runtime/mgc0.c shows the garbage collector.

If memory-usage goes down in a stable situation that seems normal, after loading finishes, you can force a GC, and you might want to print the Memstats regularly for more info.

The fluctuations are likely due to the amount of garbage your program is creating that the garbage collector has to eventually collect. The frequency of the fluctuation is going to depend on how much / how often you are creating garbage and when the garbage collector collects it.

Whether a variable is allocated to the stack or the heap is determined by the compiler. Generally pointers, maps, and slices can be allocated to the heap, but this only happens if the compiler's escape analysis determines that a variable escapes. Anything that is allocated to the heap will need to be garbage collected.

Even though Go handles the stack vs heap details, creating as little garbage as possible can be of great benefit. You can read about an extreme case where the garbage collector paused for 10 seconds. The Go garbage collector isn't perfect, but it is improving. The more it improves less you'll have to worry about it. But you should at least be aware of it.

You can run the following to determine what the compiler will allocate to the heap:

go build -gcflags=-m program.go

You may be surprised by what actually gets allocated to the heap. For example, even if you use a bytes.Buffer locally it still gets allocated to the heap, due to bytes.Buffer.buf being re-sliced. Regardless of whether that's supposed to happen or not, there may be situations where you think you're not creating any garbage, but in reality you are.