Measure heap growth accurately

I am trying to measure the evolution of the number of heap-allocated objects before and after I call a function. I am forcing runtime.GC() and using runtime.ReadMemStats to measure the number of heap objects I have before and after.

The problem I have is that I sometimes see unexpected heap growth. And it is different after each run.

A simple example below, where I would always expect to see a zero heap-objects growth.

https://go.dev/play/p/FBWfXQHClaG

var mem1_before, mem2_before, mem1_after, mem2_after runtime.MemStats

func measure_nothing(before, after *runtime.MemStats) {
	runtime.GC()
	runtime.ReadMemStats(before)

	runtime.GC()
	runtime.ReadMemStats(after)
}

func main() {
	measure_nothing(&mem1_before, &mem1_after)
	measure_nothing(&mem2_before, &mem2_after)

	log.Printf("HeapObjects diff = %d", int64(mem1_after.HeapObjects-mem1_before.HeapObjects))
	log.Printf("HeapAlloc diff %d", int64(mem1_after.HeapAlloc-mem1_before.HeapAlloc))

	log.Printf("HeapObjects diff = %d", int64(mem2_after.HeapObjects-mem2_before.HeapObjects))
	log.Printf("HeapAlloc diff %d", int64(mem2_after.HeapAlloc-mem2_before.HeapAlloc))
}

Sample output:

2009/11/10 23:00:00 HeapObjects diff = 0
2009/11/10 23:00:00 HeapAlloc diff 0
2009/11/10 23:00:00 HeapObjects diff = 4
2009/11/10 23:00:00 HeapAlloc diff 1864

Is what I'm trying to do unpractical? I assume the runtime is doing things that allocate/free heap-memory. Can I tell it to stop to make my measurements? (this is for a test checking for memory leaks, not production code)

You can't predict what garbage collection and reading all the memory stats require in the background. Calling those to calculate memory allocations and usage is not a reliable way.

Luckily for us, Go's testing framework can monitor and calculate memory usage.

So what you should do is write a benchmark function and let the testing framework do its job to report memory allocations and usage.

Let's assume we want to measure this foo() function:

var x []int64

func foo(allocs, size int) {
	for i := 0; i < allocs; i++ {
		x = make([]int64, size)
	}
}

All it does is allocate a slice of the given size, and it does this with the given number of times (allocs).

Let's write benchmarking functions for different scenarios:

func BenchmarkFoo_0_0(b *testing.B) {
	for i := 0; i < b.N; i++ {
		foo(0, 0)
	}
}

func BenchmarkFoo_1_1(b *testing.B) {
	for i := 0; i < b.N; i++ {
		foo(1, 1)
	}
}

func BenchmarkFoo_2_2(b *testing.B) {
	for i := 0; i < b.N; i++ {
		foo(2, 2)
	}
}

Running the benchmark with go test -bench . -benchmem, the output is:

BenchmarkFoo_0_0-8   1000000000      0.3204 ns/op    0 B/op	   0 allocs/op
BenchmarkFoo_1_1-8   67101626	    16.58 ns/op	     8 B/op	   1 allocs/op
BenchmarkFoo_2_2-8   27375050	    42.42 ns/op	    32 B/op	   2 allocs/op

As you can see, the allocations per function call is the same what we pass as the allocs argument. The allocated memory is the expected allocs * size * 8 bytes.

Note that the reported allocations per op is an integer value (it's the result of an integer division), so if the benchmarked function only occasionally allocates, it might not be reported in the integer result. For details, see https://stackoverflow.com/questions/56255211/output-from-benchmem/56256892#56256892.

Like in this example:

var x []int64

func bar() {
	if rand.Float64() < 0.3 {
		x = make([]int64, 10)
	}
}

This bar() function does 1 allocation with 30% probability (and none with 70% probability), which means on average it does 0.3 allocations. Benchmarking it:

func BenchmarkBar(b *testing.B) {
    for i := 0; i < b.N; i++ {
        bar()
    }
}

Output is:

BenchmarkBar-8   38514928	    29.60 ns/op	   24 B/op	   0 allocs/op

We can see there is 24 bytes allocation (0.3 * 10 * 8 bytes), which is correct, but the reported allocations per op is 0.

Luckily for us, we can also benchmark a function from our main app using the testing.Benchmark() function. It returns a testing.BenchmarkResult including all details about memory usage. We have access to the total number of allocations and to the number of iterations, so we can calculate allocations per op using floating point numbers:

func main() {
	rand.Seed(time.Now().UnixNano())

	tr := testing.Benchmark(BenchmarkBar)
	fmt.Println("Allocs/op", tr.AllocsPerOp())
	fmt.Println("B/op", tr.AllocedBytesPerOp())

	fmt.Println("Precise allocs/op:", float64(tr.MemAllocs)/float64(tr.N))
}

This will output:

Allocs/op 0
B/op 24
Precise allocs/op: 0.3000516369276302

We can see the expected ~0.3 allocations per op.

Now if we go ahead and benchmark your measure_nothing() function:

func BenchmarkNothing(b *testing.B) {
	for i := 0; i < b.N; i++ {
		measure_nothing(&mem1_before, &mem1_after)
	}
}

We get this output:

Allocs/op 0
B/op 11
Precise allocs/op: 0.12182030338389732

As you can see, running the garbage collector twice and reading memory stats twice occasionally needs allocation (~1 out of 10 calls: 0.12 times on average).