which fields in memstats struct refer only to heap, only to stack

Go runtime has a lot of different variables related to heap and stack and some of the stack numbers are part of the heap numbers, leading to confusion (for me). For example, in this link. it says

// Stack numbers are part of the heap numbers, separate those out for user consumption
	stats.StackSys = stats.StackInuse
	stats.HeapInuse -= stats.StackInuse
	stats.HeapSys -= stats.StackInuse

In the runtime docs (excerpt below), it gives 7 different heap related fields (that is, fields of the memstat struct) without clearly explaining which ones include stack, and similarly, which stack fields are included in heap, and how that relates to total allocations.

this is a problem, because I want to compare heap against stack, but I don't want to choose a heap variable that includes stack (obviously).

Questions
1). Does total allocation field include heap, stack or both?
2) which heap fields do not include numbers stack?
3) which heap fields include numbers for stack?
4) which stack fields do not include numbers for heap?

  Alloc      uint64 // bytes allocated and still in use
        TotalAlloc uint64 // bytes allocated (even if freed)
        Sys        uint64 // bytes obtained from system (sum of XxxSys below)
        Lookups    uint64 // number of pointer lookups
        Mallocs    uint64 // number of mallocs
        Frees      uint64 // number of frees

        // Main allocation heap statistics.
        HeapAlloc    uint64 // bytes allocated and still in use
        HeapSys      uint64 // bytes obtained from system
        HeapIdle     uint64 // bytes in idle spans
        HeapInuse    uint64 // bytes in non-idle span
        HeapReleased uint64 // bytes released to the OS
        HeapObjects  uint64 // total number of allocated objects

        // Low-level fixed-size structure allocator statistics.
        //	Inuse is bytes used now.
        //	Sys is bytes obtained from system.
        StackInuse  uint64 // bytes used by stack allocator
        StackSys    uint64

These questions are a little hard to answer because the goroutine stacks are allocated from the heap. Go does not have the clear separation between stack and heap that exists in C.

Does total allocation field include heap, stack or both?

The TotalAlloc field of the MemStats struct includes all memory that the Go runtime has requested from the OS for the Go heap. It does not include memory allocated for goroutine stacks. Initially I thought it did, but I was wrong. Sorry for the confusion. I hope this answer is more accurate.

(To be precise, I should mention that in a program that uses cgo every thread (not goroutine--there are normally more goroutines than threads) will have a stack allocated by the OS; that stack is not allocated by the Go runtime and is not counted in TotalAlloc. It is only used by cgo calls.)

which heap fields do not include numbers stack?
which heap fields include numbers for stack?

These fields include numbers for goroutine stacks: HeapIdle, HeapReleased.

These fields do not include numbers for goroutine stacks: HeapAlloc, HeapInUse, HeapObjects.

The HeapSys field does not include memory used by currently active goroutine stacks, but does include memory for goroutine stacks that were once in use but were then freed.

which stack fields do not include numbers for heap?

I don't know how to answer this question in a way that makes sense. The stack fields report information specifically about goroutine stacks.

From running (variations of) a test program and peeking at the Go source, I'm seeing:

• Alloc and TotalAlloc appear to cover only non-stack allocations. Allocating big locals did not add their size to TotalAlloc, even when it caused stacks to grow.

• If memory is currently reserved for a goroutine's stack it counts in the StackX vars and not the HeapX vars. This is the subtraction you found in the source. It also implies anything that allocates space for stacks can reduce HeapSys and HeapIdle but leave HeapInuse alone.

  • Because you asked: stack fields never include heap allocations--stacks come from the heap but not vice versa.
  • Variables you think are on the heap (ssp := new(SomeStruct)) might in fact be stack-allocated if escape analysis can determine they don't outlive the function call. This is almost always useful to you, because those vars can be freed at function exit without generating garbage for the GC. Don't worry about this too much.

• Once a goroutine quits, its stack space can be returned to the heap. (If its stack is small, though, it's likely be cached to be reused as a future goroutine's stack.) Then it will not show up as stack space and may show up as available heap space again. I'm seeing this both empirically and in the Go source (proc.c's gfput calling runtime·stackfree). That means that exiting goroutines or old stacks being returned after stack growth can appear to grow HeapSys and HeapIdle, but it's really just space shifting between uses.

• There appears to be no TotalAlloc-style running counter covering all pages ever allocated for stacks. If a goroutine's stack is freed and reused it only gets counted once.

• There is definitely no TotalAlloc-style running counter covering all stack-allocated variables ever. That would involve tracking overhead per function call.

Stack related problems are relatively rare, because stack-allocated variables are freed on function return, and large stacks themselves are freed on goroutine exit. They can happen, like if goroutines are leaking (never exiting even as you create new ones), or if you make huge stack allocations (var bigLocal [1e7]uint64, or ridiculous deep recursion) in goroutines that don't exit. But it's a lot more common to have trouble with the heap, since stuff on the heap isn't freed until GC (tools like the standard Pool help you recycle heap-allocated items to delay the need for GC).

So, practically speaking, I would mostly keep an eye on Alloc and TotalAlloc for heap overuse, and if somehow stack space becomes a problem, look at the stack numbers (and maybe check for unexpectedly many running goroutines).

These observations are implementation-specific (I'm looking at go 1.4, not tip), and I am not an expert on the Go source code, so take as what they are. That test program, for reference:

package main
import (
"fmt"
"reflect"
"runtime"
"sync"
)
var g []byte
func usesHeap() {
g = make([]byte, 1000)
}
func usesTempStack() {
var l [1000]byte
_ = l
}
func createsGoroutineAndWaits() {
wg := new(sync.WaitGroup)
wg.Add(1)
go func() {
usesTempStack()
wg.Done()
}()
wg.Wait()
}
func createsGoroutine() {
go usesTempStack()
}
func recurse(depth int, max int) {
var l [1024]byte
_ = l
if depth < max {
recurse(depth+1, max)
}
}
func growsTheStack() {
recurse(0, 1000)
}
func checkUsageOf(lbl string, f func(), before, after *runtime.MemStats) {
_ = new(sync.WaitGroup)
runtime.ReadMemStats(before)
// using own goroutine so everyone starts w/the same stack size
wg := new(sync.WaitGroup)
wg.Add(1)
// request GC in hopes of a fair start
runtime.GC()
go func() {
runtime.ReadMemStats(before)
for i := 0; i < 1000; i++ {
f()
}
runtime.Gosched()
runtime.ReadMemStats(after)
wg.Done()
}()
wg.Wait()
fmt.Println("Results for", lbl, "\n")
beforeVal, afterVal := reflect.ValueOf(*before), reflect.ValueOf(*after)
memStatsType := beforeVal.Type()
fieldCount := memStatsType.NumField()
for i := 0; i < fieldCount; i++ {
field := memStatsType.Field(i)
if field.Type.Kind() != reflect.Uint64 {
continue
}
beforeStat, afterStat := int64(beforeVal.Field(i).Uint()), int64(afterVal.Field(i).Uint())
if beforeStat == afterStat {
continue
}
fmt.Println(field.Name, "differs by", afterStat-beforeStat)
}
fmt.Println("\n")
}
func main() {
before, after := new(runtime.MemStats), new(runtime.MemStats)
checkUsageOf("growsTheStack", growsTheStack, before, after)
checkUsageOf("createsGoroutineAndWaits", createsGoroutine, before, after)
checkUsageOf("usesHeap", usesHeap, before, after)
checkUsageOf("usesTempStack", usesTempStack, before, after)
checkUsageOf("createsGoroutine", createsGoroutine, before, after)
}