In docker cpu usage calculation what are: TotalUsage, SystemUsage, PercpuUsage and what does the calculation means?

I see in docker sources for cpu calculation but I don't get what are exactly

1. TotalUsage
2. SystemUsage
3. PercpuUsage

why is cpuDelta divided by SystemDelta and also why then multiplied by PercpuUsage?

The documentation says for cpuDelta

> // calculate the change for the cpu usage of the container in between
> readings

and for systemDelta:

> // calculate the change for the entire system between readings

so shouldn't these delta's both be summed and divided by the total cpu (or total nanoseconds passed) during that delta time? after all the container was consuming both the cpuDelta and also systemDelta

Update: I think I get it the totalUsage is the container usage and the SystemUsage is not the systemCPU used by the container but the whole system beyond the container cpu this is why they are divided.

Below mention points will give an clear idea about CPU utilization calculation in docker:

1. Metric with "system" keyword always refer to the HOST metric on which docker is running.
2. Metric without the keyword "system" always refer to the metric related with
   CONTAINER running inside docker.
3. REMEMBER the percentage formula

4. SystemUsage = The CPU utilization of the HOST on which docker is running.This is equivalent to "WHOLE" in the percentage formula.

5. TotalUsage = The CPU utilization per core of the CONTAINER which is running inside docker.This is equivalent to "PART" in the percentage formula.

6. PercpuUsage = Individual utilization of CPU cores allocated to the CONTAINER.This also let us know the number of CPU cores allocated to the CONTAINER.

7. Change in CONTAINER CPU per core utilization

   = Previous value of CONTAINER CPU - Current value of CONTAINER CPU

8. Change in HOST CPU utilization

   = Previous value of HOST CPU - Current value of HOST CPU

9. Total change in CONTAINER CPU utilization= Change in CONTAINER CPU per core utilization * Number of core allocated to container[PercpuUsage core count]

10. CPU utilization(%)=(Total change in CONTAINER CPU utilization / Change in HOST CPU utilization) * 100

The CPU utilization code written for docker in golang makes it easier to understand the terminologies mentioned above:
https://github.com/moby/moby/blob/eb131c5383db8cac633919f82abad86c99bffbe5/cli/command/container/stats_helpers.go#L175-L188

func calculateCPUPercentUnix(previousCPU, previousSystem uint64, v *types.StatsJSON) float64 {
	var (
		cpuPercent = 0.0
		// calculate the change for the cpu usage of the container in between readings
		cpuDelta = float64(v.CPUStats.CPUUsage.TotalUsage) - float64(previousCPU)
		// calculate the change for the entire system between readings
		systemDelta = float64(v.CPUStats.SystemUsage) - float64(previousSystem)
	)

	if systemDelta > 0.0 && cpuDelta > 0.0 {
		cpuPercent = (cpuDelta / systemDelta) * float64(len(v.CPUStats.CPUUsage.PercpuUsage)) * 100.0
	}
	return cpuPercent
}

CPU stats from Docker stats API

 "cpu_stats": {
	"cpu_usage": {
	  "total_usage": 18730109057613,
	  "percpu_usage": [
		2037710936955,
		1481642385431,
		1344435820732,
		1163032744042,
		1204372068508,
		5758852306067,
		1170010183583,
		988020711114,
		1191435357423,
		743892480387,
		715658853679,
		931045209692
	  ],
	  "usage_in_kernelmode": 856990000000,
	  "usage_in_usermode": 17572550000000
	},
	"system_cpu_usage": 123541016890000000,
	"throttling_data": {
	  "periods": 0,
	  "throttled_periods": 0,
	  "throttled_time": 0
	}
  },

Previous CPU stats from Docker stats API [Which is required to calculate the initial and final difference]

"precpu_stats": {
	"cpu_usage": {
	  "total_usage": 18730095876882,
	  "percpu_usage": [
		2037709694286,
		1481641884454,
		1344434727538,
		1163031727858,
		1204370950455,
		5758849538503,
		1170009778051,
		988019895578,
		1191434644418,
		743892204863,
		715658782597,
		931042048281
	  ],
	  "usage_in_kernelmode": 856980000000,
	  "usage_in_usermode": 17572550000000
	},
	"system_cpu_usage": 123541004930000000,
	"throttling_data": {
	  "periods": 0,
	  "throttled_periods": 0,
	  "throttled_time": 0
	}
  },

There was a similar discussion in issue 18615 which led to PR 13627 "Update "docker stats " calculations"

> I think cpuPercent = (cpuDelta / systemDelta) * float64(len(v.CpuStats.CpuUsage.PercpuUsage)) is right, because:

> - cpuDelta is total time consumed of all cores and

• systemDelta is also the total time consumed of all cores,

> so (cpuDelta / systemDelta) is the average cpu usage of each core, so it need multiply the number of cpu cores to calculate the total cpu usage.

> for example, on a 4-core system, cpu usage can be anywhere between 0 and 400%, so it has to be multiplied by the number of cores.