英文:
Golang and large memory chunk allocation
问题
我有一个相当愚蠢的基准测试来测试内存分配的效率：
package main
import (
"time"
"fmt"
)
func main() {
size := 1024 * 1024 * 1024
t := time.Now()
for j := 0; j < 100; j += 1 {
a := make([]int, size)
for i := 0; i < size; i += 1 {
a[i] = i
}
a = nil
}
t1 := time.Now()
fmt.Printf("Duration: %1d", t1.Sub(t).Seconds())
}
在具有16GB RAM的Mac Pro上大约需要2-3分钟，进程的内存使用量保持在5-8GB。
相似的Java代码需要3GB的内存，并在30秒内完成。
我在这里漏掉了什么？
英文:
I have the rather stupid benchmark to test the efficiency of memory allocation:
package main
import (
"time"
"fmt"
)
func main() {
size := 1024 * 1024 * 1024
t := time.Now()
for j := 0; j < 100; j += 1 {
a := make([]int, size)
for i := 0; i < size; i += 1 {
a[i] = i
}
a = nil
}
t1 := time.Now()
fmt.Printf("Duration: %1d", t1.Sub(t).Seconds())
}
It takes roughly 2-3 minutes on Mac Pro with 16GB of RAM, the memory usage is consistent at 5-8 GB for the process.
The very similar code in Java takes 3 GB of memory and completes in 30 seconds.
What am I missing here?
答案1
得分: 4
我在这里可能会错过什么？
在Java中，int类型的大小固定为4个字节。而在Go中，int是一个与架构相关的类型，在32位架构上为32位（4个字节），在64位架构上为64位（8个字节）。
很可能你正在64位架构上运行。这意味着你分配的Go切片/数组的大小为8 * 1 GB = 8 GB，而在Java中只有4 * 1 GB = 4 GB。
此外，由于你在循环中使用了int，Java只需要增加和设置4字节的值，而在Go中，你正在增加和设置8字节的值（i和j的类型将为int）。
将你的Go代码更改为使用int32，然后再试一次。
还要注意，你的内存使用量测量是有缺陷的，因为在Java中仅数组大小为4 GB，在Go中为8 GB，所以Java使用的内存为3 GB，而Go使用的内存为5-8 GB，并不是总内存使用量！
还要注意，在Go中，[]int是一个切片（slice）而不是数组（array），它们不是相同的。Go中的切片是类似结构体的头部，包含指向支持数组的指针（详见reflect.SliceHeader），因此使用它们会涉及到一个隐式的间接步骤。详情请参考：https://stackoverflow.com/questions/38645175/why-have-arrays-in-go/38645895#38645895 还有相关问题：https://stackoverflow.com/questions/30525184/array-vs-slice-accessing-speed
最后注意一点：你的代码并没有测量内存分配，因为那只是应用程序执行时间的一小部分。大部分（99.99999%）的执行时间用于递增循环变量一亿次，并将数组填充为十亿个元素。
英文:
> What am I missing here?
In Java, size of the int type is fixed 4 bytes. In Go int is an architecture dependent type, on 32-bit architectures it's 32 bits (4 bytes), and on 64-bit architectures it's 64 bits (8 bytes).
Most likely you're running it on a 64-bit arch. Which means the size of the Go slice / array you allocate is 8 * 1 GB = 8 GB, while in Java it's only 4 * 1 GB = 4 GB.
Moreover, since you're using int in your loop, Java only has to increment and set 4-byte values, while in Go you're incrementing and setting 8-byte values (type of i and j will be int).
Change your Go code to use int32, and give it a go again.
Also note that your memory usage measurement is flawed, as just the array size in Java is 4 GB and 8 GB in Go, so that 3 GB for Java and 5-8 GB in Go is not the total memory usage!
Also note that []int in Go is a slice and not an array, they are not the same. Slices in Go are struct-like headers containing a pointer to a backing array (see reflect.SliceHeader for details), so there is an implicit indirection step involved using them. For details see https://stackoverflow.com/questions/38645175/why-have-arrays-in-go/38645895#38645895 Also related: https://stackoverflow.com/questions/30525184/array-vs-slice-accessing-speed
One last note: your code does not measure memory allocation, as that is just the tiny part of the app's execution time. The majority (like 99.99999%) of the execution time is to increment the loop variable a billion times and to fill the arrays with a billion elements.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论