为什么固定大小的切片分配比可变大小的bytes.Buffer更便宜？

huangapple
go评论117阅读模式
英文:

Why fixed-sized slices are not cheaper to allocate than variable-sized bytes.Buffer?

问题

这是我的测试代码：

  1. package app
  3. import (
  4.     "bytes"
  5.     "testing"
  6. )
  8. const ALLOC_SIZE = 64 * 1024
  10. func BenchmarkFunc1(b *testing.B) {
  11.     for i := 0; i < b.N; i++ {
  12.         v := make([]byte, ALLOC_SIZE)
  13.         fill(v, '1', 0, ALLOC_SIZE)
  14.     }
  15. }
  17. func BenchmarkFunc2(b *testing.B) {
  18.     for i := 0; i < b.N; i++ {
  19.         b := new(bytes.Buffer)
  20.         b.Grow(ALLOC_SIZE)
  21.         fill(b.Bytes(), '2', 0, ALLOC_SIZE)
  22.     }
  23. }
  25. func fill(slice []byte, val byte, start, end int) {
  26.     for i := start; i < end; i++ {
  27.         slice = append(slice, val)
  28.     }
  29. }

结果：

  1. at 19:05:47 ❯ go test -bench . -benchmem -gcflags=-m
  2. # app [app.test]
  3. ./main_test.go:25:6: can inline fill
  4. ./main_test.go:10:6: can inline BenchmarkFunc1
  5. ./main_test.go:13:7: inlining call to fill
  6. ./main_test.go:20:9: inlining call to bytes.(*Buffer).Grow
  7. ./main_test.go:21:15: inlining call to bytes.(*Buffer).Bytes
  8. ./main_test.go:21:7: inlining call to fill
  9. ./main_test.go:10:21: b does not escape
  10. ./main_test.go:12:12: make([]byte, ALLOC_SIZE) escapes to heap
  11. ./main_test.go:20:9: BenchmarkFunc2 ignoring self-assignment in bytes.b.buf = bytes.b.buf[:bytes.m·3]
  12. ./main_test.go:17:21: b does not escape
  13. ./main_test.go:19:11: new(bytes.Buffer) does not escape
  14. ./main_test.go:25:11: slice does not escape
  15. # app.test
  16. /var/folders/45/vh6dxx396d590hxtz7_9_smmhqf0sq/T/go-build1328509211/b001/_testmain.go:35:6: can inline init.0
  17. /var/folders/45/vh6dxx396d590hxtz7_9_smmhqf0sq/T/go-build1328509211/b001/_testmain.go:43:24: inlining call to testing.MainStart
  18. /var/folders/45/vh6dxx396d590hxtz7_9_smmhqf0sq/T/go-build1328509211/b001/_testmain.go:43:42: testdeps.TestDeps{} escapes to heap
  19. /var/folders/45/vh6dxx396d590hxtz7_9_smmhqf0sq/T/go-build1328509211/b001/_testmain.go:43:24: &testing.M{...} escapes to heap
  20. goos: darwin
  21. goarch: amd64
  22. pkg: app
  23. cpu: Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz
  24. BenchmarkFunc1-8            8565            118348 ns/op          393217 B/op          4 allocs/op
  25. BenchmarkFunc2-8           23332             53043 ns/op           65536 B/op          1 allocs/op
  26. PASS
  27. ok      app     2.902s

我的假设是使用由make创建的固定大小的切片比bytes.Buffer更便宜，因为编译器可能能够在编译时知道需要分配的内存大小。使用bytes.Buffer看起来像是运行时的东西。然而，结果并不是我所期望的。

对此有什么解释吗？

英文:

Here is my test code:

  1. package app
  3. import (
  4.     &quot;bytes&quot;
  5.     &quot;testing&quot;
  6. )
  8. const ALLOC_SIZE = 64 * 1024
  10. func BenchmarkFunc1(b *testing.B) {
  11.     for i := 0; i &lt; b.N; i++ {
  12.         v := make([]byte, ALLOC_SIZE)
  13.         fill(v, &#39;1&#39;, 0, ALLOC_SIZE)
  14.     }
  15. }
  17. func BenchmarkFunc2(b *testing.B) {
  18.     for i := 0; i &lt; b.N; i++ {
  19.         b := new(bytes.Buffer)
  20.         b.Grow(ALLOC_SIZE)
  21.         fill(b.Bytes(), &#39;2&#39;, 0, ALLOC_SIZE)
  22.     }
  23. }
  25. func fill(slice []byte, val byte, start, end int) {
  26.     for i := start; i &lt; end; i++ {
  27.         slice = append(slice, val)
  28.     }
  29. }

Result:

  1. at 19:05:47 ❯ go test -bench . -benchmem -gcflags=-m
  2. # app [app.test]
  3. ./main_test.go:25:6: can inline fill
  4. ./main_test.go:10:6: can inline BenchmarkFunc1
  5. ./main_test.go:13:7: inlining call to fill
  6. ./main_test.go:20:9: inlining call to bytes.(*Buffer).Grow
  7. ./main_test.go:21:15: inlining call to bytes.(*Buffer).Bytes
  8. ./main_test.go:21:7: inlining call to fill
  9. ./main_test.go:10:21: b does not escape
  10. ./main_test.go:12:12: make([]byte, ALLOC_SIZE) escapes to heap
  11. ./main_test.go:20:9: BenchmarkFunc2 ignoring self-assignment in bytes.b.buf = bytes.b.buf[:bytes.m&#183;3]
  12. ./main_test.go:17:21: b does not escape
  13. ./main_test.go:19:11: new(bytes.Buffer) does not escape
  14. ./main_test.go:25:11: slice does not escape
  15. # app.test
  16. /var/folders/45/vh6dxx396d590hxtz7_9_smmhqf0sq/T/go-build1328509211/b001/_testmain.go:35:6: can inline init.0
  17. /var/folders/45/vh6dxx396d590hxtz7_9_smmhqf0sq/T/go-build1328509211/b001/_testmain.go:43:24: inlining call to testing.MainStart
  18. /var/folders/45/vh6dxx396d590hxtz7_9_smmhqf0sq/T/go-build1328509211/b001/_testmain.go:43:42: testdeps.TestDeps{} escapes to heap
  19. /var/folders/45/vh6dxx396d590hxtz7_9_smmhqf0sq/T/go-build1328509211/b001/_testmain.go:43:24: &amp;testing.M{...} escapes to heap
  20. goos: darwin
  21. goarch: amd64
  22. pkg: app
  23. cpu: Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz
  24. BenchmarkFunc1-8            8565            118348 ns/op          393217 B/op          4 allocs/op
  25. BenchmarkFunc2-8           23332             53043 ns/op           65536 B/op          1 allocs/op
  26. PASS
  27. ok      app     2.902s

My assumption was using fixed-sized slice created by make is way cheaper than bytes.Buffer, because the compiler might be able to know the size of memory required to be allocated at the compile-time. Using bytes.Buffer looks something like runtime-thing. However, the result is not what I have expected to be.

Any explaination on this?

答案1

得分: 7

你正在混淆切片的容量和长度。

  1. v := make([]byte, ALLOC_SIZE)

v现在是一个长度为64k，容量为64k的切片。将任何内容附加到该切片会强制Go将支持数组复制到一个新的更大数组中。

  1. b := new(bytes.Buffer)
  2. b.Grow(ALLOC_SIZE)
  3. v := b.Bytes()

在这里，v是一个长度为零，容量为64k的切片。您可以将64k字节附加到该切片而无需重新分配，因为它最初是空的，但64k的支持数组已准备好使用。

总之，您正在将一个已经填满容量的切片与一个具有相同容量的空切片进行比较。

为了进行公平比较，请将您的第一个基准测试更改为同样分配一个空切片：

  1. func BenchmarkFunc1(b *testing.B) {
  2.     for i := 0; i < b.N; i++ {
  3.         v := make([]byte, 0, ALLOC_SIZE) // 注意三个参数的形式
  4.         fill(v, '1', 0, ALLOC_SIZE)
  5.     }
  6. }
  1. goos: linux
  2. goarch: amd64
  3. pkg: foo
  4. cpu: Intel(R) Core(TM) i5-10210U CPU @ 1.60GHz
  5. BenchmarkFunc1-8           23540             51990 ns/op           65536 B/op          1 allocs/op
  6. BenchmarkFunc2-8           24939             45096 ns/op           65536 B/op          1 allocs/op

关于切片、数组、长度和容量之间的关系在https://blog.golang.org/slices-intro中有详细解释。

英文:

You are confusing capacity and length of slices.

  1. v := make([]byte, ALLOC_SIZE)

v is now a slice with length 64k and capacity 64k. Appending anything to this slice forces Go to copy the backing array into a new, larger one.

  1. b := new(bytes.Buffer)
  2. b.Grow(ALLOC_SIZE)
  3. v := b.Bytes()

Here, v is a slice with length zero and capacity 64k. You can append 64k bytes to this slice without any reallocation, because it is initially empty but the 64k backing array is ready to be used.

In summary, you are comparing a slice that is already filled to capacity to an empty slice with the same capacity.

To make a fair comparison change your first benchmark to allocate an empty slice as well:

  1. func BenchmarkFunc1(b *testing.B) {
  2.     for i := 0; i &lt; b.N; i++ {
  3.         v := make([]byte, 0, ALLOC_SIZE) // note the three argument form
  4.         fill(v, &#39;1&#39;, 0, ALLOC_SIZE)
  5.     }
  6. }
  1. goos: linux
  2. goarch: amd64
  3. pkg: foo
  4. cpu: Intel(R) Core(TM) i5-10210U CPU @ 1.60GHz
  5. BenchmarkFunc1-8           23540             51990 ns/op           65536 B/op          1 allocs/op
  6. BenchmarkFunc2-8           24939             45096 ns/op           65536 B/op          1 allocs/op

The relationship between slices, arrays, length, and capacity is explained in great detail in https://blog.golang.org/slices-intro

huangapple
  • 本文由 发表于 2021年7月19日 18:18:45
  • 转载请务必保留本文链接：https://go.coder-hub.com/68438818.html
  • go
  • heap-memory
  • memory-management
  • performance
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定