为什么固定大小的切片分配比可变大小的bytes.Buffer更便宜?

huangapple go评论167阅读模式
英文:

Why fixed-sized slices are not cheaper to allocate than variable-sized bytes.Buffer?

问题

这是我的测试代码:

package app

import (
    "bytes"
    "testing"
)

const ALLOC_SIZE = 64 * 1024

func BenchmarkFunc1(b *testing.B) {
    for i := 0; i < b.N; i++ {
        v := make([]byte, ALLOC_SIZE)
        fill(v, '1', 0, ALLOC_SIZE)
    }
}

func BenchmarkFunc2(b *testing.B) {
    for i := 0; i < b.N; i++ {
        b := new(bytes.Buffer)
        b.Grow(ALLOC_SIZE)
        fill(b.Bytes(), '2', 0, ALLOC_SIZE)
    }
}

func fill(slice []byte, val byte, start, end int) {
    for i := start; i < end; i++ {
        slice = append(slice, val)
    }
}

结果:

at 19:05:47 ❯ go test -bench . -benchmem -gcflags=-m
# app [app.test]
./main_test.go:25:6: can inline fill
./main_test.go:10:6: can inline BenchmarkFunc1
./main_test.go:13:7: inlining call to fill
./main_test.go:20:9: inlining call to bytes.(*Buffer).Grow
./main_test.go:21:15: inlining call to bytes.(*Buffer).Bytes
./main_test.go:21:7: inlining call to fill
./main_test.go:10:21: b does not escape
./main_test.go:12:12: make([]byte, ALLOC_SIZE) escapes to heap
./main_test.go:20:9: BenchmarkFunc2 ignoring self-assignment in bytes.b.buf = bytes.b.buf[:bytes.m·3]
./main_test.go:17:21: b does not escape
./main_test.go:19:11: new(bytes.Buffer) does not escape
./main_test.go:25:11: slice does not escape
# app.test
/var/folders/45/vh6dxx396d590hxtz7_9_smmhqf0sq/T/go-build1328509211/b001/_testmain.go:35:6: can inline init.0
/var/folders/45/vh6dxx396d590hxtz7_9_smmhqf0sq/T/go-build1328509211/b001/_testmain.go:43:24: inlining call to testing.MainStart
/var/folders/45/vh6dxx396d590hxtz7_9_smmhqf0sq/T/go-build1328509211/b001/_testmain.go:43:42: testdeps.TestDeps{} escapes to heap
/var/folders/45/vh6dxx396d590hxtz7_9_smmhqf0sq/T/go-build1328509211/b001/_testmain.go:43:24: &testing.M{...} escapes to heap
goos: darwin
goarch: amd64
pkg: app
cpu: Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz
BenchmarkFunc1-8            8565            118348 ns/op          393217 B/op          4 allocs/op
BenchmarkFunc2-8           23332             53043 ns/op           65536 B/op          1 allocs/op
PASS
ok      app     2.902s

我的假设是使用由make创建的固定大小的切片比bytes.Buffer更便宜,因为编译器可能能够在编译时知道需要分配的内存大小。使用bytes.Buffer看起来像是运行时的东西。然而,结果并不是我所期望的。

对此有什么解释吗?

英文:

Here is my test code:

package app

import (
    &quot;bytes&quot;
    &quot;testing&quot;
)

const ALLOC_SIZE = 64 * 1024

func BenchmarkFunc1(b *testing.B) {
    for i := 0; i &lt; b.N; i++ {
        v := make([]byte, ALLOC_SIZE)
        fill(v, &#39;1&#39;, 0, ALLOC_SIZE)
    }
}

func BenchmarkFunc2(b *testing.B) {
    for i := 0; i &lt; b.N; i++ {
        b := new(bytes.Buffer)
        b.Grow(ALLOC_SIZE)
        fill(b.Bytes(), &#39;2&#39;, 0, ALLOC_SIZE)
    }
}

func fill(slice []byte, val byte, start, end int) {
    for i := start; i &lt; end; i++ {
        slice = append(slice, val)
    }
}

Result:

at 19:05:47 ❯ go test -bench . -benchmem -gcflags=-m
# app [app.test]
./main_test.go:25:6: can inline fill
./main_test.go:10:6: can inline BenchmarkFunc1
./main_test.go:13:7: inlining call to fill
./main_test.go:20:9: inlining call to bytes.(*Buffer).Grow
./main_test.go:21:15: inlining call to bytes.(*Buffer).Bytes
./main_test.go:21:7: inlining call to fill
./main_test.go:10:21: b does not escape
./main_test.go:12:12: make([]byte, ALLOC_SIZE) escapes to heap
./main_test.go:20:9: BenchmarkFunc2 ignoring self-assignment in bytes.b.buf = bytes.b.buf[:bytes.m&#183;3]
./main_test.go:17:21: b does not escape
./main_test.go:19:11: new(bytes.Buffer) does not escape
./main_test.go:25:11: slice does not escape
# app.test
/var/folders/45/vh6dxx396d590hxtz7_9_smmhqf0sq/T/go-build1328509211/b001/_testmain.go:35:6: can inline init.0
/var/folders/45/vh6dxx396d590hxtz7_9_smmhqf0sq/T/go-build1328509211/b001/_testmain.go:43:24: inlining call to testing.MainStart
/var/folders/45/vh6dxx396d590hxtz7_9_smmhqf0sq/T/go-build1328509211/b001/_testmain.go:43:42: testdeps.TestDeps{} escapes to heap
/var/folders/45/vh6dxx396d590hxtz7_9_smmhqf0sq/T/go-build1328509211/b001/_testmain.go:43:24: &amp;testing.M{...} escapes to heap
goos: darwin
goarch: amd64
pkg: app
cpu: Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz
BenchmarkFunc1-8            8565            118348 ns/op          393217 B/op          4 allocs/op
BenchmarkFunc2-8           23332             53043 ns/op           65536 B/op          1 allocs/op
PASS
ok      app     2.902s

My assumption was using fixed-sized slice created by make is way cheaper than bytes.Buffer, because the compiler might be able to know the size of memory required to be allocated at the compile-time. Using bytes.Buffer looks something like runtime-thing. However, the result is not what I have expected to be.

Any explaination on this?

答案1

得分: 7

你正在混淆切片的容量和长度。

v := make([]byte, ALLOC_SIZE)

v现在是一个长度为64k,容量为64k的切片。将任何内容附加到该切片会强制Go将支持数组复制到一个新的更大数组中。

b := new(bytes.Buffer)
b.Grow(ALLOC_SIZE)
v := b.Bytes()

在这里,v是一个长度为零,容量为64k的切片。您可以将64k字节附加到该切片而无需重新分配,因为它最初是空的,但64k的支持数组已准备好使用。

总之,您正在将一个已经填满容量的切片与一个具有相同容量的空切片进行比较。

为了进行公平比较,请将您的第一个基准测试更改为同样分配一个空切片:

func BenchmarkFunc1(b *testing.B) {
    for i := 0; i < b.N; i++ {
        v := make([]byte, 0, ALLOC_SIZE) // 注意三个参数的形式
        fill(v, '1', 0, ALLOC_SIZE)
    }
}
goos: linux
goarch: amd64
pkg: foo
cpu: Intel(R) Core(TM) i5-10210U CPU @ 1.60GHz
BenchmarkFunc1-8           23540             51990 ns/op           65536 B/op          1 allocs/op
BenchmarkFunc2-8           24939             45096 ns/op           65536 B/op          1 allocs/op

关于切片、数组、长度和容量之间的关系在https://blog.golang.org/slices-intro中有详细解释。

英文:

You are confusing capacity and length of slices.

v := make([]byte, ALLOC_SIZE)

v is now a slice with length 64k and capacity 64k. Appending anything to this slice forces Go to copy the backing array into a new, larger one.

b := new(bytes.Buffer)
b.Grow(ALLOC_SIZE)
v := b.Bytes()

Here, v is a slice with length zero and capacity 64k. You can append 64k bytes to this slice without any reallocation, because it is initially empty but the 64k backing array is ready to be used.

In summary, you are comparing a slice that is already filled to capacity to an empty slice with the same capacity.

To make a fair comparison change your first benchmark to allocate an empty slice as well:

func BenchmarkFunc1(b *testing.B) {
    for i := 0; i &lt; b.N; i++ {
        v := make([]byte, 0, ALLOC_SIZE) // note the three argument form
        fill(v, &#39;1&#39;, 0, ALLOC_SIZE)
    }
}
goos: linux
goarch: amd64
pkg: foo
cpu: Intel(R) Core(TM) i5-10210U CPU @ 1.60GHz
BenchmarkFunc1-8           23540             51990 ns/op           65536 B/op          1 allocs/op
BenchmarkFunc2-8           24939             45096 ns/op           65536 B/op          1 allocs/op

The relationship between slices, arrays, length, and capacity is explained in great detail in https://blog.golang.org/slices-intro

huangapple
  • 本文由 发表于 2021年7月19日 18:18:45
  • 转载请务必保留本文链接:https://go.coder-hub.com/68438818.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定