这段代码是如何生成内存对齐的切片的?

huangapple go评论89阅读模式
英文:

How is this code generating memory aligned slices?

问题

我正在为你翻译代码部分,请稍等片刻。

package main

import (
	"fmt"
	"golang.org/x/sys/unix"
	"unsafe"
	"yottaStore/yottaStore-go/src/yfs/test/utils"
)

const (
	AlignSize = 4096
	BlockSize = 4096
)

// 看起来像是黑魔法
func Alignment(block []byte, AlignSize int) int {
	return int(uintptr(unsafe.Pointer(&block[0])) & uintptr(AlignSize-1))
}

func main() {

	path := "/path/to/file.txt"
	fd, err := unix.Open(path, unix.O_RDONLY|unix.O_DIRECT, 0666)
	defer unix.Close(fd)

	if err != nil {
		panic(err)
	}

	file := make([]byte, 4096*2)

	a := Alignment(file, AlignSize)

	offset := 0
	if a != 0 {
		offset = AlignSize - a
	}

	file = file[offset : offset+BlockSize]


	n, readErr := unix.Pread(fd, file, 0)

	if readErr != nil {
		panic(readErr)
	}

	fmt.Println(a, offset, offset+utils.BlockSize, len(file))
	fmt.Println("Content is: ", string(file))
}
package main
import (
	"fmt"
	"golang.org/x/sys/unix"
	"unsafe"
)

func main() {

	path := "/path/to/file.txt"
	fd, err := unix.Open(path, unix.O_RDONLY|unix.O_DIRECT, 0666)
	defer unix.Close(fd)

	if err != nil {
		panic(err)
	}

	file := make([]byte, 4096)

	fmt.Println("Pointer: ", &file[0])

	n, readErr := unix.Pread(fd, file, 0)

	fmt.Println("Return is: ", n)

	if readErr != nil {
		panic(readErr)
	}

	fmt.Println("Content is: ", string(file))
}

请注意,我只翻译了代码部分,其他内容不包括在内。

英文:

I'm trying to do direct i/o on linux, so I need to create memory aligned buffers. I copied some code to do it, but I don't understand how it works:

package main

import (
	"fmt"
	"golang.org/x/sys/unix"
	"unsafe"
	"yottaStore/yottaStore-go/src/yfs/test/utils"
)

const (
	AlignSize = 4096
	BlockSize = 4096
)

// Looks like dark magic
func Alignment(block []byte, AlignSize int) int {
	return int(uintptr(unsafe.Pointer(&block[0])) & uintptr(AlignSize-1))
}

func main() {

	path := "/path/to/file.txt"
	fd, err := unix.Open(path, unix.O_RDONLY|unix.O_DIRECT, 0666)
	defer unix.Close(fd)

	if err != nil {
		panic(err)
	}

	file := make([]byte, 4096*2)

	a := Alignment(file, AlignSize)

	offset := 0
	if a != 0 {
		offset = AlignSize - a
	}

	file = file[offset : offset+BlockSize]


	n, readErr := unix.Pread(fd, file, 0)
    
	if readErr != nil {
		panic(readErr)
	}

    fmt.Println(a, offset, offset+utils.BlockSize, len(file))
	fmt.Println("Content is: ", string(file))
}

I understand that I'm generating a slice twice as big than what I need, and then extracting a memory aligned block from it, but the Alignment function doesn't make sense to me.

  • How does the Alignment function works?
  • If I try to fmt.Println the intermediate steps of that function I get different results, why? I guess because observing it changes its memory alignment (like in quantum physics :D)

Edit:
Example with fmt.println, where I don't need any more alignment:

package main
import (
	"fmt"
	"golang.org/x/sys/unix"
	"unsafe"
)

func main() {

	path := "/path/to/file.txt"
	fd, err := unix.Open(path, unix.O_RDONLY|unix.O_DIRECT, 0666)
	defer unix.Close(fd)

	if err != nil {
		panic(err)
	}

	file := make([]byte, 4096)

	fmt.Println("Pointer: ", &file[0])

	n, readErr := unix.Pread(fd, file, 0)

	fmt.Println("Return is: ", n)

	if readErr != nil {
		panic(readErr)
	}

	fmt.Println("Content is: ", string(file))
}

答案1

得分: 1

你的AlignSize的值是2的幂。在二进制表示中,它包含一个1位,后面全是零:

fmt.Printf("%b", AlignSize) // 1000000000000

通过make()分配的切片可能具有更多或更少随机的内存地址,其中二进制中的1和0随机排列;或者更准确地说,它的支撑数组的起始地址。

由于你分配了两倍所需的大小,这保证了支撑数组将覆盖一个地址空间,其中一个地址在中间某个位置结束,以与AlignSize的二进制表示中的尾部零一样多,并且在此处的数组起始处有BlockSize的空间。我们想要找到这个地址。

这就是Alignment()函数的作用。它使用&block[0]获取支撑数组的起始地址。在Go中没有指针算术,所以为了做这样的事情,我们必须将指针转换为整数(当然有整数算术)。为了做到这一点,我们必须将指针转换为unsafe.Pointer:所有指针都可以转换为此类型,并且unsafe.Pointer可以转换为uintptr(它是一个无符号整数,足够大以存储指针值的未解释位),在其上进行整数算术。

我们使用值uintptr(AlignSize-1)进行按位与运算。由于AlignSize是2的幂(包含一个单独的1位后跟零),减去1的结果是一个二进制表示中全是1的数字,尾部有与AlignSize相同数量的零。看看这个例子:

x := 0b1010101110101010101
fmt.Printf("AlignSize   : %22b\n", AlignSize)
fmt.Printf("AlignSize-1 : %22b\n", AlignSize-1)
fmt.Printf("x           : %22b\n", x)
fmt.Printf("result of & : %22b\n", x&(AlignSize-1))

输出:

AlignSize   :          1000000000000
AlignSize-1 :           111111111111
x           :    1010101110101010101
result of & :           110101010101

因此,&的结果是一个偏移量,如果你从AlignSize中减去它,你会得到一个地址,其尾部有与AlignSize本身一样多的零:结果是对AlignSize的倍数进行了“对齐”。

因此,我们将使用从offset开始的file切片的一部分,我们只需要BlockSize

file = file[offset : offset+BlockSize]

编辑:

查看你修改后的代码尝试打印步骤:我得到的输出如下:

Pointer:  0xc0000b6000
Unsafe pointer:  0xc0000b6000
Unsafe pointer, uintptr:  824634466304
Unpersand:  0
Cast to int:  0
Return is:  0
Content is: 

注意这里什么都没有改变。简单地说,fmt包使用十六进制表示打印指针值,前缀为0xuintptr值以整数形式打印,使用十进制表示。这些值是相等的:

fmt.Println(0xc0000b6000, 824634466304) // 输出: 824634466304 824634466304

还要注意,其余部分都是0,因为在我的情况下,0xc0000b6000已经是4096的倍数,在二进制中是1100000000000000000100001110000000000000

编辑 #2:

当你使用fmt.Println()来调试计算的部分时,这可能会改变逃逸分析,并可能改变切片的分配(从堆栈到堆)。这还取决于使用的Go版本。不要依赖于你的切片被分配到一个已经对齐到AlignSize的地址上。

有关更多详细信息,请参见相关问题:

https://stackoverflow.com/questions/65874745/mix-print-and-fmt-println-and-stack-growing/65874913#65874913

https://stackoverflow.com/questions/52421103/why-struct-arrays-comparing-has-different-result/52421277#52421277

https://stackoverflow.com/questions/48052722/addresses-of-slices-of-empty-structs/48053270#48053270

英文:

Your AlignSize has a value of a power of 2. In binary representation it contains a 1 bit followed by full of zeros:

fmt.Printf("%b", AlignSize) // 1000000000000

A slice allocated by make() may have a memory address that is more or less random, consisting of ones and zeros following randomly in binary; or more precisely the starting address of its backing array.

Since you allocate twice the required size, that's a guarantee that the backing array will cover an address space that has an address in the middle somewhere that ends with as many zeros as the AlignSize's binary representation, and has BlockSize room in the array starting at this. We want to find this address.

This is what the Alignment() function does. It gets the starting address of the backing array with &block[0]. In Go there's no pointer arithmetic, so in order to do something like that, we have to convert the pointer to an integer (there is integer arithmetic of course). In order to do that, we have to convert the pointer to unsafe.Pointer: all pointers are convertible to this type, and unsafe.Pointer can be converted to uintptr (which is an unsigned integer large enough to store the uninterpreted bits of a pointer value), on which–being an integer–we can perform integer arithmetic.

We use bitwise AND with the value uintptr(AlignSize-1). Since AlignSize is a power of 2 (contains a single 1 bit followed by zeros), the number one less is a number whose binary representation is full of ones, as many as trailing zeros AlignSize has. See this example:

x := 0b1010101110101010101
fmt.Printf("AlignSize   : %22b\n", AlignSize)
fmt.Printf("AlignSize-1 : %22b\n", AlignSize-1)
fmt.Printf("x           : %22b\n", x)
fmt.Printf("result of & : %22b\n", x&(AlignSize-1))

Output:

AlignSize   :          1000000000000
AlignSize-1 :           111111111111
x           :    1010101110101010101
result of & :           110101010101

So the result of & is the offset which if you subtract from AlignSize, you get an address that has as many trailing zeros as AlignSize itself: the result is "aligned" to the multiple of AlignSize.

So we will use the part of the file slice starting at offset, and we only need BlockSize:

file = file[offset : offset+BlockSize]

Edit:

Looking at your modified code trying to print the steps: I get an output like:

Pointer:  0xc0000b6000
Unsafe pointer:  0xc0000b6000
Unsafe pointer, uintptr:  824634466304
Unpersand:  0
Cast to int:  0
Return is:  0
Content is: 

Note nothing is changed here. Simply the fmt package prints pointer values using hexadecimal representation, prefixed by 0x. uintptr values are printed as integers, using decimal representation. Those values are equal:

fmt.Println(0xc0000b6000, 824634466304) // output: 824634466304 824634466304

Also note the rest is 0 because in my case 0xc0000b6000 is already a multiple of 4096, in binary it is 1100000000000000000100001110000000000000.

Edit #2:

When you use fmt.Println() to debug parts of the calculation, that may change escape analysis and may change the allocation of the slice (from stack to heap). This depends on the used Go version too. Do not rely on your slice being allocated at an address that is (already) aligned to AlignSize.

See related questions for more details:

https://stackoverflow.com/questions/65874745/mix-print-and-fmt-println-and-stack-growing/65874913#65874913

https://stackoverflow.com/questions/52421103/why-struct-arrays-comparing-has-different-result/52421277#52421277

https://stackoverflow.com/questions/48052722/addresses-of-slices-of-empty-structs/48053270#48053270

huangapple
  • 本文由 发表于 2022年8月31日 15:59:39
  • 转载请务必保留本文链接:https://go.coder-hub.com/73552648.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定