英文:
How is this code generating memory aligned slices?
问题
我正在为你翻译代码部分,请稍等片刻。
package main
import (
"fmt"
"golang.org/x/sys/unix"
"unsafe"
"yottaStore/yottaStore-go/src/yfs/test/utils"
)
const (
AlignSize = 4096
BlockSize = 4096
)
// 看起来像是黑魔法
func Alignment(block []byte, AlignSize int) int {
return int(uintptr(unsafe.Pointer(&block[0])) & uintptr(AlignSize-1))
}
func main() {
path := "/path/to/file.txt"
fd, err := unix.Open(path, unix.O_RDONLY|unix.O_DIRECT, 0666)
defer unix.Close(fd)
if err != nil {
panic(err)
}
file := make([]byte, 4096*2)
a := Alignment(file, AlignSize)
offset := 0
if a != 0 {
offset = AlignSize - a
}
file = file[offset : offset+BlockSize]
n, readErr := unix.Pread(fd, file, 0)
if readErr != nil {
panic(readErr)
}
fmt.Println(a, offset, offset+utils.BlockSize, len(file))
fmt.Println("Content is: ", string(file))
}
package main
import (
"fmt"
"golang.org/x/sys/unix"
"unsafe"
)
func main() {
path := "/path/to/file.txt"
fd, err := unix.Open(path, unix.O_RDONLY|unix.O_DIRECT, 0666)
defer unix.Close(fd)
if err != nil {
panic(err)
}
file := make([]byte, 4096)
fmt.Println("Pointer: ", &file[0])
n, readErr := unix.Pread(fd, file, 0)
fmt.Println("Return is: ", n)
if readErr != nil {
panic(readErr)
}
fmt.Println("Content is: ", string(file))
}
请注意,我只翻译了代码部分,其他内容不包括在内。
英文:
I'm trying to do direct i/o on linux, so I need to create memory aligned buffers. I copied some code to do it, but I don't understand how it works:
package main
import (
"fmt"
"golang.org/x/sys/unix"
"unsafe"
"yottaStore/yottaStore-go/src/yfs/test/utils"
)
const (
AlignSize = 4096
BlockSize = 4096
)
// Looks like dark magic
func Alignment(block []byte, AlignSize int) int {
return int(uintptr(unsafe.Pointer(&block[0])) & uintptr(AlignSize-1))
}
func main() {
path := "/path/to/file.txt"
fd, err := unix.Open(path, unix.O_RDONLY|unix.O_DIRECT, 0666)
defer unix.Close(fd)
if err != nil {
panic(err)
}
file := make([]byte, 4096*2)
a := Alignment(file, AlignSize)
offset := 0
if a != 0 {
offset = AlignSize - a
}
file = file[offset : offset+BlockSize]
n, readErr := unix.Pread(fd, file, 0)
if readErr != nil {
panic(readErr)
}
fmt.Println(a, offset, offset+utils.BlockSize, len(file))
fmt.Println("Content is: ", string(file))
}
I understand that I'm generating a slice twice as big than what I need, and then extracting a memory aligned block from it, but the Alignment
function doesn't make sense to me.
- How does the
Alignment
function works? - If I try to
fmt.Println
the intermediate steps of that function I get different results, why? I guess because observing it changes its memory alignment (like in quantum physics :D)
Edit:
Example with fmt.println
, where I don't need any more alignment:
package main
import (
"fmt"
"golang.org/x/sys/unix"
"unsafe"
)
func main() {
path := "/path/to/file.txt"
fd, err := unix.Open(path, unix.O_RDONLY|unix.O_DIRECT, 0666)
defer unix.Close(fd)
if err != nil {
panic(err)
}
file := make([]byte, 4096)
fmt.Println("Pointer: ", &file[0])
n, readErr := unix.Pread(fd, file, 0)
fmt.Println("Return is: ", n)
if readErr != nil {
panic(readErr)
}
fmt.Println("Content is: ", string(file))
}
答案1
得分: 1
你的AlignSize
的值是2的幂。在二进制表示中,它包含一个1
位,后面全是零:
fmt.Printf("%b", AlignSize) // 1000000000000
通过make()
分配的切片可能具有更多或更少随机的内存地址,其中二进制中的1和0随机排列;或者更准确地说,它的支撑数组的起始地址。
由于你分配了两倍所需的大小,这保证了支撑数组将覆盖一个地址空间,其中一个地址在中间某个位置结束,以与AlignSize
的二进制表示中的尾部零一样多,并且在此处的数组起始处有BlockSize
的空间。我们想要找到这个地址。
这就是Alignment()
函数的作用。它使用&block[0]
获取支撑数组的起始地址。在Go中没有指针算术,所以为了做这样的事情,我们必须将指针转换为整数(当然有整数算术)。为了做到这一点,我们必须将指针转换为unsafe.Pointer
:所有指针都可以转换为此类型,并且unsafe.Pointer
可以转换为uintptr
(它是一个无符号整数,足够大以存储指针值的未解释位),在其上进行整数算术。
我们使用值uintptr(AlignSize-1)
进行按位与运算。由于AlignSize
是2的幂(包含一个单独的1
位后跟零),减去1的结果是一个二进制表示中全是1的数字,尾部有与AlignSize
相同数量的零。看看这个例子:
x := 0b1010101110101010101
fmt.Printf("AlignSize : %22b\n", AlignSize)
fmt.Printf("AlignSize-1 : %22b\n", AlignSize-1)
fmt.Printf("x : %22b\n", x)
fmt.Printf("result of & : %22b\n", x&(AlignSize-1))
输出:
AlignSize : 1000000000000
AlignSize-1 : 111111111111
x : 1010101110101010101
result of & : 110101010101
因此,&
的结果是一个偏移量,如果你从AlignSize
中减去它,你会得到一个地址,其尾部有与AlignSize
本身一样多的零:结果是对AlignSize
的倍数进行了“对齐”。
因此,我们将使用从offset
开始的file
切片的一部分,我们只需要BlockSize
:
file = file[offset : offset+BlockSize]
编辑:
查看你修改后的代码尝试打印步骤:我得到的输出如下:
Pointer: 0xc0000b6000
Unsafe pointer: 0xc0000b6000
Unsafe pointer, uintptr: 824634466304
Unpersand: 0
Cast to int: 0
Return is: 0
Content is:
注意这里什么都没有改变。简单地说,fmt
包使用十六进制表示打印指针值,前缀为0x
。uintptr
值以整数形式打印,使用十进制表示。这些值是相等的:
fmt.Println(0xc0000b6000, 824634466304) // 输出: 824634466304 824634466304
还要注意,其余部分都是0
,因为在我的情况下,0xc0000b6000
已经是4096
的倍数,在二进制中是1100000000000000000100001110000000000000
。
编辑 #2:
当你使用fmt.Println()
来调试计算的部分时,这可能会改变逃逸分析,并可能改变切片的分配(从堆栈到堆)。这还取决于使用的Go版本。不要依赖于你的切片被分配到一个已经对齐到AlignSize
的地址上。
有关更多详细信息,请参见相关问题:
https://stackoverflow.com/questions/48052722/addresses-of-slices-of-empty-structs/48053270#48053270
英文:
Your AlignSize
has a value of a power of 2. In binary representation it contains a 1
bit followed by full of zeros:
fmt.Printf("%b", AlignSize) // 1000000000000
A slice allocated by make()
may have a memory address that is more or less random, consisting of ones and zeros following randomly in binary; or more precisely the starting address of its backing array.
Since you allocate twice the required size, that's a guarantee that the backing array will cover an address space that has an address in the middle somewhere that ends with as many zeros as the AlignSize
's binary representation, and has BlockSize
room in the array starting at this. We want to find this address.
This is what the Alignment()
function does. It gets the starting address of the backing array with &block[0]
. In Go there's no pointer arithmetic, so in order to do something like that, we have to convert the pointer to an integer (there is integer arithmetic of course). In order to do that, we have to convert the pointer to unsafe.Pointer
: all pointers are convertible to this type, and unsafe.Pointer
can be converted to uintptr
(which is an unsigned integer large enough to store the uninterpreted bits of a pointer value), on which–being an integer–we can perform integer arithmetic.
We use bitwise AND with the value uintptr(AlignSize-1)
. Since AlignSize
is a power of 2 (contains a single 1
bit followed by zeros), the number one less is a number whose binary representation is full of ones, as many as trailing zeros AlignSize
has. See this example:
x := 0b1010101110101010101
fmt.Printf("AlignSize : %22b\n", AlignSize)
fmt.Printf("AlignSize-1 : %22b\n", AlignSize-1)
fmt.Printf("x : %22b\n", x)
fmt.Printf("result of & : %22b\n", x&(AlignSize-1))
Output:
AlignSize : 1000000000000
AlignSize-1 : 111111111111
x : 1010101110101010101
result of & : 110101010101
So the result of &
is the offset which if you subtract from AlignSize
, you get an address that has as many trailing zeros as AlignSize
itself: the result is "aligned" to the multiple of AlignSize
.
So we will use the part of the file
slice starting at offset
, and we only need BlockSize
:
file = file[offset : offset+BlockSize]
Edit:
Looking at your modified code trying to print the steps: I get an output like:
Pointer: 0xc0000b6000
Unsafe pointer: 0xc0000b6000
Unsafe pointer, uintptr: 824634466304
Unpersand: 0
Cast to int: 0
Return is: 0
Content is:
Note nothing is changed here. Simply the fmt
package prints pointer values using hexadecimal representation, prefixed by 0x
. uintptr
values are printed as integers, using decimal representation. Those values are equal:
fmt.Println(0xc0000b6000, 824634466304) // output: 824634466304 824634466304
Also note the rest is 0
because in my case 0xc0000b6000
is already a multiple of 4096
, in binary it is 1100000000000000000100001110000000000000
.
Edit #2:
When you use fmt.Println()
to debug parts of the calculation, that may change escape analysis and may change the allocation of the slice (from stack to heap). This depends on the used Go version too. Do not rely on your slice being allocated at an address that is (already) aligned to AlignSize
.
See related questions for more details:
https://stackoverflow.com/questions/48052722/addresses-of-slices-of-empty-structs/48053270#48053270
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论