Go可以使用页面级别的内存吗?

huangapple go评论90阅读模式
英文:

Can Go work with memory at the page level?

问题

我是你的中文翻译助手,以下是翻译的内容:

我是一名Go开发者,正在阅读一本名为《数据库内部原理》的书。作者详细讨论了在非常低的层次上(具体来说,是在页面级别上)处理内存的方法。

由于我正试图构建自己的数据库,我已经查阅了Go文档和其他文章,以了解在这个层次上处理内存的相关讨论,但似乎没有任何内容涉及在特定内存页面中处理数据的方法。

我的问题是:

Go能做到这一点吗?

如果Go不能做到这一点,C或C++可以吗?

如果Go、C或C++都不能做到这一点,也许我没有理解作者所说的在页面级别上处理内存的含义,我应该如何思考?

如果需要的话,请告诉我需要在问题中添加哪些进一步的细节。

书中摘录:

背景是我们正在讨论实现B树和将节点分组到页面中。这在第2章的“磁盘结构”一节中可以找到。

将二叉树的节点分组到页面中,如图2-6所示,可以改善局部性。要找到下一个节点,只需要在已经获取的页面中跟随一个指针即可。然而,节点和它们之间的指针仍然会产生一些开销。在磁盘上布局结构及其进一步的维护都是非平凡的任务,特别是如果键和值没有预排序并以随机顺序添加。平衡需要对页面进行重组,这又会导致指针更新。

书中还提供了这张图片:
Go可以使用页面级别的内存吗?

英文:

I am a Go developer reading a book called Database Internals. The author talks extensively about working with memory at a very low level (specifically, at the page level).

As I am trying to build my own database, I have looked through Go documentation and other articles for discussions about work with memory at this level, however nothings seems to talk about working with data in a specific page of memory.

My questions are:

Can Go do this?

If Go can't do this, is this something that C or C++ can do?

If none of Go, C, or C++ can do that, perhaps I'm not understanding what the author means by working with memory at the page level, how should I be thinking about it?

If necessary, please let me know of any further detail that needs to added to the question.

Excerpt from the book

Context is that we're talking about implementing B-Trees and grouping nodes in pages. This is found in chapter 2, section titled "On-Disk Structures"

> PAGED BINARY TREES Laying out a binary tree by grouping nodes into
> pages, as Figure 2-6 shows, improves the situation with locality. To
> find the next node, it’s only necessary to follow a pointer in an
> already fetched page. However, there’s still some overhead incurred by
> the nodes and pointers between them. Laying the structure out on disk
> and its further maintenance are nontrivial endeavors, especially if
> keys and values are not presorted and added in random order. Balancing
> requires page reorganization, which in turn causes pointer updates.

It also provides this image:
Go可以使用页面级别的内存吗?

答案1

得分: 3

这本书讨论的是“On-Disk Structures”(磁盘上的结构)。在这个上下文中,Page(页)简单地指的是一块数据块。磁盘访问以扇区或簇为单位进行,因此数据应该被优化为尽可能适应这些块的局部性。数据库文件设计的挑战在于将数据的时间局部性转化为空间局部性。

但这个概念也可以应用于RAM。在用户空间虚拟内存中以页级别工作,意味着要了解底层的内存架构,并对其进行优化。

在x86上,用户空间虚拟内存以4 KB的页面组织。

这意味着当处理大量数据时,值得使用大小为4 KB的内存区域,并对齐到4 KB的倍数。

在C语言中,有很多方法可以实现这一点,例如aligned_alloc

但Go语言更进一步 - 它已经将大数组至少对齐到页面大小(在Mac、BSD、Linux和Windows上)。

func main() {
    buf := make([]byte, 1024*1024)
    fmt.Printf("%p\n", unsafe.Pointer(&buf[0]))
}

将打印类似以下的内容:

0xc000180000

如果遇到不支持对齐的实现,你总是可以通过分配PageSize-1额外的字节,然后跳过不以PageSize的倍数开始的字节,使用以下对齐公式来手动对齐byte切片:

> offset = (alignment - base) & (alignment - 1)

func main() {
    buf := make([]byte, 1024*1024 + 4096 - 1)
    base := unsafe.Pointer(&buf[0])
    offset := (4096 - uintptr(base)) & (4096 - 1)
    aligned := buf[offset:]
    fmt.Printf("base   : %p\n", unsafe.Pointer(&buf[0]))
    fmt.Printf("aligned: %p\n", unsafe.Pointer(&aligned[0]))
}

(我无法想到一个平台会打印出不同的值)


* 一些平台支持2MB和1GB的大页面。Go会在可用时自动使用它们。

英文:

The book is talking about "On-Disk Structures". Page in this context simply means a block of data. Disk access works in sectors or clusters, so the data should be optimized for locality in order to fit in those blocks as best as possible. The challenge in database file design is to translate data's temporal locality into spatial locality.

But the concept can apply to RAM, too. Working at page level in user-space virtual memory simply means being aware of the underlying memory architecture, and optimizing for it.

On x86, user-space virtual memory is organized in pages of 4 KB*.

That means that when working with large amounts of data, it can be worthwhile to work with memory regions which are a multiple of 4 KB, aligned to 4 KB.

In C there are many ways to achieve that, e.g. aligned_alloc.

But Go goes further - it already aligns large arrays at least on page size (on Mac, BSD, Linux and Windows).

func main() {
	buf := make([]byte, 1024*1024)
	fmt.Printf("%p\n", unsafe.Pointer(&buf[0]))
}

Will print something like:

0xc000180000

And in case you encounter an implementation that doesn't, you can always align a byte slice "manually" by allocating PageSize-1 extra bytes and then skipping the bytes at the beginning which do not start at a multiple of PageSize, using the alignment formula:

> offset = (alignment - base) & (alignment - 1)

func main() {
	buf := make([]byte, 1024*1024 + 4096 - 1)
	base := unsafe.Pointer(&buf[0])
	offset := (4096 - uintptr(base)) & (4096 - 1)
	aligned := buf[offset:]
	fmt.Printf("base   : %p\n", unsafe.Pointer(&buf[0]))
	fmt.Printf("aligned: %p\n", unsafe.Pointer(&aligned[0]))
}

(can't think of a platform where the above would print different values)


* Some platforms have large page support for 2MB and 1GB pages. Go uses those automatically when available.

答案2

得分: 1

我可能漏掉了一些东西,但是与虚拟内存相反,与物理内存一起工作并不是一种语言特性,而是操作系统的一部分。所以对于你的问题,答案是:是的,Go语言也可以做到这一点。

每个操作系统都提供了与实际地址空间交互的方法,这些服务需要提升的权限,并以驱动程序/内核模块的形式提供,可以使用这些访问权限进行内存映射IO或预分配一块物理内存块进行操作(在MMI将其用作分页内存之前)。

我猜你希望在系统加载时(通过驱动程序)预分配内存块并进行操作。

如果你打算在"物理内存"上访问"虚拟内存",我会劝阻你这样做,并说这是一种非常微妙和脆弱的机制,除非你找到了巨大的优势,否则我无法看到任何理由去这样做。

英文:

I might be missing something,
but working with Physical memory (in contrary to Virtual memory) is not a Language feature, but of the OS.
So the answer to your question is: Yes, Go can do this too.

Each operating system provides methods to work with the actual address space,
These services require elevated permissions and come as Driver/Kernel-Module
and using this access for Memory Mapped IO or pre-allocating block of physical memory to work with (before the MMI uses it as paged memory).

I guess pre-allocating memory block on system load (by driver) and working with that is what you wish to do.

If you intend to access "Virtual Memory" on the "Physical Memory" I'd discourage you and say that it is an extremely delicate and fragile mechanics and unless you find a HUGE advantage for doing so, I cant see any justification of going down this road.

huangapple
  • 本文由 发表于 2021年7月3日 04:11:15
  • 转载请务必保留本文链接:https://go.coder-hub.com/68230802.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定