Go:如何在不复制的情况下将数组的指针传递给gob?

huangapple go评论69阅读模式
英文:

Go: Passing pointers of an array to gob without copying?

问题

我有一个非常非常大的地图数组(不是切片),然后我尝试对其进行编码。我真的需要避免复制数组,但我无法弄清楚如何做到这一点。

到目前为止,我有以下代码:

func doSomething() {
    var mygiantvar [5]map[string]Searcher
    mygiantvar = Load()
    Save(`file.gob.gz`, &mygiantvar)
}

func Save(filename string, variable *[5]map[string]Searcher) error {
    // 打开文件进行写入
    fi, err := os.Create(filename)
    if err != nil {
        return err
    }
    defer fi.Close()
    // 连接gzip写入器
    fz := gzip.NewWriter(fi)
    defer fz.Close()
    // 通过gob编码器进行编码
    encoder := gob.NewEncoder(fz)
    err = encoder.Encode(*variable)
    if err != nil {
        return err
    }
    return nil
}

根据我的理解,这将把mygiantvar的指针传递给Save函数,Save函数保存第一个副本。但是,整个数组肯定会被复制到encoder.Encode中,然后在许多其他函数中进行复制,对吗?

这个mygiantvar变量的大小可能达到10GB。因此,它必须避免被复制。

但是也许只有实际的数组5部分被复制,而其中的地图是数组内的指针,因此会复制指向地图的指针数组而不是地图本身?我对此一无所知-这一切都非常令人困惑。

有什么想法吗?

英文:

I have a very, very large array (not slice) of maps that I am then trying to encode. I really need to avoid making a copy of the array but I can't figure out how to do this.

So I far I have this:

func doSomething() {
 var mygiantvar [5]map[string]Searcher
 mygiantvar = Load()
 Save(`file.gob.gz`, &mygiantvar)
}

func Save(filename string, variable *[5]map[string]Searcher) error {
	// Open file for writing
	fi, err := os.Create(filename)
	if err !=nil {
		return err
	}
	defer fi.Close()
	// Attach gzip writer
	fz := gzip.NewWriter(fi)
	defer fz.Close()
	// Push from the gob encoder
	encoder := gob.NewEncoder(fz)
	err = encoder.Encode(*variable)
	if err !=nil {
		return err
	}
	return nil
}

From my understanding that will pass a pointer of mygiantvar to Save, which saves the first copy. But then the entire array will surely be copied into encoder.Encode which will then copy it around many more functions, right?

This mygiantvar variable will be something like 10GB in size. So it must avoid being copied ever.

But then again perhaps only the actual array 5 part is copied but the maps inside of this are pointers inside an array, so the array of pointers to maps would be copied instead of the maps themselves? I have no idea about this - it's all very confusing.

Any ideas?

答案1

得分: 2

请注意,Encoder.Encode将传递一个interface{}

func (enc *Encoder) Encode(v interface{}) error {

这意味着你将传递给它的是一个指向任何类型的指针,就像我在"what is the meaning of interface{} in golang?"中所描述的那样(参见"Why can't I assign a *Struct to an *Interface?")。

接口值并不是具体结构体的值(因为它具有可变大小,这是不可能的),而是一种指针(更准确地说是指向结构体和类型的指针)。

Go:如何在不复制的情况下将数组的指针传递给gob?

这意味着它不会复制你的整个映射(或者在这里是你的数组)。由于数组是一个值,你可以在调用Encode()时对其进行切片以避免任何复制:

err = encoder.Encode(*variable[:])

参见"Go Slices: usage and internals"。

这也是使用数组创建切片的语法:

x := [3]string{"Лайка", "Белка", "Стрелка"}
s := x[:] // 引用 x 的存储的切片

如果这样不起作用,你可以保留*variable(这里是一个数组:[5]map[string]Searcher),因为映射类型是引用类型,就像指针或切片一样:复制的内容不会很大。参见"Go maps in action"。

当传递给interface{}时,数组将被复制,但映射的内容不会被复制。参见这个[kbd>play.golang.org</kbd>示例:

package main

import "fmt"

func main() {
    var a [1]map[string]int
    a[0] = make(map[string]int)
    a[0]["test"] = 0
    modify(a)
    fmt.Println(a)
}

func modify(arr interface{}) {
    a := arr.([1]map[string]int)
    a[0]["test"] = -1
}

输出:

[map[test:-1]]
英文:

Note that Encoder.Encode will pass around an interface{}.

func (enc *Encoder) Encode(v interface{}) error {

That means a kind of a pointer to whatever you will be passing to it, as I described in "what is the meaning of interface{} in golang?"
(see also "Why can't I assign a *Struct to an *Interface?")

> An interface value isn't the value of the concrete struct (as it has a variable size, this wouldn't be possible), but it's a kind of pointer (to be more precise a pointer to the struct and a pointer to the type)

Go:如何在不复制的情况下将数组的指针传递给gob?

That means it won't copy the full content of your map (or here of your array).
Since array is a value, you could slice it to avoid any copy during the call to Encode():

err = encoder.Encode(*variable[:])

See "Go Slices: usage and internals"

> This is also the syntax to create a slice given an array:

x := [3]string{&quot;Лайка&quot;, &quot;Белка&quot;, &quot;Стрелка&quot;}
s := x[:] // a slice referencing the storage of x

If that doesn't work, you can keep *variable (here an array: [5]map[string]Searcher), as map types are reference types, like pointers or slices: the copy won't be huge.
See "Go maps in action".

While the array will be copied when passed to interface{}, the map content won't be copied.
See this <kbd>play.golang.org</kbd> example:

package main

import &quot;fmt&quot;

func main() {
	var a [1]map[string]int
	a[0] = make(map[string]int)
	a[0][&quot;test&quot;] = 0
	modify(a)
	fmt.Println(a)
}

func modify(arr interface{}) {
	a := arr.([1]map[string]int)
	a[0][&quot;test&quot;] = -1
}

Output:

[map[test:-1]]

huangapple
  • 本文由 发表于 2014年8月3日 13:47:32
  • 转载请务必保留本文链接:https://go.coder-hub.com/25102030.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定