英文:
Using Snappy compression on Gob data in Go?
问题
我需要将一个结构保存到磁盘并在以后再次读取，我试图将IO降到最低，但又不花费太长时间来压缩和解压缩文件，所以我打算使用Snappy进行压缩，因为它非常快速和相对高效。
通常情况下，我会在保存到文件时使用gzip压缩gob，像这样：
func (t *Object) Save(filename string) error {
// 打开文件进行写入
fi, err := os.Create(filename)
if err != nil {
return err
}
defer fi.Close()
// 附加gzip写入器
fz := gzip.NewWriter(fi)
defer fz.Close()
// 从gob编码器推送
encoder := gob.NewEncoder(fz)
err = encoder.Encode(t.Classifier)
if err != nil {
return err
}
return nil
}
但是Snappy没有附加到其他所有东西似乎都使用的这些Reader/Writer接口。相反，它只提供基本功能：
https://godoc.org/code.google.com/p/snappy-go/snappy
func Encode(dst, src []byte) ([]byte, error)
func Decode(dst, src []byte) ([]byte, error)
在将Gob数据保存到文件时，使用Snappy包进行压缩的最有效方法是什么（以及读取它）？理想情况下，我不想使用ioutil.ReadAll仅从gob读取器中读取到字节切片，然后再次压缩它，因为这似乎是一种非常繁重的方式，会产生很多浪费的内存。
我承认我不完全理解读取器和写入器接口的工作原理。
英文:
I need to save a structure to disk and read it in again later, I'm trying to keep IO down to a minimum, but also not spend ages compressing and uncompressing the file, so I intend to use Snappy for compression as it's very fast and relatively efficient.
Normally I would gzip compress a gob when saving it to file, like this:
func (t *Object) Save(filename string) error {
// Open file for writing
fi, err := os.Create(filename)
if err != nil {
return err
}
defer fi.Close()
// Attach gzip writer
fz := gzip.NewWriter(fi)
defer fz.Close()
// Push from the gob encoder
encoder := gob.NewEncoder(fz)
err = encoder.Encode(t.Classifier)
if err != nil {
return err
}
return nil
}
But Snappy does not attach to these Reader/Writer interfaces that everything else seems to use. Instead it just provides the basic functionality:
https://godoc.org/code.google.com/p/snappy-go/snappy
func Encode(dst, src []byte) ([]byte, error)
func Decode(dst, src []byte) ([]byte, error)
What would be the most efficient way to use this Snappy package for compressing the Gob data as its being saved to file (and also reading it back)? Ideally I don't want to use ioutil.ReadAll to just read from the gob reader into a slice of bytes and then compress that all over again as that seems to be a very heavy way of doing it which would create a lot of wasted memory.
I admit that I don't fully understand how the reader and writer interfaces work.
答案1
得分: 5
你考虑过使用snappystream包吗？例如，
package main
import (
"encoding/gob"
"fmt"
"os"
"github.com/mreiferson/go-snappystream"
)
type Object struct {
Classifier struct{}
}
func (t *Object) Save(filename string) error {
// 打开文件进行写入
fi, err := os.Create(filename)
if err != nil {
return err
}
defer fi.Close()
// 使用snappy writer
fs := snappystream.NewBufferedWriter(fi)
// 通过gob编码器进行推送
encoder := gob.NewEncoder(fs)
err = encoder.Encode(t.Classifier)
if err != nil {
return err
}
err = fs.Close()
if err != nil {
return err
}
err = fi.Close()
if err != nil {
return err
}
return nil
}
func main() {
obj := Object{}
err := obj.Save("obj.sz")
if err != nil {
fmt.Println(err)
}
}
你可以在这里找到更多关于snappystream包的信息。
英文:
> package snappystream
>
> import "github.com/mreiferson/go-snappystream"
>
> snappystream wraps snappy-go and supplies a Reader and Writer for the
> snappy framed stream format.
Have you considered package snappystream? For example,
package main
import (
"encoding/gob"
"fmt"
"os"
"github.com/mreiferson/go-snappystream"
)
type Object struct {
Classifier struct{}
}
func (t *Object) Save(filename string) error {
// Open file for writing
fi, err := os.Create(filename)
if err != nil {
return err
}
defer fi.Close()
// Attach snappy writer
fs := snappystream.NewBufferedWriter(fi)
// Push from the gob encoder
encoder := gob.NewEncoder(fs)
err = encoder.Encode(t.Classifier)
if err != nil {
return err
}
err = fs.Close()
if err != nil {
return err
}
err = fi.Close()
if err != nil {
return err
}
return nil
}
func main() {
obj := Object{}
err := obj.Save("obj.sz")
if err != nil {
fmt.Println(err)
}
}
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论