Using Snappy compression on Gob data in Go?

huangapple go评论132阅读模式
英文:

Using Snappy compression on Gob data in Go?

问题

我需要将一个结构保存到磁盘并在以后再次读取,我试图将IO降到最低,但又不花费太长时间来压缩和解压缩文件,所以我打算使用Snappy进行压缩,因为它非常快速和相对高效。

通常情况下,我会在保存到文件时使用gzip压缩gob,像这样:

  1. func (t *Object) Save(filename string) error {
  2. // 打开文件进行写入
  3. fi, err := os.Create(filename)
  4. if err != nil {
  5. return err
  6. }
  7. defer fi.Close()
  8. // 附加gzip写入器
  9. fz := gzip.NewWriter(fi)
  10. defer fz.Close()
  11. // 从gob编码器推送
  12. encoder := gob.NewEncoder(fz)
  13. err = encoder.Encode(t.Classifier)
  14. if err != nil {
  15. return err
  16. }
  17. return nil
  18. }

但是Snappy没有附加到其他所有东西似乎都使用的这些Reader/Writer接口。相反,它只提供基本功能:
https://godoc.org/code.google.com/p/snappy-go/snappy

func Encode(dst, src []byte) ([]byte, error)

func Decode(dst, src []byte) ([]byte, error)

在将Gob数据保存到文件时,使用Snappy包进行压缩的最有效方法是什么(以及读取它)?理想情况下,我不想使用ioutil.ReadAll仅从gob读取器中读取到字节切片,然后再次压缩它,因为这似乎是一种非常繁重的方式,会产生很多浪费的内存。

我承认我不完全理解读取器和写入器接口的工作原理。

英文:

I need to save a structure to disk and read it in again later, I'm trying to keep IO down to a minimum, but also not spend ages compressing and uncompressing the file, so I intend to use Snappy for compression as it's very fast and relatively efficient.

Normally I would gzip compress a gob when saving it to file, like this:

  1. func (t *Object) Save(filename string) error {
  2. // Open file for writing
  3. fi, err := os.Create(filename)
  4. if err != nil {
  5. return err
  6. }
  7. defer fi.Close()
  8. // Attach gzip writer
  9. fz := gzip.NewWriter(fi)
  10. defer fz.Close()
  11. // Push from the gob encoder
  12. encoder := gob.NewEncoder(fz)
  13. err = encoder.Encode(t.Classifier)
  14. if err != nil {
  15. return err
  16. }
  17. return nil
  18. }

But Snappy does not attach to these Reader/Writer interfaces that everything else seems to use. Instead it just provides the basic functionality:
https://godoc.org/code.google.com/p/snappy-go/snappy

func Encode(dst, src []byte) ([]byte, error)

func Decode(dst, src []byte) ([]byte, error)

What would be the most efficient way to use this Snappy package for compressing the Gob data as its being saved to file (and also reading it back)? Ideally I don't want to use ioutil.ReadAll to just read from the gob reader into a slice of bytes and then compress that all over again as that seems to be a very heavy way of doing it which would create a lot of wasted memory.

I admit that I don't fully understand how the reader and writer interfaces work.

答案1

得分: 5

你考虑过使用snappystream包吗?例如,

  1. package main
  2. import (
  3. "encoding/gob"
  4. "fmt"
  5. "os"
  6. "github.com/mreiferson/go-snappystream"
  7. )
  8. type Object struct {
  9. Classifier struct{}
  10. }
  11. func (t *Object) Save(filename string) error {
  12. // 打开文件进行写入
  13. fi, err := os.Create(filename)
  14. if err != nil {
  15. return err
  16. }
  17. defer fi.Close()
  18. // 使用snappy writer
  19. fs := snappystream.NewBufferedWriter(fi)
  20. // 通过gob编码器进行推送
  21. encoder := gob.NewEncoder(fs)
  22. err = encoder.Encode(t.Classifier)
  23. if err != nil {
  24. return err
  25. }
  26. err = fs.Close()
  27. if err != nil {
  28. return err
  29. }
  30. err = fi.Close()
  31. if err != nil {
  32. return err
  33. }
  34. return nil
  35. }
  36. func main() {
  37. obj := Object{}
  38. err := obj.Save("obj.sz")
  39. if err != nil {
  40. fmt.Println(err)
  41. }
  42. }

你可以在这里找到更多关于snappystream包的信息。

英文:

> package snappystream
>
> import "github.com/mreiferson/go-snappystream"
>
> snappystream wraps snappy-go and supplies a Reader and Writer for the
> snappy framed stream format.

Have you considered package snappystream? For example,

  1. package main
  2. import (
  3. "encoding/gob"
  4. "fmt"
  5. "os"
  6. "github.com/mreiferson/go-snappystream"
  7. )
  8. type Object struct {
  9. Classifier struct{}
  10. }
  11. func (t *Object) Save(filename string) error {
  12. // Open file for writing
  13. fi, err := os.Create(filename)
  14. if err != nil {
  15. return err
  16. }
  17. defer fi.Close()
  18. // Attach snappy writer
  19. fs := snappystream.NewBufferedWriter(fi)
  20. // Push from the gob encoder
  21. encoder := gob.NewEncoder(fs)
  22. err = encoder.Encode(t.Classifier)
  23. if err != nil {
  24. return err
  25. }
  26. err = fs.Close()
  27. if err != nil {
  28. return err
  29. }
  30. err = fi.Close()
  31. if err != nil {
  32. return err
  33. }
  34. return nil
  35. }
  36. func main() {
  37. obj := Object{}
  38. err := obj.Save("obj.sz")
  39. if err != nil {
  40. fmt.Println(err)
  41. }
  42. }

huangapple
  • 本文由 发表于 2014年11月12日 13:50:49
  • 转载请务必保留本文链接:https://go.coder-hub.com/26880174.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定