How to measure both before and after byte size and time to compress

huangapple go评论82阅读模式
英文:

How to measure both before and after byte size and time to compress

问题

我想压缩一个字符串(实际上是一个 JSON 响应)

var b bytes.Buffer
gz := gzip.NewWriter(&b)
if _, err := gz.Write([]byte("YourDataHere")); err != nil {
    panic(err)
}

我该如何轻松地输出压缩前后的字节大小,更重要的是,我该如何计算压缩和解压缩回字符串所需的时间?

英文:

I want to gzip a string (it is actually a JSON response)

var b bytes.Buffer
gz := gzip.NewWriter(&b)
if _, err := gz.Write([]byte("YourDataHere")); err != nil {
    panic(err)
}

How can I easily output the size of bytes before and after compression and more importantly how can I time it takes to compress and decompress back to a string?

答案1

得分: 1

你可以按照Nipun Talukdar的评论来计算大小。

len([]byte("YourDataHere")) b.Len()

使用time.Now()time.Since()来获取所花费的时间。

var b bytes.Buffer
input := []byte("YourDataHere")

fmt.Println("Input size : ", len(input))

gz := gzip.NewWriter(&b)

start := time.Now()

gz.Write(input)
if _, err := gz.Flush(); err != nil {
    panic(err)
}

totalTime := time.Since(start)

fmt.Println("Compressed size : ", b.Len(), "\nTime taken : ", totalTime)
gz.Close()

相同的方法也可以用于解压缩。你还可以创建一个支持函数来计时。

func timer(startTime time.Time) {
    totalTime := time.Since(startTime)
    log.Println("Time taken : ", totalTime)
}

用法:在函数的开头使用defer timer(time.Now())

英文:

You can calculate the size as per Nipun Talukdar's comment.

len([]byte("YourDataHere"))
b.Len()

And use time.Now() and time.Since() to get the time taken.

var b bytes.Buffer
input := []byte("YourDataHere")

fmt.Println("Input size : ", len(input))

gz := gzip.NewWriter(&b)


start := time.Now() 

gz.Write(input)
if _, err := gz.Flush(); err != nil {
    	panic(err)
}


totalTime := time.Since(start) 

fmt.Println("Compressed size : ", b.Len(), "\nTime taken : ", totalTime)
gz.Close()

Same method can be applied with unzipping.
You can also create a support function that can do the timing.

func timer(startTime time.Time) {

totalTime := time.Since(startTime)
log.Println("Time taken : ",totalTime)

}

Usage : defer timer(time.Now()) at the start of the function.

答案2

得分: 0

这里有一个在Go语言中如何实现的示例:

https://golang.org/test/bench/go1/gzip_test.go

幸运的是,它是根据BSD许可证发布的...

// 版权所有 2011 年 Go 作者。保留所有权利。
// 使用本源代码受 BSD 风格的许可证管辖,
// 可在 LICENSE 文件中找到该许可证。

// 此基准测试 gzip 和 gunzip 的性能。

package go1

import (
    "bytes"
    gz "compress/gzip"
    "io"
    "io/ioutil"
    "testing"
)

var (
    jsongunz = bytes.Repeat(jsonbytes, 10)
    jsongz   []byte
)

func init() {
    var buf bytes.Buffer
    c := gz.NewWriter(&buf)
    c.Write(jsongunz)
    c.Close()
    jsongz = buf.Bytes()
}

func gzip() {
    c := gz.NewWriter(ioutil.Discard)
    if _, err := c.Write(jsongunz); err != nil {
        panic(err)
    }
    if err := c.Close(); err != nil {
        panic(err)
    }
}

func gunzip() {
    r, err := gz.NewReader(bytes.NewBuffer(jsongz))
    if err != nil {
        panic(err)
    }
    if _, err := io.Copy(ioutil.Discard, r); err != nil {
        panic(err)
    }
    r.Close()
}

func BenchmarkGzip(b *testing.B) {
    b.SetBytes(int64(len(jsongunz)))
    for i := 0; i < b.N; i++ {
        gzip()
    }
}

func BenchmarkGunzip(b *testing.B) {
    b.SetBytes(int64(len(jsongunz)))
    for i := 0; i < b.N; i++ {
        gunzip()
    }
}

希望对你有帮助!

英文:

There are examples of how to do this in Go here

https://golang.org/test/bench/go1/gzip_test.go

Thankfully it's BSD licensed...

// Copyright 2011 The Go Authors.  All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// This benchmark tests gzip and gunzip performance.
package go1 
import (
&quot;bytes&quot;
gz &quot;compress/gzip&quot;
&quot;io&quot;
&quot;io/ioutil&quot;
&quot;testing&quot;
)
var (
jsongunz = bytes.Repeat(jsonbytes, 10) 
jsongz   []byte
)
func init() {
var buf bytes.Buffer
c := gz.NewWriter(&amp;buf)
c.Write(jsongunz)
c.Close()
jsongz = buf.Bytes()
}
func gzip() {
c := gz.NewWriter(ioutil.Discard)
if _, err := c.Write(jsongunz); err != nil {
panic(err)
}
if err := c.Close(); err != nil {
panic(err)
}
}
func gunzip() {
r, err := gz.NewReader(bytes.NewBuffer(jsongz))
if err != nil {
panic(err)
}
if _, err := io.Copy(ioutil.Discard, r); err != nil {
panic(err)
}
r.Close()
}
func BenchmarkGzip(b *testing.B) {
b.SetBytes(int64(len(jsongunz)))
for i := 0; i &lt; b.N; i++ {
gzip()
}
}
func BenchmarkGunzip(b *testing.B) {
b.SetBytes(int64(len(jsongunz)))
for i := 0; i &lt; b.N; i++ {
gunzip()
}
}

答案3

得分: 0

很遗憾,gzip.Writer无法报告写入底层流的压缩字节数。当底层流不在内存中时,情况会变得更加复杂。

为了解决这个问题,我编写了一个“计数io.Writer”,将其放置在gzip.Writer和底层流之间,以便我可以计算和提取已写入的压缩字节数。

Go Playground中尝试以下代码。

package main

import (
	"compress/gzip"
	"fmt"
	"io"
	"os"
)

// countingWriter是一个计算写入总字节数的io.Writer。
type countingWriter struct {
	w     io.Writer
	Count int
}

var _ io.Writer = &countingWriter{}

func newCountingWriter(w io.Writer) *countingWriter {
	return &countingWriter{w: w}
}

func (cw *countingWriter) Write(p []byte) (int, error) {
	n, err := cw.w.Write(p)
	cw.Count += n
	return n, err
}

func ExampleUse(w io.Writer) (int, error) {
	cw := newCountingWriter(w)
	zw, err := gzip.NewWriterLevel(cw, gzip.BestCompression)
	if err != nil {
		return 0, err
	}

	if _, err := zw.Write([]byte("hello world")); err != nil {
		return cw.Count, err
	}

	err = zw.Close()
	return cw.Count, err
}

func main() {
	n, err := ExampleUse(os.Stderr)
	if err != nil {
		panic(err)
	}
	fmt.Printf("写入了%d个字节\n", n)
}
英文:

It's unfortunate that gzip.Writer can't report the number of compressed bytes written to the underlying stream. It gets more complicated when that underlying stream is not in-memory.

To solve this, I wrote a "counting io.Writer" that I place in between gzip.Writer and the underlying stream, so I can count and extract the number of compressed bytes written.

Try out the following code in the Go Playground.

package main

import (
	&quot;compress/gzip&quot;
	&quot;fmt&quot;
	&quot;io&quot;
	&quot;os&quot;
)

// countingWriter is an io.Writer that counts the total bytes written to it.
type countingWriter struct {
	w     io.Writer
	Count int
}

var _ io.Writer = &amp;countingWriter{}

func newCountingWriter(w io.Writer) *countingWriter {
	return &amp;countingWriter{w: w}
}

func (cw *countingWriter) Write(p []byte) (int, error) {
	n, err := cw.w.Write(p)
	cw.Count += n
	return n, err
}

func ExampleUse(w io.Writer) (int, error) {
	cw := newCountingWriter(w)
	zw, err := gzip.NewWriterLevel(cw, gzip.BestCompression)
	if err != nil {
		return 0, err
	}

	if _, err := zw.Write([]byte(&quot;hello world&quot;)); err != nil {
		return cw.Count, err
	}

	err = zw.Close()
	return cw.Count, err
}

func main() {
	n, err := ExampleUse(os.Stderr)
	if err != nil {
		panic(err)
	}
	fmt.Printf(&quot;wrote %d bytes\n&quot;, n)
}

huangapple
  • 本文由 发表于 2016年4月19日 11:00:10
  • 转载请务必保留本文链接:https://go.coder-hub.com/36707746.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定