如何通过通道传递压缩字节?

huangapple go评论85阅读模式
英文:

How to pass compressed bytes through channel?

问题

我正在尝试从缓冲读取器压缩文件,并通过字节通道传递压缩字节,但结果很差 :), 这是我目前想到的,显然这不起作用...

func Compress(r io.Reader) (<-chan byte) {
    c := make(chan byte)
    go func(){
        var wBuff bytes.Buffer
        rBuff := make([]byte, 1024)
        writer := zlib.NewWriter(*wBuff)
        for {
            n, err := r.Read(rBuff)
            if err != nil && err != io.EOF { panic(err) }
            if n == 0 { break }
            writer.Write(rBuff) // 压缩并写入压缩数据
            // 如何通过通道发送已写入的压缩字节?
            // 据我了解,wBuff 最终将包含整个压缩数据?
        }
        writer.Close()
        close(c) // 表示没有更多的数据
    }()
    return c
}

请谅解,因为我对Go语言非常陌生。

英文:

I'm trying to compress file from buffered reader and pass compressed bytes through byte channel, but with poor results :), here's what I came up till now, obviously this don't works...

func Compress(r io.Reader) (&lt;-chan byte) {
    c := make(chan byte)
    go func(){
        var wBuff bytes.Buffer
        rBuff := make([]byte, 1024)
        writer := zlib.NewWriter(*wBuff)
        for {
            n, err := r.Read(rBuff)
            if err != nil &amp;&amp; err != io.EOF { panic(err) }
            if n == 0 { break }
            writer.Write(rBuff) // Compress and write compressed data
            // How to send written compressed bytes through channel?
            // as fas as I understand wBuff will eventually contain
            // whole compressed data?
        }
        writer.Close()
        close(c) // Indicate that no more data follows
    }()
    return c
}

Please bear with me, as I'm very new to Go

答案1

得分: 3

我建议使用[]byte而不是byte。这样更高效。由于并发内存访问,可能需要通过通道发送缓冲区的副本,而不是发送[]byte缓冲区本身。

您可以定义一个type ChanWriter chan []byte,并让它实现io.Writer接口。然后将ChanWriter传递给zlib.NewWriter

您可以为压缩操作创建一个goroutine,然后立即从Compress函数返回ChanWriter的通道。如果没有goroutine,则函数没有理由返回一个通道,首选的返回类型是io.Reader

Compress函数的返回类型应该更改为类似chan <- BytesWithError的东西。在这种情况下,ChanWriter可以定义为type ChanWriter chan BytesWithError

英文:

I suggest to use []byte instead of byte. It is more efficient. Because of concurrent memory accesses it may be necessary to send a copy of the buffer through the channel rather than sending the []byte buffer itself.

You can define a type ChanWriter chan []byte and let it implement the io.Writer interface. Then pass the ChanWriter to zlib.NewWriter.

You can create a goroutine for doing the compression and then immediately return the ChanWriter's channel from your Compress function. If there is no goroutine then there is no reason for the function to return a channel and the preferred return type is io.Reader.

The return type of the Compress function should be changed into something like chan &lt;-BytesWithError. In this case ChanWriter can be defined as type ChanWriter chan BytesWithError.

答案2

得分: 2

通过一个通道逐个发送字节并不是特别高效的。另一种更有用的方法是返回一个实现了io.Reader接口的对象,通过从原始的io.Reader读取一个块并在返回之前压缩其输出来实现Read()方法。

英文:

Sending bytes one by one down a channel is not going to be particularly efficient. Another approach that may be more useful would be to return an object implementing the io.Reader interface, implementing the Read() method by reading a block from a original io.Reader and compressing its output before returning it.

答案3

得分: 1

你的writer.Write(rBuff)语句总是写入len(rBuff)个字节,即使n != len(rBuff)

writer.Write(rBuff[:n])

此外,你的Read循环是

for {
	n, err := r.Read(rBuff)
	if err != nil && err != io.EOF {
		panic(err)
	}
	if n == 0 {
		break
	}
	writer.Write(rBuff[:n])
	// ...
}

等同于

for {
	n, err := r.Read(rBuff)
	if err != nil && err != io.EOF {
		panic(err)
	}
	// !(err != nil && err != io.EOF)
	// !(err != nil) || !(err != io.EOF)
	// err == nil || err == io.EOF
	if err == nil || err == io.EOF {
		if n == 0 {
			break
		}
	}
	writer.Write(rBuff[:n])
	// ...
}

循环在err == nil && n == 0时提前退出。

相反,写成

for {
	n, err := r.Read(rBuf)
	if err != nil {
		if err != io.EOF {
			panic(err)
		}
		if n == 0 {
			break
		}
	}
	writer.Write(rBuf[:n])
	// ...
}
英文:

Your writer.Write(rBuff) statement always writes len(rBuff) bytes, even when n != len(rBuff).

writer.Write(rBuff[:n])

Also, your Read loop is

for {
	n, err := r.Read(rBuff)
	if err != nil &amp;&amp; err != io.EOF {
		panic(err)
	}
	if n == 0 {
		break
	}
	writer.Write(rBuff[:n])
	// ...
}

which is equivalent to

for {
	n, err := r.Read(rBuff)
	if err != nil &amp;&amp; err != io.EOF {
		panic(err)
	}
	// !(err != nil &amp;&amp; err != io.EOF)
	// !(err != nil) || !(err != io.EOF)
	// err == nil || err == io.EOF
	if err == nil || err == io.EOF {
		if n == 0 {
			break
		}
	}
	writer.Write(rBuff[:n])
	// ...
}

The loop exits prematurely if err == nil &amp;&amp; if n == 0.

Instead, write

for {
	n, err := r.Read(rBuf)
	if err != nil {
		if err != io.EOF {
			panic(err)
		}
		if n == 0 {
			break
		}
	}
	writer.Write(rBuf[:n])
	// ...
}

答案4

得分: 0

好的,我找到了一个可行的解决方案:(请随时指出可以改进的地方,或者我做错了什么?)

func Compress(r io.Reader) (<-chan byte) {
    c := make(chan byte)

    go func(){
        var wBuff bytes.Buffer
        rBuff := make([]byte, 1024)
        writer := zlib.NewWriter(&wBuff)
        for {
            n, err := r.Read(rBuff)

            if err != nil {
                if err != io.EOF {
                    panic(err)
                }
                if n == 0 {
                    break
                }
            }

            writer.Write(rBuff[:n])

            for _, v := range wBuff.Bytes() {
                c <- v
            }

            wBuff.Truncate(0)
        }

        writer.Close()

        for _, v := range wBuff.Bytes() {
            c <- v
        }

        close(c) // 表示没有更多的数据跟随
    }()

    return c
}
英文:

Ok, I've found working solution: (Feel free to indicate where it can be improved, or maybe I'm doing something wrong?)

func Compress(r io.Reader) (&lt;-chan byte) {
    c := make(chan byte)

    go func(){
        var wBuff bytes.Buffer
        rBuff := make([]byte, 1024)
        writer := zlib.NewWriter(&amp;wBuff)
        for {
            n, err := r.Read(rBuff)

            if err != nil {
                if err != io.EOF {
                    panic(err)
                }
                if n == 0 {
                    break
                }
            }

            writer.Write(rBuff[:n])

            for _, v := range wBuff.Bytes() {
                c &lt;- v
            }

            wBuff.Truncate(0)
        }

        writer.Close()

        for _, v := range wBuff.Bytes() {
            c &lt;- v
        }

        close(c) // Indicate that no more data follows
    }()

    return c
}

huangapple
  • 本文由 发表于 2013年5月21日 08:48:01
  • 转载请务必保留本文链接:https://go.coder-hub.com/16660446.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定