英文:
Pipe command output to Reader through gzip
问题
我正在尝试执行一个shell命令并压缩其输出。
问题是,我之后需要与一个期望Reader的API进行交互。
为此,我尝试了以下简化的代码:
package main
import (
"encoding/hex"
"testing"
"fmt"
"io"
"io/ioutil"
"os/exec"
"compress/gzip"
)
func TestPipe(t *testing.T) {
cmd := exec.Command("echo", "hello_from_echo")
reader, writer := io.Pipe()
gzW := gzip.NewWriter(writer)
cmd.Stdout = gzW
cmd.Start()
go func() {
fmt.Println("Waiting")
cmd.Wait()
fmt.Println("wait done")
// writer.Close()
// gzW.Close()
}()
msg, _ := ioutil.ReadAll( reader )
fmt.Println( hex.EncodeToString( msg ) )
}
问题是ReadAll一直挂起。如果我关闭gzW
,情况并没有真正改变。然而,如果我关闭writer
变量,现在程序会在不挂起的情况下完成,但输出是:
$ go test -run Pipe
Waiting
wait done
1f8b080000096e8800ff
PASS
然而,无论我回显什么,输出都是相同的。如果我在命令行中这样尝试:echo "hello_from_echo" | gzip | hexdump
,输出完全不同,所以这种方法肯定有问题。
有什么线索可以解决这个问题吗?
提前谢谢。
英文:
I'm trying to execute a shell command and compress it's output.
The problem is that I then need to interface with an API that expects a Reader.
For that I tried with the following (simplified code):
package main
import (
"encoding/hex"
"testing"
"fmt"
"io"
"io/ioutil"
"os/exec"
"compress/gzip"
)
func TestPipe(t *testing.T) {
cmd := exec.Command("echo", "hello_from_echo")
reader, writer := io.Pipe()
gzW := gzip.NewWriter(writer)
cmd.Stdout = gzW
cmd.Start()
go func() {
fmt.Println("Waiting")
cmd.Wait()
fmt.Println("wait done")
// writer.Close()
// gzW.Close()
}()
msg, _ := ioutil.ReadAll( reader )
fmt.Println( hex.EncodeToString( msg ) )
}
The problem is that ReadAll hangs forever. If I close gzW
nothing really changes. However, if I close the writer
variable, now the program finishes without hanging, but the output is:
$ go test -run Pipe
Waiting
wait done
1f8b080000096e8800ff
PASS
However, no matter what I echo the output is the same. If I try it from the command line like this: echo "hello_from_echo" | gzip | hexdump
the output is totally different, so there's something wrong with that approach.
Any clue what could be the problem?
Thanks in advance
答案1
得分: 3
你正在错误地关闭gzip写入器和管道写入器的顺序。你需要先关闭gzip.Writer
以刷新任何缓冲区并写入gzip页脚,然后再关闭PipeWriter
以解除ReadAll
的阻塞。另外,添加WaitGroup
可以确保你不会在任何关闭调用上被阻塞。
cmd := exec.Command("echo", "hello_from_echo and more")
pr, pw := io.Pipe()
gzW := gzip.NewWriter(pw)
cmd.Stdout = gzW
cmd.Start()
var wg sync.WaitGroup
wg.Add(1)
go func() {
defer wg.Done()
err := cmd.Wait()
if err != nil {
log.Println(err)
}
gzW.Close()
pw.Close()
}()
buf, err := ioutil.ReadAll(pr)
if err != nil {
t.Fatal(err)
}
wg.Wait()
fmt.Println(hex.EncodeToString(buf))
英文:
You're closing the gzip writer and pipe writer in the wrong order. You need to close the gzip.Writer
to flush any buffers and write the gzip footer, then you can close the PipeWriter
to unblock the ReadAll
. Also adding the WaitGroup
ensures that you're not blocked on any of the close calls.
cmd := exec.Command("echo", "hello_from_echo and more")
pr, pw := io.Pipe()
gzW := gzip.NewWriter(pw)
cmd.Stdout = gzW
cmd.Start()
var wg sync.WaitGroup
wg.Add(1)
go func() {
defer wg.Done()
err := cmd.Wait()
if err != nil {
log.Println(err)
}
gzW.Close()
pw.Close()
}()
buf, err := ioutil.ReadAll(pr)
if err != nil {
t.Fatal(err)
}
wg.Wait()
fmt.Println(hex.EncodeToString(buf))
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论