将命令输出通过gzip传输给Reader。

huangapple go评论93阅读模式
英文:

Pipe command output to Reader through gzip

问题

我正在尝试执行一个shell命令并压缩其输出。
问题是,我之后需要与一个期望Reader的API进行交互。

为此,我尝试了以下简化的代码:

package main

import (
    "encoding/hex"
    "testing"
    "fmt"
    "io"
    "io/ioutil"
    "os/exec"
    "compress/gzip"
)

func TestPipe(t *testing.T) {
    cmd := exec.Command("echo", "hello_from_echo")

    reader, writer := io.Pipe()
    gzW := gzip.NewWriter(writer)

    cmd.Stdout = gzW 
    cmd.Start()
    
    go func() {
        fmt.Println("Waiting")
        cmd.Wait()
        fmt.Println("wait done")
        // writer.Close()
        // gzW.Close()
    }()

    msg, _ := ioutil.ReadAll( reader )
    fmt.Println( hex.EncodeToString( msg ) )
}

问题是ReadAll一直挂起。如果我关闭gzW,情况并没有真正改变。然而,如果我关闭writer变量,现在程序会在不挂起的情况下完成,但输出是:

$ go test -run Pipe
Waiting
wait done
1f8b080000096e8800ff
PASS

然而,无论我回显什么,输出都是相同的。如果我在命令行中这样尝试:echo "hello_from_echo" | gzip | hexdump,输出完全不同,所以这种方法肯定有问题。

有什么线索可以解决这个问题吗?

提前谢谢。

英文:

I'm trying to execute a shell command and compress it's output.
The problem is that I then need to interface with an API that expects a Reader.

For that I tried with the following (simplified code):

package main

import (
    "encoding/hex"
    "testing"
    "fmt"
	"io"
	"io/ioutil"
	"os/exec"
    "compress/gzip"
)

func TestPipe(t *testing.T) {
    cmd := exec.Command("echo", "hello_from_echo")

    reader, writer := io.Pipe()
    gzW := gzip.NewWriter(writer)

    cmd.Stdout = gzW 
    cmd.Start()
    
    go func() {
        fmt.Println("Waiting")
        cmd.Wait()
        fmt.Println("wait done")
        // writer.Close()
        // gzW.Close()
    }()

    msg, _ := ioutil.ReadAll( reader )
    fmt.Println( hex.EncodeToString( msg ) )
}

The problem is that ReadAll hangs forever. If I close gzW nothing really changes. However, if I close the writer variable, now the program finishes without hanging, but the output is:

$ go test -run Pipe
Waiting
wait done
1f8b080000096e8800ff
PASS

However, no matter what I echo the output is the same. If I try it from the command line like this: echo "hello_from_echo" | gzip | hexdump the output is totally different, so there's something wrong with that approach.

Any clue what could be the problem?

Thanks in advance

答案1

得分: 3

你正在错误地关闭gzip写入器和管道写入器的顺序。你需要先关闭gzip.Writer以刷新任何缓冲区并写入gzip页脚,然后再关闭PipeWriter以解除ReadAll的阻塞。另外,添加WaitGroup可以确保你不会在任何关闭调用上被阻塞。

cmd := exec.Command("echo", "hello_from_echo and more")

pr, pw := io.Pipe()
gzW := gzip.NewWriter(pw)

cmd.Stdout = gzW
cmd.Start()

var wg sync.WaitGroup
wg.Add(1)
go func() {
	defer wg.Done()
	err := cmd.Wait()
	if err != nil {
		log.Println(err)
	}
	gzW.Close()
	pw.Close()
}()

buf, err := ioutil.ReadAll(pr)
if err != nil {
	t.Fatal(err)
}
wg.Wait()
fmt.Println(hex.EncodeToString(buf))
英文:

You're closing the gzip writer and pipe writer in the wrong order. You need to close the gzip.Writer to flush any buffers and write the gzip footer, then you can close the PipeWriter to unblock the ReadAll. Also adding the WaitGroup ensures that you're not blocked on any of the close calls.

cmd := exec.Command("echo", "hello_from_echo and more")

pr, pw := io.Pipe()
gzW := gzip.NewWriter(pw)

cmd.Stdout = gzW
cmd.Start()

var wg sync.WaitGroup
wg.Add(1)
go func() {
	defer wg.Done()
	err := cmd.Wait()
	if err != nil {
		log.Println(err)
	}
	gzW.Close()
	pw.Close()
}()

buf, err := ioutil.ReadAll(pr)
if err != nil {
	t.Fatal(err)
}
wg.Wait()
fmt.Println(hex.EncodeToString(buf))

huangapple
  • 本文由 发表于 2016年10月29日 03:55:09
  • 转载请务必保留本文链接:https://go.coder-hub.com/40312349.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定