如何将exec函数的stdout导入到另一个读取器中?

huangapple go评论91阅读模式
英文:

How can I pipe the stdout of an exec function into the reader of another?

问题

我正在尝试将mongodump的stdout流式传输到S3。我已经正确设置了S3任意长度流的语法,但我不明白如何将这两个函数结合起来。我不想在开始上传到S3之前处理整个mongodump命令。目前我有以下代码:

dumpCmd := exec.Command("mongodump", "--host", "<host>", "--port", "<port>", "--archive")
dumpCmd.Stdout = os.Stdout

uploader := s3manager.NewUploader(session.New(&aws.Config{Region: aws.String("us-east-1")}))
result, err := uploader.Upload(&s3manager.UploadInput{
    Body:   dumpCmd.Stdout,
    Bucket: aws.String("myBucket"),
    Key:    aws.String("myKey"),
})
if err != nil {
    log.Fatalln("Failed to upload", err)
}

log.Println("Successfully uploaded to", result.Location)

不幸的是,dumpCmd.Stdout是一个写入器(writer),而不是读取器(reader),我不知道如何将写入器的输出导入读取器。

在使用管道(pipe)之后,我现在遇到了一个新的错误,可能需要一个新的问题:

dumpCmd := exec.Command("mongodump", "--host", "<host>", "--port", "<port>", "--archive")
body, err := dumpCmd.StdoutPipe()
if err != nil {
    // 处理错误
}

if err := dumpCmd.Start(); err != nil {
    // 处理错误
}

uploader := s3manager.NewUploader(session.New(&aws.Config{Region: aws.String("us-east-1")}))
result, err := uploader.Upload(&s3manager.UploadInput{
    Body:   body,
    Bucket: aws.String("myBucket"),
    Key:    aws.String("myKey"),
})
if err != nil {
    log.Fatalln("Failed to upload", err)
}

if err := dumpCmd.Wait(); err != nil {
    // 处理错误
}

log.Println("Successfully uploaded to", result.Location)

错误信息如下:

2016/03/10 12:39:18 Failed to upload MultipartUpload: upload multipart failed
    upload id: QOWW4jBHH4PKjs1Tloc8dlCTtFN94vDHIJIWJChsrjxLZggScZbRUhM4FU9V.xOnIg9uYnBWqOA1x1xqStfA1p8vdAOHNyUp4gOO5b1gbuXvUitQyLdfFhKg9MnyxsV1
caused by: RequestError: send request failed
caused by: Put https://myBucket/myKey?partNumber=1&uploadId=QOWW4jBHH4PKjs1Tloc8dlCTtFN94vDHIJIWJChsrjxLZggScZbRUhM4FU9V.xOnIg9uYnBWqOA1x1xqStfA1p8vdAOHNyUp4gOO5b1gbuXvUitQyLdfFhKg9MnyxsV1: read |0: illegal seek
exit status 1

请注意,这只是一个翻译,我无法提供关于代码的技术支持。如果你对代码有进一步的问题,请另行提问。

英文:

I'm trying to stream the stdout from mongodump into s3. I have gotten the syntax for the S3 arbitrary-length stream right, but I don't understand how to couple the two functions. I don't want to process the entire mongodump command before I start uploading to S3. This is what I have so far:

dumpCmd := exec.Command(&quot;mongodump&quot;, &quot;--host&quot;, &quot;&lt;host&gt;&quot;, &quot;--port&quot;, &quot;&lt;port&gt;&quot;, &quot;--archive&quot;)
dumpCmd.Stdout = os.Stdout


uploader := s3manager.NewUploader(session.New(&amp;aws.Config{Region: aws.String(&quot;us-east-1&quot;)}))
result, err := uploader.Upload(&amp;s3manager.UploadInput{
	Body:   dumpCmd.Stdout,
	Bucket: aws.String(&quot;myBucket&quot;),
	Key:    aws.String(&quot;myKey&quot;),
})
if err != nil {
	log.Fatalln(&quot;Failed to upload&quot;, err)
}

log.Println(&quot;Successfully uploaded to&quot;, result.Location)

unfortunately dumpCmd.Stdout is a writer, not a reader, and I don't know how to pipe the output of the writer into the reader.

After using a pipe I'm now getting a new error, which may belong in a new question:

dumpCmd := exec.Command(&quot;mongodump&quot;, &quot;--host&quot;, &quot;&lt;host&gt;&quot;, &quot;--port&quot;, &quot;&lt;port&gt;&quot;, &quot;--archive&quot;)
body, err := dumpCmd.StdoutPipe()
if err != nil {
	// handle error
}

if err := dumpCmd.Start(); err != nil {
	// handle error
}

uploader := s3manager.NewUploader(session.New(&amp;aws.Config{Region: aws.String(&quot;us-east-1&quot;)}))
result, err := uploader.Upload(&amp;s3manager.UploadInput{
	Body:   body,
	Bucket: aws.String(&quot;myBucket&quot;),
	Key:    aws.String(&quot;myKey&quot;),
})
if err != nil {
	log.Fatalln(&quot;Failed to upload&quot;, err)
}

if err := dumpCmd.Wait(); err != nil {
	// handle error
}

log.Println(&quot;Successfully uploaded to&quot;, result.Location)

error:

2016/03/10 12:39:18 Failed to upload MultipartUpload: upload multipart failed
    upload id: QOWW4jBHH4PKjs1Tloc8dlCTtFN94vDHIJIWJChsrjxLZggScZbRUhM4FU9V.xOnIg9uYnBWqOA1x1xqStfA1p8vdAOHNyUp4gOO5b1gbuXvUitQyLdfFhKg9MnyxsV1
caused by: RequestError: send request failed
caused by: Put https://myBucket/myKey?partNumber=1&amp;uploadId=QOWW4jBHH4PKjs1Tloc8dlCTtFN94vDHIJIWJChsrjxLZggScZbRUhM4FU9V.xOnIg9uYnBWqOA1x1xqStfA1p8vdAOHNyUp4gOO5b1gbuXvUitQyLdfFhKg9MnyxsV1: read |0: illegal seek
exit status 1

答案1

得分: 3

使用管道将命令的输出连接到上传的输入。

这里的棘手问题是,上传程序在请求主体上执行了查找操作,但是管道不支持查找操作。为了解决这个问题,创建了一个包装器,将管道包装起来,隐藏了上传程序的查找方法。

dumpCmd := exec.Command("mongodump", "--host", "<host>", "--port", "<port>", "--archive")
body, err := dumpCmd.StdoutPipe()
if err != nil {
    // 处理错误
}

if err := dumpCmd.Start(); err != nil {
    // 处理错误
}

// 包装管道,隐藏上传程序的查找方法
bodyWrap := struct {
    io.Reader
}{body}

uploader := s3manager.NewUploader(session.New(&aws.Config{Region: aws.String("us-east-1")}))
result, err := uploader.Upload(&s3manager.UploadInput{
    Body:   bodyWrap,
    Bucket: aws.String("net-openwhere-mongodb-snapshots-dev"),
    Key:    aws.String("myKey"),
})
if err != nil {
    log.Fatalln("上传失败", err)
}

if err := dumpCmd.Wait(); err != nil {
   // 处理错误
}

log.Println("成功上传至", result.Location)
英文:

Use a pipe to connect the output of the command to the input of the upload.

The tricky issue here is that the uploader seeks on the body, but the seek is not supported by the pipe. To circumvent this, a wrapper is created around the pipe to hide the Seek method from the uploader.

dumpCmd := exec.Command(&quot;mongodump&quot;, &quot;--host&quot;, &quot;&lt;host&gt;&quot;, &quot;--port&quot;, &quot;&lt;port&gt;&quot;, &quot;--archive&quot;)
body, err := dumpCmd.StdoutPipe()
if err != nil {
    // handle error
}

if err := dumpCmd.Start(); err != nil {
    // handle error
}

// Wrap the pipe to hide the seek methods from the uploader
bodyWrap := struct {
    io.Reader
}{body}

uploader := s3manager.NewUploader(session.New(&amp;aws.Config{Region: aws.String(&quot;us-east-1&quot;)}))
result, err := uploader.Upload(&amp;s3manager.UploadInput{
	Body:   bodyWrap,
	Bucket: aws.String(&quot;net-openwhere-mongodb-snapshots-dev&quot;),
	Key:    aws.String(&quot;myKey&quot;),
})
if err != nil {
	log.Fatalln(&quot;Failed to upload&quot;, err)
}

if err := dumpCmd.Wait(); err != nil {
   // handle error
}

log.Println(&quot;Successfully uploaded to&quot;, result.Location)

答案2

得分: -1

你可以使用io.Copy方法(godoc),其定义如下:

func Copy

func Copy(dst Writer, src Reader) (written int64, err error)

Copy函数从src复制到dst,直到在src上达到EOF或发生错误。它返回复制的字节数和在复制过程中遇到的第一个错误(如果有的话)。

成功的Copy返回err == nil,而不是err == EOF。因为Copy被定义为从src读取直到EOF,所以它不将来自Read的EOF视为要报告的错误。

如果src实现了WriterTo接口,则通过调用src.WriteTo(dst)来实现复制。否则,如果dst实现了ReaderFrom接口,则通过调用dst.ReadFrom(src)来实现复制。

英文:

You can use the io.Copy method (godoc), which is defined as follow:

> func Copy
>
> func Copy(dst Writer, src Reader) (written int64, err error)
>
> Copy copies from src to dst until either EOF is reached on src or an
> error occurs. It returns the number of bytes copied and the first
> error encountered while copying, if any.
>
> A successful Copy returns err == nil, not err == EOF. Because Copy is
> defined to read from src until EOF, it does not treat an EOF from Read
> as an error to be reported.
>
> If src implements the WriterTo interface, the copy is implemented by
> calling src.WriteTo(dst). Otherwise, if dst implements the ReaderFrom
> interface, the copy is implemented by calling dst.ReadFrom(src).

huangapple
  • 本文由 发表于 2016年3月11日 01:17:32
  • 转载请务必保留本文链接:https://go.coder-hub.com/35923059.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定