英文:
How can I pipe the stdout of an exec function into the reader of another?
问题
我正在尝试将mongodump的stdout流式传输到S3。我已经正确设置了S3任意长度流的语法,但我不明白如何将这两个函数结合起来。我不想在开始上传到S3之前处理整个mongodump命令。目前我有以下代码:
dumpCmd := exec.Command("mongodump", "--host", "<host>", "--port", "<port>", "--archive")
dumpCmd.Stdout = os.Stdout
uploader := s3manager.NewUploader(session.New(&aws.Config{Region: aws.String("us-east-1")}))
result, err := uploader.Upload(&s3manager.UploadInput{
Body: dumpCmd.Stdout,
Bucket: aws.String("myBucket"),
Key: aws.String("myKey"),
})
if err != nil {
log.Fatalln("Failed to upload", err)
}
log.Println("Successfully uploaded to", result.Location)
不幸的是,dumpCmd.Stdout
是一个写入器(writer),而不是读取器(reader),我不知道如何将写入器的输出导入读取器。
在使用管道(pipe)之后,我现在遇到了一个新的错误,可能需要一个新的问题:
dumpCmd := exec.Command("mongodump", "--host", "<host>", "--port", "<port>", "--archive")
body, err := dumpCmd.StdoutPipe()
if err != nil {
// 处理错误
}
if err := dumpCmd.Start(); err != nil {
// 处理错误
}
uploader := s3manager.NewUploader(session.New(&aws.Config{Region: aws.String("us-east-1")}))
result, err := uploader.Upload(&s3manager.UploadInput{
Body: body,
Bucket: aws.String("myBucket"),
Key: aws.String("myKey"),
})
if err != nil {
log.Fatalln("Failed to upload", err)
}
if err := dumpCmd.Wait(); err != nil {
// 处理错误
}
log.Println("Successfully uploaded to", result.Location)
错误信息如下:
2016/03/10 12:39:18 Failed to upload MultipartUpload: upload multipart failed
upload id: QOWW4jBHH4PKjs1Tloc8dlCTtFN94vDHIJIWJChsrjxLZggScZbRUhM4FU9V.xOnIg9uYnBWqOA1x1xqStfA1p8vdAOHNyUp4gOO5b1gbuXvUitQyLdfFhKg9MnyxsV1
caused by: RequestError: send request failed
caused by: Put https://myBucket/myKey?partNumber=1&uploadId=QOWW4jBHH4PKjs1Tloc8dlCTtFN94vDHIJIWJChsrjxLZggScZbRUhM4FU9V.xOnIg9uYnBWqOA1x1xqStfA1p8vdAOHNyUp4gOO5b1gbuXvUitQyLdfFhKg9MnyxsV1: read |0: illegal seek
exit status 1
请注意,这只是一个翻译,我无法提供关于代码的技术支持。如果你对代码有进一步的问题,请另行提问。
英文:
I'm trying to stream the stdout from mongodump into s3. I have gotten the syntax for the S3 arbitrary-length stream right, but I don't understand how to couple the two functions. I don't want to process the entire mongodump command before I start uploading to S3. This is what I have so far:
dumpCmd := exec.Command("mongodump", "--host", "<host>", "--port", "<port>", "--archive")
dumpCmd.Stdout = os.Stdout
uploader := s3manager.NewUploader(session.New(&aws.Config{Region: aws.String("us-east-1")}))
result, err := uploader.Upload(&s3manager.UploadInput{
Body: dumpCmd.Stdout,
Bucket: aws.String("myBucket"),
Key: aws.String("myKey"),
})
if err != nil {
log.Fatalln("Failed to upload", err)
}
log.Println("Successfully uploaded to", result.Location)
unfortunately dumpCmd.Stdout is a writer, not a reader, and I don't know how to pipe the output of the writer into the reader.
After using a pipe I'm now getting a new error, which may belong in a new question:
dumpCmd := exec.Command("mongodump", "--host", "<host>", "--port", "<port>", "--archive")
body, err := dumpCmd.StdoutPipe()
if err != nil {
// handle error
}
if err := dumpCmd.Start(); err != nil {
// handle error
}
uploader := s3manager.NewUploader(session.New(&aws.Config{Region: aws.String("us-east-1")}))
result, err := uploader.Upload(&s3manager.UploadInput{
Body: body,
Bucket: aws.String("myBucket"),
Key: aws.String("myKey"),
})
if err != nil {
log.Fatalln("Failed to upload", err)
}
if err := dumpCmd.Wait(); err != nil {
// handle error
}
log.Println("Successfully uploaded to", result.Location)
error:
2016/03/10 12:39:18 Failed to upload MultipartUpload: upload multipart failed
upload id: QOWW4jBHH4PKjs1Tloc8dlCTtFN94vDHIJIWJChsrjxLZggScZbRUhM4FU9V.xOnIg9uYnBWqOA1x1xqStfA1p8vdAOHNyUp4gOO5b1gbuXvUitQyLdfFhKg9MnyxsV1
caused by: RequestError: send request failed
caused by: Put https://myBucket/myKey?partNumber=1&uploadId=QOWW4jBHH4PKjs1Tloc8dlCTtFN94vDHIJIWJChsrjxLZggScZbRUhM4FU9V.xOnIg9uYnBWqOA1x1xqStfA1p8vdAOHNyUp4gOO5b1gbuXvUitQyLdfFhKg9MnyxsV1: read |0: illegal seek
exit status 1
答案1
得分: 3
使用管道将命令的输出连接到上传的输入。
这里的棘手问题是,上传程序在请求主体上执行了查找操作,但是管道不支持查找操作。为了解决这个问题,创建了一个包装器,将管道包装起来,隐藏了上传程序的查找方法。
dumpCmd := exec.Command("mongodump", "--host", "<host>", "--port", "<port>", "--archive")
body, err := dumpCmd.StdoutPipe()
if err != nil {
// 处理错误
}
if err := dumpCmd.Start(); err != nil {
// 处理错误
}
// 包装管道,隐藏上传程序的查找方法
bodyWrap := struct {
io.Reader
}{body}
uploader := s3manager.NewUploader(session.New(&aws.Config{Region: aws.String("us-east-1")}))
result, err := uploader.Upload(&s3manager.UploadInput{
Body: bodyWrap,
Bucket: aws.String("net-openwhere-mongodb-snapshots-dev"),
Key: aws.String("myKey"),
})
if err != nil {
log.Fatalln("上传失败", err)
}
if err := dumpCmd.Wait(); err != nil {
// 处理错误
}
log.Println("成功上传至", result.Location)
英文:
Use a pipe to connect the output of the command to the input of the upload.
The tricky issue here is that the uploader seeks on the body, but the seek is not supported by the pipe. To circumvent this, a wrapper is created around the pipe to hide the Seek method from the uploader.
dumpCmd := exec.Command("mongodump", "--host", "<host>", "--port", "<port>", "--archive")
body, err := dumpCmd.StdoutPipe()
if err != nil {
// handle error
}
if err := dumpCmd.Start(); err != nil {
// handle error
}
// Wrap the pipe to hide the seek methods from the uploader
bodyWrap := struct {
io.Reader
}{body}
uploader := s3manager.NewUploader(session.New(&aws.Config{Region: aws.String("us-east-1")}))
result, err := uploader.Upload(&s3manager.UploadInput{
Body: bodyWrap,
Bucket: aws.String("net-openwhere-mongodb-snapshots-dev"),
Key: aws.String("myKey"),
})
if err != nil {
log.Fatalln("Failed to upload", err)
}
if err := dumpCmd.Wait(); err != nil {
// handle error
}
log.Println("Successfully uploaded to", result.Location)
答案2
得分: -1
你可以使用io.Copy
方法(godoc),其定义如下:
func Copy
func Copy(dst Writer, src Reader) (written int64, err error)
Copy函数从src复制到dst,直到在src上达到EOF或发生错误。它返回复制的字节数和在复制过程中遇到的第一个错误(如果有的话)。
成功的Copy返回err == nil,而不是err == EOF。因为Copy被定义为从src读取直到EOF,所以它不将来自Read的EOF视为要报告的错误。
如果src实现了WriterTo接口,则通过调用src.WriteTo(dst)来实现复制。否则,如果dst实现了ReaderFrom接口,则通过调用dst.ReadFrom(src)来实现复制。
英文:
You can use the io.Copy
method (godoc), which is defined as follow:
> func Copy
>
> func Copy(dst Writer, src Reader) (written int64, err error)
>
> Copy copies from src to dst until either EOF is reached on src or an
> error occurs. It returns the number of bytes copied and the first
> error encountered while copying, if any.
>
> A successful Copy returns err == nil, not err == EOF. Because Copy is
> defined to read from src until EOF, it does not treat an EOF from Read
> as an error to be reported.
>
> If src implements the WriterTo interface, the copy is implemented by
> calling src.WriteTo(dst). Otherwise, if dst implements the ReaderFrom
> interface, the copy is implemented by calling dst.ReadFrom(src).
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论