如何将可执行二进制文件的输出写入内存而不是磁盘？

huangapple
go评论126阅读模式
英文:

How to write the output of an executable binary into memory rather than disk

问题

我得到了一个类似下面的二进制文件：

  1. > ./my_bin raw.avi output_file.avi

output_file.avi 是我想要的结果，在任务成功时，终端会打印一些详细信息，例如：

版权所有 2022 公司名称... 成功。

我想在我的代码中运行这个命令，并将 output_file.avi 重定向到某个字节数组中，这样我就不必从磁盘上读取它并删除它了。我的方法看起来像下面的 Golang 代码片段：

  1. func wrongOne(stdin []byte) ([]byte, error) {
  2.     inBuf := bytes.NewBuffer(stdin)
  3.     outBuf := bytes.NewBuffer(nil)
  4.     cmd := exec.Command("./my_bin", "/dev/stdin", "/dev/stdout")
  5.     cmd.Stdin = inBuf
  6.     cmd.Stdout = outBuf
  7.     err := cmd.Run()
  8.     if err != nil {
  9.         return nil, err
  10.     }
  11.     return outBuf.Bytes(), nil // 错误的
  12. }

然而，返回的字节数组比下面的方法更长，这导致 MD5 校验失败。

  1. func correctOne(stdin []byte) ([]byte, error) {
  2.     inBuf := bytes.NewBuffer(stdin)
  3.     cmd := exec.Command("./my_bin", "/dev/stdin", "output_file")
  4.     cmd.Stdin = inBuf
  5.     err := cmd.Run()
  6.     if err != nil {
  7.         return nil, err
  8.     }
  9.     return os.ReadFile("output_file")
  10. }

wrongOne 函数可以修改为以下代码以得到正确的结果：

  1. func modifiedWrongOne(stdin []byte) ([]byte, error) {
  2.     inBuf := bytes.NewBuffer(stdin)
  3.     outBuf := bytes.NewBuffer(nil)
  4.     cmd := exec.Command("./my_bin", "/dev/stdin", "/dev/stdout")
  5.     cmd.Stdin = inBuf
  6.     cmd.Stdout = outBuf
  7.     err := cmd.Run()
  8.     if err != nil {
  9.         return nil, err
  10.     }
  11.     correct, _ := correctOne(stdin)
  12.     return outBuf.Bytes()[:len(correct)], nil // 不同之处
  13. }

我推测输出的详细信息包含在 /dev/stdout 中，所以 wrongOne 函数不起作用。即，

wrongOne 的输出 = correctOne 的输出 + []byte{"版权所有 2022 公司名称... 成功。"}

有没有办法在不将其保存为文件并从磁盘上读取的情况下，将 output_file.avi 放入管道中？谢谢！

英文:

I got a binary that works like the below:

  1. > ./my_bin raw.avi output_file.avi

output_file.avi is what I want, some verbose information will print in the terminal when the job is succeeded, like:
> Copyright 2022 Company Inc... Success.

I want to run this command inside my code and redirect the output_file.avi into some byte array so that I don't have to read it from disk and delete it. My approach looks like the below Golang snippet:

  1. func wrongOne(stdin []byte) ([]byte, error) {
  2. 	inBuf := bytes.NewBuffer(stdin)
  3. 	outBuf := bytes.NewBuffer(nil)
  4. 	cmd := exec.Command("./my_bin", "/dev/stdin", "/dev/stdout")
  5. 	cmd.Stdin = inBuf
  6. 	cmd.Stdout = outBuf
  7. 	err := cmd.Run()
  8. 	if err != nil {
  9. 		return nil, err
  10. 	}
  11. 	return outBuf.Bytes(), nil // wrong
  12. }

However, the return byte array is longer than the below approach, which leads to failure on the MD5 check.

  1. func correctOne(stdin []byte) ([]byte, error) {
  2. 	inBuf := bytes.NewBuffer(stdin)
  3. 	cmd := exec.Command("./my_bin", "/dev/stdin", "output_file")
  4. 	cmd.Stdin = inBuf
  5. 	err := cmd.Run()
  6. 	if err != nil {
  7. 		return nil, err
  8. 	}
  9. 	return os.ReadFile("output_file")
  10. }

the wrongOne function can be modified to following code to be correct:

  1. func modifiedWrongOne(stdin []byte) ([]byte, error) {
  2. 	inBuf := bytes.NewBuffer(stdin)
  3. 	outBuf := bytes.NewBuffer(nil)
  4. 	cmd := exec.Command("./my_bin", "/dev/stdin", "/dev/stdout")
  5. 	cmd.Stdin = inBuf
  6. 	cmd.Stdout = outBuf
  7. 	err := cmd.Run()
  8. 	if err != nil {
  9. 		return nil, err
  10. 	}
  11. 	correct, _ := correctOne(stdin)
  12. 	return outBuf.Bytes()[:len(correct)], nil // diff
  13. }

I presume that the output verbose information is included in the /dev/stdout so that the wrongOne function doesn't works. i.e.,

> the output of wrongOne = the output of correctOne + []byte{"Copyright 2022 Company Inc... Success."}

Is there any solution that I can get the output_file.avi in the pipe without save it as file and read it from disk? Thanks!

答案1

得分: 4

该命令将版权声明写入标准输出(stdout)。为了避免将版权声明与输出文件混合在一起，可以使用除了/dev/stdout之外的其他文件作为输出文件。

下面的函数使用Cmd.ExtraFiles将一个管道连接到子进程的文件描述符(fd) 3。该函数将数据从管道复制到一个字节缓冲区，并将这些字节返回给调用者。

  1. func otherOne(stdin []byte) ([]byte, error) {
  2.     r, w, err := os.Pipe()
  3.     if err != nil {
  4.         return nil, err
  5.     }
  6.     defer r.Close()
  7.     defer w.Close()
  9.     cmd := exec.Command("./my_bin", "/dev/stdin", "/dev/fd/3")
  10.     cmd.Stdin = bytes.NewReader(stdin)
  11.     cmd.ExtraFiles = []*os.File{w} // 第一个文件是文件描述符 3
  12.     if err := cmd.Start(); err != nil {
  13.         return nil, err
  14.     }
  15.     w.Close()
  16.     var outbuf bytes.Buffer
  17.     if _, err := io.Copy(&outbuf, r); err != nil {
  18.         return nil, err
  19.     }
  20.     if err := cmd.Wait(); err != nil {
  21.         return nil, err
  22.     }
  23.     return outbuf.Bytes(), nil
  24. }
英文:

The command writes the copyright notice to stdout. To avoid commingling the copyright notice with the output file, use a file other than /dev/stdout as the output file.

The function below uses Cmd.ExtraFiles to connect a pipe to fd 3 in the child process. The function copies data from the pipe to a byte buffer and returns those bytes to the caller.

  1. func otherOne(stdin []byte) ([]byte, error) {
  2. 	r, w, err := os.Pipe()
  3. 	if err != nil {
  4. 		return nil, err
  5. 	}
  6. 	defer r.Close()
  7. 	defer w.Close()
  9. 	cmd := exec.Command("./my_bin", "/dev/stdin", "/dev/fd/3")
  10. 	cmd.Stdin = bytes.NewReader(stdin)
  11. 	cmd.ExtraFiles = []*os.File{w} // The first file is fd 3.
  12.     if err := cmd.Start(); err != nil {
  13. 		return nil, err
  14. 	}
  15. 	w.Close()
  16. 	var outbuf bytes.Buffer
  17. 	if _, err := io.Copy(&outbuf, r); err != nil {
  18. 		return nil, err
  19. 	}
  20. 	if err := cmd.Wait(); err != nil {
  21. 		return nil, err
  22. 	}
  23. 	return outbuf.Bytes(), nil
  24. }

答案2

得分: 0

几个月后，我找到了另一种解决这个问题的方法。基本思路与Cerise类似，即使用/dev/fd/3来重定向输出文件。之后，我们将/dev/fd/3重定向到/dev/stdout，将详细日志重定向到/dev/stderr，分别使用3>&1和1>&2。还添加了一个名为gen.sh的附加脚本。以下是解决方案：

  1. #gen.sh
  2. ./mybin /dev/stdin /dev/fd/3 3>&1 1>&2
  1. // gen.go
  2. func gen(stdin []byte) ([]byte, error) {
  3. 	inBuf := bytes.NewBuffer(stdin)
  4. 	outBuf := bytes.NewBuffer(nil)
  5. 	cmd := exec.Command("gen.sh")
  6. 	cmd.Stdin = inBuf
  7. 	cmd.Stdout = outBuf
  8. 	cmd.Stderr = os.Stderr
  9. 	err := cmd.Run()
  10. 	if err != nil {
  11. 		return nil, err
  12. 	}
  13. 	return outBuf.Bytes(), nil
  14. }
英文:

After months, I figure out another way to solve this problem. The basic idea is similar with Cerise, i.e., using /dev/fd/3 to redirect the output file. After that, we redirect /dev/fd/3 to /dev/stdout, verbose log to /dev/stderr
by 3>&1, 1>&2, respectively. An additional gen.sh is added. Here's the solution:

  1. #gen.sh
  2. ./mybin /dev/stdin /dev/fd/3 3>&1 1>&2
  1. // gen.go
  2. func gen(stdin []byte) ([]byte, error) {
  3. 	inBuf := bytes.NewBuffer(stdin)
  4. 	outBuf := bytes.NewBuffer(nil)
  5. 	cmd := exec.Command("./gen.sh")
  6. 	cmd.Stdin = inBuf
  7. 	cmd.Stdout = outBuf
  8. 	cmd.Stderr = os.Stderr
  9. 	err := cmd.Run()
  10. 	if err != nil {
  11. 		return nil, err
  12. 	}
  13. 	return outBuf.Bytes(), nil
  14. }

huangapple
  • 本文由 发表于 2022年8月31日 13:21:39
  • 转载请务必保留本文链接：https://go.coder-hub.com/73551178.html
  • bash
  • go
  • linux
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定