英文:
Copy exec.Command output to file as the buffer receives data
问题
我有一个脚本，运行时会将大量文本输出到STDOUT。我想执行这个脚本，并将输出写入文件，而不需要一次性将整个缓冲区保存在内存中（这个脚本一次输出的文本可能有几兆字节）。
下面的代码可以工作，但由于我在多个goroutine中执行，内存消耗会增加到超过5GB，我真的想避免这种情况：
var out bytes.Buffer
cmd := exec.Command("/path/to/script/binary", "arg1", "arg2")
cmd.Stdout = &out
err := cmd.Run()
if err != nil {
log.Fatal(err)
}
out.WriteTo(io) // io是连接到新文件的写入器
理想情况下，当out填满时，我希望将其清空到目标文件中，以保持内存使用量低。我尝试将代码更改为以下方式：
cmd := exec.Command("/path/to/script/binary", "arg1", "arg2")
cmd.Start()
stdout, _ := cmd.StdoutPipe()
r := *bufio.NewReader(stdout)
r.WriteTo(io)
cmd.Wait()
然而，当我打印这些变量时，stdout是<nil>，r是{[0 0 0 0 0...]}，并且r.WriteTo会引发恐慌：invalid memory address or nil pointer dereference。
是否有可能在生成时将cmd的输出写入文件以降低内存使用量？谢谢！
英文:
I have a script that dumps quite a bit of text into STDOUT when run. I'm trying to execute this script and write the output to a file without holding the entire buffer in memory at one time. (We're talking many megabytes of text that this script outputs at one time.)
The following works, but because I'm doing this across several goroutines, my memory consumption shoots up to > 5GB which I would really like to avoid:
var out bytes.Buffer
cmd := exec.Command("/path/to/script/binary", "arg1", "arg2")
cmd.Stdout = &out
err := cmd.Run()
if err != nil {
log.Fatal(err)
}
out.WriteTo(io) // io is the writer connected to the new file
Ideally as out fills up, I want to be emptying it into my target file to keep memory usage low. I've tried changing this to:
cmd := exec.Command("/path/to/script/binary", "arg1", "arg2")
cmd.Start()
stdout, _ := cmd.StdoutPipe()
r := *bufio.NewReader(stdout)
r.WriteTo(io)
cmd.Wait()
However when I print out these variables stdout is <nil>, r is {[0 0 0 0 0...]}, and r.WriteTo panics: invalid memory address or nil pointer dereference.
Is it possible to write the output of cmd as it is generated to keep memory usage down? Thanks!
答案1
得分: 3
为什么你不直接将内容写入文件呢？
file, _ := os.Create("/some/file")
cmd.Stdout = file
或者使用你的 io 变量（顺便说一句，这是一个糟糕的变量名，因为它既是标准库包的名称，又是含糊不清的——它是什么意思？）
cmd.Stdout = io
英文:
Why don't you just write to a file directly?
file, _ := os.Create("/some/file")
cmd.Stdout = file
Or use your io thing (that's a terrible name for a variable, by the way, since it's a) the name of a standard library package, b) ambiguous--what does it mean?)
cmd.Stdout = io
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论