英文:
How to replace goroutines loop by channels in Go
问题
我有一个循环，每次迭代都有一个新的通道来源需要处理。好的，最好是展示我的代码。我有一个文件列表，我想要对每个文件进行类似于"tail -f"的操作。我正在使用github.com/ActiveState/tail包。
for _, tailFile := range files {
t, _ := tail.TailFile(tailFile, c)
// 每个文件一个goroutine
go func() {
for line := range t.Lines { // t.Lines是一个通道
// 在这里进行一些操作
}
}()
}
我可能有成千上万个文件，我想要并行运行我的"tail"操作。正如你所看到的，我的程序将会有成千上万个goroutine。这个循环能够改成只使用一个goroutine和通道吗？
英文:
I have a loop, for each iteration I have a new channel source, that I should handle. Ok, it's better to show my code. I have list of files, each file I want to tail (like tail -f). I am using github.com/ActiveState/tail package.
for _, tailFile := range files {
t, _ := tail.TailFile(tailFile, c)
// Goroutine per tailing file
go func() {
for line := range t.Lines { // t.Lines is a channel
// Do some magic here
}
}()
}
I can have thousands of files and I want to run my tail in parallel. As you see my program will have thousands of goroutines. Can this loop be changed to channels, with only 1 goroutine?
答案1
得分: 4
在博客文章pipelines中，你会找到一种类似的方法（每个文件一个goroutine）。
在parallel.go中的MD5All实现中，为每个文件启动一个新的goroutine。在一个包含许多大文件的目录中，这可能会分配超过机器可用内存的内存。
我们可以通过限制并行读取文件的数量来限制这些分配。在bounded.go中，我们通过创建固定数量的goroutine来读取文件来实现这一点。
我们的流水线现在有三个阶段：遍历目录树，读取和处理文件，以及收集摘要。
如果你发现自己受到由过多的goroutine分配的内存限制，你可以组织自己的代码使用有限数量的goroutine。（goroutine本身是廉价的，但为“magic”部分分配的内存可能很大）
英文:
You will find a similar approach (one goroutine per file) in the blog post pipelines.
> The MD5All implementation in parallel.go starts a new goroutine for each file. In a directory with many large files, this may allocate more memory than is available on the machine.
>
> We can limit these allocations by bounding the number of files read in parallel. In bounded.go, we do this by creating a fixed number of goroutines for reading files.
Our pipeline now has three stages: walk the tree, read and digest the files, and collect the digests.
You could organize your own code to use a limited number of goroutines, if you find yourself limited by memory allocated by a too large number of goroutines. (the goroutine itself is cheap, but the memory allocated for the "magic" part can be big)
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论