英文:
How to tune the number of goroutines used to download N files?
问题
我正在制作一个 Golang 程序，需要下载 N 个文件，然后对每个文件执行一些任务。特别是对每个 SWF 文件进行一系列操作的流水线：
- 发送 HTTP GET 请求以获取文件
- 将文件保存到本地
- 将文件转换为 PNG 格式
按顺序执行可能非常低效。使用 N 个 goroutine 也可能不是最优解。
如何选择/限制 goroutine 的数量？
英文:
I'm making a Golang program which needs to download N files and then perform some task on each one. In particular there is a pipeline of operation to do on each single SWF file:
- Make an http get request to get the file
- Save it locally
- Convert it in PNG
Doing it sequentially could be very inefficient. Doing it with N goroutines could also be non-optimal.
How do I choose / limit the number of goroutines?
答案1
得分: 2
一个goutine的设计是轻量级的：
> 一个goutine有一个简单的模型：它是在同一地址空间中与其他goutine并发执行的函数。它是轻量级的，成本几乎只是分配堆栈空间。而且堆栈的初始大小很小，所以它们很便宜，并且通过根据需要分配（和释放）堆存储来增长。
限制不在于您生成的goutine的数量，而在于它们所做的事情。
在您的情况下，同时打开文件（以写入下载的流）可能会达到每个进程或通常情况下打开的文件数的限制。
英文:
A goroutine is lightweight by design:
> A goroutine has a simple model: it is a function executing concurrently with other goroutines in the same address space. It is lightweight, costing little more than the allocation of stack space. And the stacks start small, so they are cheap, and grow by allocating (and freeing) heap storage as required.
The limit is not in the number of goroutines you generate, but in what they do.
In your case, opening files (to write a downloaded stream) concurrently could meet the limit of number of opened files per process or in general
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论