英文:
Goroutines not Using Max CPU and Cores
问题
我正在实现我的第一个Golang
应用程序,使用GoRoutines
时遇到了一些问题,无法充分利用最大CPU和核心
,我不太清楚原因。
当使用诸如htop
之类的工具时,CPU
没有充分利用其最大功率,一次只有1..4个线程
处于活动状态。此外,所有核心
都处于活动状态,但利用率在25%-40%
左右。
我使用了以下代码来获取要实例化的goroutines
数量:
func MaxParallelism() int {
maxProcs := runtime.GOMAXPROCS(0)
numCPU := runtime.NumCPU()
if maxProcs < numCPU {
return maxProcs
}
return numCPU
}
这是我设置应用程序的方式:
// goroutines的公共通道
tasks := make(chan *exec.Cmd, 64)
// 生成最大核心数(8)的goroutines
var wg sync.WaitGroup
cores := MaxParallelism()
for i := 0; i < cores; i++ {
wg.Add(1)
go func(num int, w *sync.WaitGroup) {
defer w.Done()
var (
out []byte
err error
)
for cmd := range tasks {
out, err = cmd.Output()
if err != nil {
fmt.Printf("无法获取stdout:", err)
}
// ...
}
}(i, &wg)
}
// 生成任务
for i := 0; i < 100000; i++ {
tasks <- exec.Command(cmd1, args...)
tasks <- exec.Command(cmd2, args...)
}
close(tasks)
// 等待工作线程完成
wg.Wait()
我分享了两个执行应用程序时的htop
截图:
我不知道这是否有帮助,但我是通过Intellij Idea
启动它的。
如何正确使用最大CPU和核心?
提前感谢。
英文:
I'm realizing my very first Golang
application and I'm having some issues on using MAX CPU & Cores
when using GoRoutines
and I don't really know why.
When using a tool such as htop
, CPU
isn't used at its max power and only 1..4 threads
are active at time. Also, all cores
are active but they are around 25%-40%
utilization.
I used:
func MaxParallelism() int {
maxProcs := runtime.GOMAXPROCS(0)
numCPU := runtime.NumCPU()
if maxProcs < numCPU {
return maxProcs
}
return numCPU
}
In order to get the number of goroutines
to be instantiated.
Here's How I set up the application:
//Common Channel for the goroutines
tasks := make(chan *exec.Cmd, 64)
//Spawning Max Cores Number goroutines (8)
var wg sync.WaitGroup
cores := MaxParallelism()
for i := 0; i < cores; i++ {
wg.Add(1)
go func(num int, w *sync.WaitGroup) {
defer w.Done()
var (
out []byte
err error
)
for cmd := range tasks {
out, err = cmd.Output()
if err != nil {
fmt.Printf("Can't get stdout:", err)
}
. . .
}
}(i, &wg)
}
//Generate Tasks
for i := 0; i < 100000; i++ {
tasks <- exec.Command(cmd1, args...)
tasks <- exec.Command(cmd2, args...)
}
close(tasks)
// wait for the workers to finish
wg.Wait()
I share two screenshots of htop
while executing the application
I don't know If it May help but I'm launching it through Intellij Idea
.
How do I use Max CPU properly and Cores?
Thanks in advance.
答案1
得分: 4
Goroutines和线程不同,因此不应该期望有任何CPU亲和性。Goroutines相对于线程的一些优势包括:
- 在典型系统上,你可以运行更多的Goroutines而不是线程。
- Goroutines具有可增长的分段堆栈。
- Goroutines的启动时间比线程更快。
- Goroutines内置了安全地在它们之间进行通信的原语(通道)。
- Goroutines允许你在共享数据结构时避免使用互斥锁。
- Goroutines被复用到少量的操作系统线程上,而不是一对一的映射。
- 你可以编写大规模并发的服务器而不必使用事件驱动编程。
对于你在评论中提出的问题,没有确定的答案。正如其他人提到的,这取决于你的代码做什么。如果你的代码涉及I/O操作,比如读写文件,那么你可能永远无法使用100%的CPU。因为无论你启动多少个Goroutines,I/O操作都是慢的。相反,如果你的Goroutine是一个紧密循环的计算任务,那么8个Goroutines很可能会完全占用你的8个CPU。
英文:
Goroutines and threads are not the same. Ergo you should not expect any CPU affinity. See more for details here http://tleyden.github.io/blog/2014/10/30/goroutines-vs-threads/.
> Here are some of the advantages of Goroutines over threads:
>
> * You can run more goroutines on a typical system than you can threads.
> * Goroutines have growable segmented stacks.
> * Goroutines have a faster startup time than threads.
> * Goroutines come with built-in primitives to communicate safely between themselves (channels).
> * Goroutines allow you to avoid having to resort to mutex locking when sharing data structures.
> * Goroutines are multiplexed onto a small number of OS threads, rather than a 1:1 mapping.
> * You can write massively concurrent servers without having to resort to evented programming.
EDIT:
Answer to your question in the comments. There is no definitive answer. As others mentioned, it depends on what your code does. You may never end up using 100% CPU if you, for example, do I/O, which is slow. So no matter how many routines you start, I/O is slow. Contrarily, if your goroutine has a very tight loop doing just some computation then it's likely that 8 goroutines will consume your 8 CPUs completely.
答案2
得分: 2
在阅读了一些资料、得到了一些提示并尝试了一些方法后,发现代码编写得很好,没有实际的问题。
基本上,Go
的扩展性非常好,生成的 tasks
越多,使用的 CPU
资源就越多。
例如,将 for
循环的范围设置得更高,比如 1,000,000
,可以使 Go
使用所有可用的核心,利用它们的能力达到约 55%
。
希望对某人有所帮助。
英文:
After reading around, getting some hints and trying some stuff, it came out that the code was written just fine and there were not real problems caused by it.
Essentially, Go
scales pretty good and, the more tasks
you generate, the more CPU
resources are used.
For instance, setting an higher range on the for
, e.g. 1.000.000
, makes Go
use all the cores
available at ~55%
of their possibilities.
I hope it will help someone.
答案3
得分: 0
记录一下,这是一段可能会使你的CPU达到最大负载的代码。我同时运行了256个例程(#1例程和for循环内生成的例程)。
如果只执行一个例程(#1例程),你会发现只使用了一个CPU核心:
package main
func main() {
for {}
}
如果执行两个例程(#1例程和生成的例程),你会发现使用了两个CPU核心:
package main
func main() {
go func(){
for {}
}()
for {}
}
Go语言会自动将例程分配到CPU核心上。
值得注意的是,Go例程和函数不是完全相同的概念。
英文:
For the record, here's a code which will likely max out your CPU. I run concurrently 256 routines (#1 routine + those spawned within the for loop).
package main
func main() {
for i := 0; i < 255; i++ {
go func() {
for {}
}()
}
for {}
}
<img src="https://i.stack.imgur.com/WfL4e.png" />
If you execute only one routine (the #1 routine), you'll see only one CPU core is used:
package main
func main() {
for {}
}
<img src="https://i.stack.imgur.com/OzmRw.png" />
If you execute 2 routines (the #1 routine + the spawned one), you'll see 2 CPU cores are used:
package main
func main() {
go func(){
for {}
}()
for {}
}
<img src="https://i.stack.imgur.com/8KHCI.png" />
Go scales routines across CPU cores automatically.
<a href="https://stackoverflow.com/a/53389196/8555937">Worth noting that go routines are not the same as functions.</a>
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论