英文:
When will Go scheduler create a new M and P?
问题
刚学习了golang的GMP模型，现在我理解了goroutines、OS线程以及golang上下文/处理器是如何相互配合的。但我仍然不明白何时会创建M和P。
举个例子，我有一个测试代码，在数据库上运行一些操作，有两个测试用例（两批goroutines）：
func Test_GMP(t *testing.T) {
for _ = range []struct {
name string
}{
{"first batch"},
{"second batch"},
} {
goroutineSize := 50
done := make(chan error, goroutineSize)
for i := 0; i < goroutineSize; i++ {
go func() {
// 执行一些数据库操作...
// 每个goroutine应该在这里被阻塞一段时间...
// 传播结果
done <- nil
}()
}
for i := 0; i < goroutineSize; i++ {
select {
case err := <-done:
assert.NoError(t, err)
case <-time.After(10 * time.Second):
t.Fatal("timeout waiting for txFunc goroutine")
}
}
close(done)
}
}
据我理解，如果需要，会创建M。在第一批goroutines中，会创建8个（我的计算机上的虚拟核心数）OS线程，而第二批将只重用这8个OS线程，而不会创建新的。这样理解对吗？
如果您能提供更多关于这个主题的资料或博客，我将不胜感激。
英文:
Just learned golang GMP model, now I understand how goroutines, OS threads, and golang contexts/processors cooperate with each other. But I still don't understand when will an M and P be created?
For example, I have a test code to run some operations on DB and there are two test cases (two batches of goroutines):
func Test_GMP(t *testing.T) {
for _ = range []struct {
name string
}{
{"first batch"},
{"second batch"},
} {
goroutineSize := 50
done := make(chan error, goroutineSize)
for i := 0; i < goroutineSize; i++ {
go func() {
// do some databases operations...
// each goroutine should be blocked here for some time...
// propogate the result
done <- nil
}()
}
for i := 0; i < goroutineSize; i++ {
select {
case err := <-done:
assert.NoError(t, err)
case <-time.After(10 * time.Second):
t.Fatal("timeout waiting for txFunc goroutine")
}
}
close(done)
}
}
In my understanding, if M is created in need. In the first batch of goroutines, 8 (the number of virtual cores on my computer) OS threads will be created and the second batch will just reuse the 8 OS threads without creating new ones. Is that correct?
Appreciate if you can provide more materials or blogs on this topic.
答案1
得分: 3
M只有在你的进程不阻塞或没有任何系统调用时才可重用。在你的情况下，你的go func()内部有阻塞任务。因此，M的数量不会限制为8（我的计算机上的虚拟核心数）。第一批将被阻塞并从P中移除，并等待阻塞进程完成，而新的M将与P关联起来。
-
通过
Go func()创建一个goroutine； -
有两个队列存储G，一个是本地调度器P的本地队列，一个是全局G队列。新创建的G将保存在P的本地队列中，如果P的本地队列已满，则保存在全局队列中；
-
G只能在M中运行，一个M必须持有一个P，M和P是1：1的关系。M将从P的本地队列中弹出一个可执行的G。如果本地队列为空，将认为其他MP组合窃取了一个可执行的G来执行；
-
M调度G执行的过程是一个循环机制；
-
当M执行系统调用或剩余的阻塞操作时，M将被阻塞，如果有一些正在执行的g，运行时将从P中移除该线程M，然后创建一个新的操作系统线程（如果有空闲线程可用于多路复用空闲线程）来为该P提供服务；
-
当M系统调用结束时，该G将尝试获取一个空闲的P来执行，并将其放入该P的本地队列中。如果获取到P，则该线程M进入睡眠状态，将其添加到空闲线程中，然后将该G放入全局队列中。
-
P的数量：
环境变量$GOMAXPROCS是由运行时方法gomaxprocs()在调度环境变量时确定的。在GO1.5之后，默认情况下，GomaxProcs将设置为可用的核心数，在默认情况下为1。这意味着在任何时候只能同时运行$GOMAXPROCS个goroutine。 -
M的数量：
GO语言本身的限制：当GO程序启动时，最大M的数量将设置为最大M的数量。然而，内核很难支持这么多线程，所以可以忽略这个限制。在runtime/debug中有一个SetMaxThreads函数，可以设置最大M的数量。当一个M被阻塞时，会创建一个新的M。
M和P的数量没有绝对的关系，一个M被阻塞时，P会创建或切换到另一个M，所以即使默认的P数量为1，可能会有很多M存在。
请参考以下链接获取更多详细信息：
英文:
M is reusable only if your processes are not blocking or not any sys-calls. In your case you have blocking tasks inside your go func(). So, number of M will not be limited to 8 (the number of virtual cores on my computer). First batch will block and remove from P and wait for blocking processes get finished while new M create an associate with P.
> 1. We create a goroutine through Go func ();
>
> 2. There are two queues that store G, one is the local queue of local scheduler P, one is the global G queue. The newly created G will be
> saved in the local queue in the P, and if the local queues of P are
> full, they will be saved in the global queue;
>
> 3. G can only run in m, one m must hold a P, M and P are 1: 1
> relationship. M will pop up a executable G from the local queue of P.
> If the local queue is empty, you will think that other MP combinations
> steals an executable G to execute;
>
> 4. A process executed by M Scheduling G is a loop mechanism;
>
> 5. When M executes syscall or the remaining blocking operation, M will block, if there are some g in execution, Runtime will remove this
> thread M from P, then create one The new operating system thread (if
> there is an idle thread available to multiplex idle threads) to serve
> this P;
>
> 6. When the M system call ends, this G will try to get an idle P execute and put it into this P's local queue. If you get P, then this
> thread m becomes a sleep state, add it to the idle thread, and then
> this G will be placed in the global queue.
>
> 1. P Quantity:
>
> The environment variable $ GomaxProcs is determined by the Runtime
> method gomaxprocs () when the environment variable is scheduled. After
> GO1.5, GomaxProcs will be set by default to the available cores, and
> before default it is 1.This means that only $ GOMAXPROCS Goroutine is
> run at the same time at any time executed.
>
> 2. M quantity:
>
> The GO language itself limits: When the GO program starts, the maximum
> number of M will set the maximum number of M. However, the kernel is
> difficult to support so many threads, so this limit can be ignored.
> SetMaxThreads function in runtime / debug, set the maximum number of M
> A M blocking, you will create new M.
>
> The number of M and P has no absolute relationship, one m block, p
> will create or switch another M, so even if the default number of P is
> 1, there may be many M out.
Please refer following for more details,
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论