Why does it not create many threads when many goroutines are blocked in writing file in golang?

huangapple go评论77阅读模式
英文:

Why does it not create many threads when many goroutines are blocked in writing file in golang?

问题

我们知道,在Go语言中,当goroutine需要执行阻塞调用(如系统调用或通过cgo调用C库)时,可能会创建一个线程。以下是一些测试代码:

package main

import (
    "io/ioutil"
    "os"
    "runtime"
    "strconv"
)

func main() {
    runtime.GOMAXPROCS(2)
    data, err := ioutil.ReadFile("./55555.log")
    if err != nil {
        println(err)
        return
    }
    for i := 0; i < 200; i++ {
        go func(n int) {
            for {
                err := ioutil.WriteFile("testxxx"+strconv.Itoa(n), []byte(data), os.ModePerm)
                if err != nil {
                    println(err)
                    break
                }
            }
        }(i)
    }
    select {}
}

当我运行它时,并没有创建很多线程。

➜ =99=[root /root]$ cat /proc/9616/status | grep -i thread
Threads:    5

有什么想法吗?

英文:

As we know in go, a thread may be created when the goroutine has to perform a blocking call, such as a system call, or a call to a C library via cgo. Some test code:

   package main
    
   import (
        &quot;io/ioutil&quot;
        &quot;os&quot;
        &quot;runtime&quot;
        &quot;strconv&quot;
    )
    
    func main() {
        runtime.GOMAXPROCS(2)
        data, err := ioutil.ReadFile(&quot;./55555.log&quot;)
        if err != nil {
            println(err)
            return
        }
        for i := 0; i &lt; 200; i++ {
            go func(n int) {
                for {
                    err := ioutil.WriteFile(&quot;testxxx&quot;+strconv.Itoa(n), []byte(data), os.ModePerm)
                    if err != nil {
                        println(err)
                        break
                    }
                }
            }(i)
        }
        select {}
    }

When I run it, it didn't create many threads.

➜ =99=[root /root]$ cat /proc/9616/status | grep -i thread
Threads:	5

Any ideas?

答案1

得分: 12

我稍微修改了你的程序,以输出一个更大的块。

package main

import (
	"io/ioutil"
	"os"
	"runtime"
	"strconv"
)

func main() {
	runtime.GOMAXPROCS(2)
	data := make([]byte, 128*1024*1024)
	for i := 0; i < 200; i++ {
		go func(n int) {
			for {
				err := ioutil.WriteFile("testxxx"+strconv.Itoa(n), []byte(data), os.ModePerm)
				if err != nil {
					println(err)
					break
				}
			}
		}(i)
	}
	select {}
}

这样就会显示超过200个线程,正如你所期望的。

$ cat /proc/17033/status | grep -i thread
Threads:	203

所以我认为在你原来的测试中,系统调用退出得太快,无法显示你期望的效果。

英文:

I altered your program slightly to output a much bigger block

package main

import (
	&quot;io/ioutil&quot;
	&quot;os&quot;
	&quot;runtime&quot;
	&quot;strconv&quot;
)

func main() {
	runtime.GOMAXPROCS(2)
	data := make([]byte, 128*1024*1024)
	for i := 0; i &lt; 200; i++ {
		go func(n int) {
			for {
				err := ioutil.WriteFile(&quot;testxxx&quot;+strconv.Itoa(n), []byte(data), os.ModePerm)
				if err != nil {
					println(err)
					break
				}
			}
		}(i)
	}
	select {}
}

This then shows >200 threads as you expected

$ cat /proc/17033/status | grep -i thread
Threads:	203

So I think the syscalls were exiting too quickly in your original test to show the effect you were expecting.

答案2

得分: 5

一个 goroutine 是一个轻量级的线程,它不等同于操作系统线程。语言规范将其定义为“在同一地址空间内独立并发控制的线程”。

引用 runtime 包的文档:

>GOMAXPROCS 变量限制了可以同时执行用户级 Go 代码的操作系统线程的数量。在代表 Go 代码进行系统调用的阻塞线程数量上没有限制;它们不计入 GOMAXPROCS 的限制。

仅仅因为你启动了 200 个 goroutine,并不意味着会为它们启动 200 个线程。你将 GOMAXPROCS 设置为 2,这意味着可以同时运行 2 个“活动”的 goroutine。如果一个 goroutine 被阻塞(例如 I/O 等待),可能会产生新的线程。你没有提到你的测试文件有多大,你启动的 goroutine 可能会很快完成写入操作。

Effective Go 博客文章将它们定义为:

>它们被称为 goroutines,因为现有的术语——线程、协程、进程等——传达了不准确的内涵。一个 goroutine 有一个简单的模型:它是在同一地址空间中与其他 goroutine 并发执行的函数。它是轻量级的,成本几乎只有分配堆栈空间。而且堆栈的初始大小很小,所以它们很廉价,并且通过根据需要分配(和释放)堆存储来增长。
>
>Goroutine 被复用到多个操作系统线程上,因此如果一个 goroutine 阻塞,比如在等待 I/O 时,其他 goroutine 会继续运行。它们的设计隐藏了许多线程创建和管理的复杂性。

英文:

A goroutine is a lightweight thread, it is not equivalent to an operating system thread. The Language specification specifies it as an "independent concurrent thread of control within the same address space".

Quoting from the documentation of package runtime:

>The GOMAXPROCS variable limits the number of operating system threads that can execute user-level Go code simultaneously. There is no limit to the number of threads that can be blocked in system calls on behalf of Go code; those do not count against the GOMAXPROCS limit.

Just because you start 200 goroutines, it doesn't mean 200 threads will be started for them. You set GOMAXPROCS to 2 which means there can be 2 "active" goroutines running at the same time. New threads may be spawed if a goroutine gets blocked (e.g. I/O wait). You didn't mention how big your test file is, goroutines you start might finish writing it too quickly.

The Effective Go blog article defines them as:

>They're called goroutines because the existing terms—threads, coroutines, processes, and so on—convey inaccurate connotations. A goroutine has a simple model: it is a function executing concurrently with other goroutines in the same address space. It is lightweight, costing little more than the allocation of stack space. And the stacks start small, so they are cheap, and grow by allocating (and freeing) heap storage as required.
>
>Goroutines are multiplexed onto multiple OS threads so if one should block, such as while waiting for I/O, others continue to run. Their design hides many of the complexities of thread creation and management.

答案3

得分: 2

问题4056讨论了如何限制实际创建的线程数(而不是goroutine)。

Go 1.2在提交665feee中引入了线程限制管理。

你可以在pkg/runtime/crash_test.go#L128-L134中看到一个测试,用于检查是否实际达到了创建的线程数:

func TestThreadExhaustion(t *testing.T) {
    output := executeTest(t, threadExhaustionSource, nil)
    want := "runtime: program exceeds 10-thread limit\nfatal error: thread exhaustion"
    if !strings.HasPrefix(output, want) {
        t.Fatalf("output does not start with %q:\n%s", want, output)
    }
}

同样的文件中有一个示例,用于为给定的goroutine创建一个实际的线程,使用runtime.LockOSThread()

func testInNewThread(name string) {
    c := make(chan bool)
    go func() {
        runtime.LockOSThread()
        test(name)
        c <- true
    }()
    <-c
}
英文:

The issue 4056 discusses how to limit the number of actual threads (not goroutine) created.

Go 1.2 introduced that thread limit management in commit 665feee.

You can see a test to check if the number of thread created is actually reached or not in pkg/runtime/crash_test.go#L128-L134:

func TestThreadExhaustion(t *testing.T) {
    output := executeTest(t, threadExhaustionSource, nil)
    want := &quot;runtime: program exceeds 10-thread limit\nfatal error: thread exhaustion&quot;
    if !strings.HasPrefix(output, want) {
        t.Fatalf(&quot;output does not start with %q:\n%s&quot;, want, output)
    }
}

That same file has an example to create an actual thread (for a given goroutine), using runtime.LockOSThread():

func testInNewThread(name string) {
    c := make(chan bool)
    go func() {
        runtime.LockOSThread()
        test(name)
        c &lt;- true
    }()
    &lt;-c
}

huangapple
  • 本文由 发表于 2015年1月28日 14:56:36
  • 转载请务必保留本文链接:https://go.coder-hub.com/28186361.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定