为什么我的 Golang 程序会创建这么多线程?

huangapple go评论121阅读模式
英文:

Why my program of golang create so many threads?

问题

我的服务器运行了一段时间,创建了大约200个连接并进行了一些计算,然后关闭了。我发现它占用了大约2.7G的内存,并且在几天后仍然没有减少。程序本身并没有占用那么多内存,我通过memstats进行了检查。通过cat /proc/11686/status | grep -i threads命令,我得到了Threads: 177的结果,所以我认为占用这么多内存的原因是创建了太多的线程。为什么go会创建这么多线程?是因为我使用了太多的go func()吗?我确定goroutine没有增加,并且它们正常退出。

PS

我的程序中有很多代码,所以我排除了细节,只保留了主要部分。

我的问题是当go创建一个线程来执行某些操作时。拥有这么多线程是正常的吗?我认为这与代码关系不大。

英文:

My server runned for a time and about 200 connection created and did some calculations and closed, I found that it took up about 2,7G memory and never decreased after serveral days. The program itself didn't occupy that much , And I checked it by memstats. by cat /proc/11686/status | grep -i threads I got Threads: 177,so I think the reason that it took up so much memory is that it created to many threads .Why go create so much threads? Is it because I use too many go func()? And I'm sure goroutines didn't increase and they exited normally.

PS

There is so many code in my program, so I exclude the details, just keep the main

And my problem is when go create a thread to do something. and is it normal to have so many thread? I think it is not concerned much to the code.

main.go

  1. package main
  2. import (
  3. "sanguo/base/log"
  4. "fmt"
  5. "runtime"
  6. "math/rand"
  7. "time"
  8. "net"
  9. "os"
  10. )
  11. type GameServer struct {
  12. Host string
  13. }
  14. func (server *GameServer) Start() {
  15. // load system data
  16. log.Debug("/*************************SREVER START********************************/")
  17. tcpAddr, err := net.ResolveTCPAddr("tcp4", server.Host)
  18. if err != nil {
  19. log.Error(err.Error())
  20. os.Exit(-1)
  21. }
  22. go func(){
  23. for{
  24. select {
  25. case <-time.After(30*time.Second):
  26. LookUp("read memstats")
  27. }
  28. }
  29. }()
  30. listener, err := net.ListenTCP("tcp", tcpAddr)
  31. if err != nil {
  32. log.Error(err.Error())
  33. os.Exit(-1)
  34. }
  35. log.Debug("/*************************SERVER SUCC********************************/")
  36. for {
  37. conn, err := listener.AcceptTCP()
  38. if err != nil {
  39. continue
  40. }
  41. log.Debug("Accept a new connection ", conn.RemoteAddr())
  42. go handleClient(conn)
  43. }
  44. }
  45. func handleClient(conn *net.TCPConn) {
  46. sess := NewSession(conn)
  47. sess.Start()
  48. }
  49. func main() {
  50. rand.Seed(time.Now().Unix())
  51. runtime.GOMAXPROCS(runtime.NumCPU())
  52. log.SetLevel(0)
  53. filew := log.NewFileWriter("log", true)
  54. err := filew.StartLogger()
  55. if err != nil {
  56. fmt.Println("Failed start log",err)
  57. return
  58. }
  59. var server GameServer
  60. server.Host = "127.0.0.1:9999"
  61. server.Start()
  62. }

session.go

  1. package main
  2. import (
  3. "io"
  4. "encoding/binary"
  5. "encoding/json"
  6. "github.com/felixge/tcpkeepalive"
  7. "net"
  8. "sanguo/base/log"
  9. "strings"
  10. "sync"
  11. "time"
  12. )
  13. type Session struct {
  14. conn *net.TCPConn //the tcp connection from client
  15. recvChan chan *bufferedManager.Token //data from client
  16. closeNotiChan chan bool //
  17. ok bool
  18. lock sync.Mutex
  19. }
  20. func NewSession(connection *net.TCPConn) (sess *Session) {
  21. var client Session
  22. client.conn = connection
  23. client.recvChan = make(chan []byte, 1024)
  24. client.closeNotiChan = make(chan bool)
  25. client.ok = true
  26. log.Debug("New Connection", &client)
  27. kaConn, err := tcpkeepalive.EnableKeepAlive(connection)
  28. if err != nil {
  29. log.Debug("EnableKeepAlive err ", err)
  30. } else {
  31. kaConn.SetKeepAliveIdle(120 * time.Second)
  32. kaConn.SetKeepAliveCount(4)
  33. kaConn.SetKeepAliveInterval(5 * time.Second)
  34. }
  35. return &client
  36. }
  37. func (sess *Session) Close() {
  38. sess.lock.Lock()
  39. if sess.ok {
  40. sess.ok = false
  41. close(sess.closeNotiChan)
  42. sess.conn.Close()
  43. log.Trace("Sess Close Succ", sess, sess.uid)
  44. }
  45. sess.lock.Unlock()
  46. }
  47. func (sess *Session) handleRecv() {
  48. defer func(){
  49. if err := recover(); err != nil {
  50. log.Critical("Panic", err)
  51. }
  52. log.Trace("Session Recv Exit", sess, sess.uid)
  53. sess.Close()
  54. }()
  55. ch := sess.recvChan
  56. header := make([]byte, 2)
  57. for {
  58. /**block until recieve len(header)**/
  59. n, err := io.ReadFull(sess.conn, header)
  60. if n == 0 && err == io.EOF {
  61. //Opposite socket is closed
  62. log.Warn("Socket Read EOF And Close", sess)
  63. break
  64. } else if err != nil {
  65. //Sth wrong with this socket
  66. log.Warn("Socket Wrong:", err)
  67. break
  68. }
  69. size := binary.LittleEndian.Uint16(header) + 4
  70. data := make([]byte, size)
  71. n, err = io.ReadFull(sess.conn, t.Data)
  72. if n == 0 && err == io.EOF {
  73. log.Warn("Socket Read EOF And Close", sess)
  74. break
  75. } else if err != nil {
  76. log.Warn("Socket Wrong:", err)
  77. break
  78. }
  79. ch <- data //send data to Client to process
  80. }
  81. }
  82. func (sess *Session) handleDispatch() {
  83. defer func(){
  84. log.Trace("Session Dispatch Exit", sess, sess.uid)
  85. sess.Close()
  86. }()
  87. for {
  88. select {
  89. case msg, _ := <-sess.recvChan:
  90. log.Debug("msg", msg)
  91. sess.SendDirectly("helloworldhellowor", 1)
  92. case <-sess.closeNotiChan:
  93. return
  94. }
  95. }
  96. }
  97. func (sess *Session) Start() {
  98. defer func() {
  99. if err := recover(); err != nil {
  100. log.Critical("Panic", err)
  101. }
  102. }()
  103. go sess.handleRecv()
  104. sess.handleDispatch()
  105. close(sess.recvChan)
  106. log.Warn("Session Start Exit", sess, sess.uid)
  107. }
  108. func (sess *Session) SendDirectly(back interface{}, op int) bool {
  109. back_json, err := json.Marshal(back)
  110. if err != nil {
  111. log.Error("Can't encode json message ", err, back)
  112. return false
  113. }
  114. log.Debug(sess.uid, "OUT cmd:", op, string(back_json))
  115. _, err = sess.conn.Write(back_json)
  116. if err != nil {
  117. log.Error("send fail", err)
  118. return false
  119. }
  120. return true
  121. }

答案1

得分: 7

使用Go语言,你可以创建许多goroutine,但这不应该增加线程的数量。在你的代码中,运行Go代码的线程数量由runtime.NumCPU()限制。

当goroutine需要执行阻塞调用(如系统调用或通过cgo调用C库)时,可能会创建一个线程。在这种情况下,运行时调度器会将运行该goroutine的线程从调度池中移除。如果调度池中的线程少于GOMAXPROCS,那么将创建一个新线程。

你可以在这里找到更多关于它如何工作的信息:
https://softwareengineering.stackexchange.com/questions/222642/are-go-langs-goroutine-pools-just-green-threads/222694#222694

要理解为什么你的代码会生成线程,你需要调查所有导致阻塞系统调用或C调用的代码路径。请注意,与网络相关的调用是非阻塞的,因为它们由标准库自动复用。然而,如果你执行一些磁盘I/O操作或调用外部库,这将生成线程。

例如,你的代码中使用的日志库可能会执行一些阻塞的I/O操作,从而创建线程(特别是如果生成的文件存储在一个慢速设备上)。

英文:

With Go, you can create many goroutines, it should not increase the number of threads. In your code, the number of threads running Go code is capped by runtime.NumCPU().

A thread may be created when the goroutine has to perform a blocking call, such as a system call, or a call to a C library via cgo. In that case, the runtime scheduler removes the thread running the goroutine from its scheduling pool. If the scheduling pool has less threads than GOMAXPROCS, then a new one will be created.

You can find a bit more information about how it works here:
https://softwareengineering.stackexchange.com/questions/222642/are-go-langs-goroutine-pools-just-green-threads/222694#222694

To understand why your code generates threads, you have to investigate all the code paths resulting in blocking system calls or C calls. Note that network related calls are non-blocking, since they are automatically multiplexed by the standard library. However, if you perform some disks I/Os, or call foreign libraries, this will generate threads.

For instance, the logging library used in your code may perform some blocking I/Os resulting in threads being created (especially if the generated files are hosted on a slow device).

huangapple
  • 本文由 发表于 2014年12月22日 18:14:27
  • 转载请务必保留本文链接:https://go.coder-hub.com/27600587.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定