为什么我的 Golang 程序会创建这么多线程?

huangapple go评论96阅读模式
英文:

Why my program of golang create so many threads?

问题

我的服务器运行了一段时间,创建了大约200个连接并进行了一些计算,然后关闭了。我发现它占用了大约2.7G的内存,并且在几天后仍然没有减少。程序本身并没有占用那么多内存,我通过memstats进行了检查。通过cat /proc/11686/status | grep -i threads命令,我得到了Threads: 177的结果,所以我认为占用这么多内存的原因是创建了太多的线程。为什么go会创建这么多线程?是因为我使用了太多的go func()吗?我确定goroutine没有增加,并且它们正常退出。

PS

我的程序中有很多代码,所以我排除了细节,只保留了主要部分。

我的问题是当go创建一个线程来执行某些操作时。拥有这么多线程是正常的吗?我认为这与代码关系不大。

英文:

My server runned for a time and about 200 connection created and did some calculations and closed, I found that it took up about 2,7G memory and never decreased after serveral days. The program itself didn't occupy that much , And I checked it by memstats. by cat /proc/11686/status | grep -i threads I got Threads: 177,so I think the reason that it took up so much memory is that it created to many threads .Why go create so much threads? Is it because I use too many go func()? And I'm sure goroutines didn't increase and they exited normally.

PS

There is so many code in my program, so I exclude the details, just keep the main

And my problem is when go create a thread to do something. and is it normal to have so many thread? I think it is not concerned much to the code.

main.go

package main

import (
	"sanguo/base/log"
	"fmt"
	"runtime"
	"math/rand"
	"time"
	"net"
	"os"
)

type GameServer struct {
	Host   string
}


func (server *GameServer) Start() {
	// load system data
	log.Debug("/*************************SREVER START********************************/")

	tcpAddr, err := net.ResolveTCPAddr("tcp4", server.Host)
	if err != nil {
		log.Error(err.Error())
		os.Exit(-1)
	}
	go func(){
		for{
			select {
			case <-time.After(30*time.Second):
				LookUp("read memstats")
			}
		}
	}()
	listener, err := net.ListenTCP("tcp", tcpAddr)
	if err != nil {
		log.Error(err.Error())
		os.Exit(-1)
	}
	log.Debug("/*************************SERVER SUCC********************************/")
	for {
		conn, err := listener.AcceptTCP()
		if err != nil {
			continue
		}
		log.Debug("Accept a new connection ", conn.RemoteAddr())
		go handleClient(conn)
	}
}

func handleClient(conn *net.TCPConn) {
	sess := NewSession(conn)
	sess.Start()
}

func main() {
	rand.Seed(time.Now().Unix())

	runtime.GOMAXPROCS(runtime.NumCPU())

	log.SetLevel(0)
	
	filew := log.NewFileWriter("log", true)
	err := filew.StartLogger()
	if err != nil {
		fmt.Println("Failed start log",err)
		return
	}

	var server GameServer
	server.Host = "127.0.0.1:9999"
	server.Start()
}

session.go

package main

import (
	"io"
	"encoding/binary"
	"encoding/json"
	"github.com/felixge/tcpkeepalive"
	"net"
	"sanguo/base/log"
	"strings"
	"sync"
	"time"
)


type Session struct {

	conn *net.TCPConn //the tcp connection from client

	recvChan      chan *bufferedManager.Token //data from client
	closeNotiChan chan bool   //

	ok   bool
	lock sync.Mutex

}


func NewSession(connection *net.TCPConn) (sess *Session) {
	var client Session

	client.conn = connection

	client.recvChan = make(chan []byte, 1024)
	client.closeNotiChan = make(chan bool)
	client.ok = true

	log.Debug("New Connection", &client)

	kaConn, err := tcpkeepalive.EnableKeepAlive(connection)
	if err != nil {
		log.Debug("EnableKeepAlive err ", err)
	} else {
		kaConn.SetKeepAliveIdle(120 * time.Second)
		kaConn.SetKeepAliveCount(4)
		kaConn.SetKeepAliveInterval(5 * time.Second)
	}
	return &client
}


func (sess *Session) Close() {
	sess.lock.Lock()
	if sess.ok {
		sess.ok = false
		close(sess.closeNotiChan)
		sess.conn.Close()
		log.Trace("Sess Close Succ", sess, sess.uid)
	}
	sess.lock.Unlock()
}

func (sess *Session) handleRecv() {
	defer func(){
		if err := recover(); err != nil {
			log.Critical("Panic", err)
		}
		log.Trace("Session Recv Exit", sess, sess.uid)
		sess.Close()
	}()
	ch := sess.recvChan
	header := make([]byte, 2)
	for {
		/**block until recieve len(header)**/
		n, err := io.ReadFull(sess.conn, header)
		if n == 0 && err == io.EOF {
			//Opposite socket is closed
			log.Warn("Socket Read EOF And Close", sess)
			break
		} else if err != nil {
			//Sth wrong with this socket
			log.Warn("Socket Wrong:", err)
			break
		}
		size := binary.LittleEndian.Uint16(header) + 4
		data := make([]byte, size)
		n, err = io.ReadFull(sess.conn, t.Data)
		if n == 0 && err == io.EOF {
			log.Warn("Socket Read EOF And Close", sess)
			break
		} else if err != nil {
			log.Warn("Socket Wrong:", err)
			break
		}
		ch <- data //send data to Client to process
	}
}

func (sess *Session) handleDispatch() {
	defer func(){
		log.Trace("Session Dispatch Exit",  sess, sess.uid)
		sess.Close()
	}()
	for {
		select {
		case msg, _ := <-sess.recvChan:
			log.Debug("msg", msg)
            sess.SendDirectly("helloworldhellowor", 1)

		case <-sess.closeNotiChan:
				return
		}
	}
}

func (sess *Session) Start() {
	defer func() {
		if err := recover(); err != nil {
			log.Critical("Panic", err)
		}
	}()
	go sess.handleRecv()

	sess.handleDispatch()

	close(sess.recvChan)
	log.Warn("Session Start Exit", sess, sess.uid)
}


func (sess *Session) SendDirectly(back interface{}, op int) bool {
	back_json, err := json.Marshal(back)
	if err != nil {
		log.Error("Can't encode json message ", err, back)
		return false
	}
	log.Debug(sess.uid, "OUT cmd:", op, string(back_json))
	_, err = sess.conn.Write(back_json)
	if err != nil {
		log.Error("send fail", err)
		return false
	}
	return true
}

答案1

得分: 7

使用Go语言,你可以创建许多goroutine,但这不应该增加线程的数量。在你的代码中,运行Go代码的线程数量由runtime.NumCPU()限制。

当goroutine需要执行阻塞调用(如系统调用或通过cgo调用C库)时,可能会创建一个线程。在这种情况下,运行时调度器会将运行该goroutine的线程从调度池中移除。如果调度池中的线程少于GOMAXPROCS,那么将创建一个新线程。

你可以在这里找到更多关于它如何工作的信息:
https://softwareengineering.stackexchange.com/questions/222642/are-go-langs-goroutine-pools-just-green-threads/222694#222694

要理解为什么你的代码会生成线程,你需要调查所有导致阻塞系统调用或C调用的代码路径。请注意,与网络相关的调用是非阻塞的,因为它们由标准库自动复用。然而,如果你执行一些磁盘I/O操作或调用外部库,这将生成线程。

例如,你的代码中使用的日志库可能会执行一些阻塞的I/O操作,从而创建线程(特别是如果生成的文件存储在一个慢速设备上)。

英文:

With Go, you can create many goroutines, it should not increase the number of threads. In your code, the number of threads running Go code is capped by runtime.NumCPU().

A thread may be created when the goroutine has to perform a blocking call, such as a system call, or a call to a C library via cgo. In that case, the runtime scheduler removes the thread running the goroutine from its scheduling pool. If the scheduling pool has less threads than GOMAXPROCS, then a new one will be created.

You can find a bit more information about how it works here:
https://softwareengineering.stackexchange.com/questions/222642/are-go-langs-goroutine-pools-just-green-threads/222694#222694

To understand why your code generates threads, you have to investigate all the code paths resulting in blocking system calls or C calls. Note that network related calls are non-blocking, since they are automatically multiplexed by the standard library. However, if you perform some disks I/Os, or call foreign libraries, this will generate threads.

For instance, the logging library used in your code may perform some blocking I/Os resulting in threads being created (especially if the generated files are hosted on a slow device).

huangapple
  • 本文由 发表于 2014年12月22日 18:14:27
  • 转载请务必保留本文链接:https://go.coder-hub.com/27600587.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定