英文:
GO: Exceeds 10000-thread on simple tcp server?
问题
我需要编写一个能够处理超过500,000个连接的TCP服务器。
我在Go语言上编写了一个简单的服务器，但是当连接超过10,000个时，服务器会崩溃，并显示错误信息"runtime: program exceeds 10000-thread limit fatal error: thread exhaustion"。
服务器在最新的Linux版本上运行。Go语言可以使用epoll来处理TCP连接，使其异步化并使用少量的文件描述符。那么为什么服务器会超过线程限制呢？
这是我的简单服务器代码：
package main
import (
"strconv"
"net"
"log"
"time"
"bufio"
"io"
)
type Handler struct {
conn net.Conn
closed chan bool
}
func (h *Handler) Listen() { // 监听传入数据的连接
defer h.conn.Close()
bf := bufio.NewReader(h.conn)
for {
line, _, err := bf.ReadLine()
if err != nil {
if err == io.EOF {
log.Println("End connection")
}
h.closed <- true // 发送给调度器，表示连接已关闭
return
}
// ... 处理数据的业务逻辑
}
}
type Dispatcher struct {
handlers map[string]*Handler `map[ip]*Handler`
}
func (d *Dispatcher) AddHandler(conn net.Conn) {
addr := conn.RemoteAddr().String()
handler := &Handler{conn, make(chan bool, 1)}
d.handlers[addr] = handler
go handler.Listen()
<-handler.closed // 当连接关闭时，从handlers中移除handler
delete(d.handlers, addr)
}
func (d *Dispatcher) ListenHandlers(port int) {
sport := strconv.Itoa(port)
ln, err := net.Listen("tcp", ":"+sport)
if err != nil {
log.Println(err)
return
}
defer ln.Close()
for {
conn, err := ln.Accept() // 接受连接
if err != nil {
log.Println(err)
continue
}
tcpconn := conn.(*net.TCPConn)
tcpconn.SetKeepAlive(true)
tcpconn.SetKeepAlivePeriod(10 * time.Second)
go d.AddHandler(conn)
}
}
func main() {
dispatcher := &Dispatcher{make(map[string]*Handler)}
dispatcher.ListenHandlers(3000)
}
更新于2015年11月10日
问题出在https://github.com/felixge/tcpkeepalive库上。请不要使用它 
英文:
I need to write a tcp server which can handle more than 500k connections.
I wrote a simple server on golang, but when connections more than 10k, the server crashed with the error message "runtime: program exceeds 10000-thread limit fatal error: thread exhaustion".
Server runs in last linux build. That go can use epoll for tcp connection, to make it async and use few fd. So why does the server exceed the thread limit?!
My simple server:
package main
import (
"strconv"
"net"
"log"
"time"
"bufio"
"io"
)
type Handler struct {
conn net.Conn
closed chan bool
}
func (h *Handler) Listen() { // listen connection for incomming data
defer h.conn.Close()
bf := bufio.NewReader(h.conn)
for {
line, _, err := bf.ReadLine()
if err != nil {
if err == io.EOF {
log.Println("End connection")
}
h.closed <- true // send to dispatcher, that connection is closed
return
}
// ... some business logic with data
}
}
type Dispatcher struct {
handlers map[string]*Handler `map[ip]*Handler`
}
func (d *Dispatcher) AddHandler(conn net.Conn) {
addr := conn.RemoteAddr().String()
handler := &Handler{conn, make(chan bool, 1)}
d.handlers[addr] = handler
go handler.Listen()
<-handler.closed // when connection closed, remove handler from handlers
delete(d.handlers, addr)
}
func (d *Dispatcher) ListenHandlers(port int) {
sport := strconv.Itoa(port)
ln, err := net.Listen("tcp", ":" + sport)
if err != nil {
log.Println(err)
return
}
defer ln.Close()
for {
conn, err := ln.Accept() // accept connection
if err != nil {
log.Println(err)
continue
}
tcpconn := conn.(*net.TCPConn)
tcpconn.SetKeepAlive(true)
tcpconn.SetKeepAlivePeriod(10 * time.Second)
go d.AddHandler(conn)
}
}
func main() {
dispatcher := &Dispatcher{make(map[string]*Handler)}
dispatcher.ListenHandlers(3000)
}
Update 11.10.2015
The problem was that https://github.com/felixge/tcpkeepalive library. Please do not use it
答案1
得分: 2
问题是由于https://github.com/felixge/tcpkeepalive库引起的。
该库获取套接字文件描述符（fd），然后复制并使用新的fd进行操作，该fd在阻塞模式下运行。因此，每次连接时，Go会创建一个新的线程。Go不会终止已创建的线程（因为这是Go语言的本机行为），并且经过一段时间后，Go会出现“超过10000个线程”的错误。就是这样！
请不要使用它
英文:
The problem was that https://github.com/felixge/tcpkeepalive library.
This library get socket file descriptor (fd), copy then and worked with new fd, which running in blocked mode. Therefore, go created new thread on every connect. Go don't kill created thread (because this is native golang behavior), and after some times go failed with "Exceeds 10000-thread". That's it !
Please do not use it
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论