英文:
Elegant way to handle goroutine in IO wait state in GRPC
问题
我们的服务器(grpc-gateway + grpc)在K8S上运行,使用go 1.13,并终止堆栈信息
Last State: Terminated
Reason: Error
Message: o.(*Reader).fill(0xc002ec78c0)
/Users/local/go/src/bufio/bufio.go:100 +0x103
bufio.(*Reader).Peek(0xc002ec78c0, 0x4, 0x0, 0x0, 0x0, 0x0, 0xc002e79ad0)
/Users/local/go/src/bufio/bufio.go:138 +0x4f
net/http.(*conn).readRequest(0xc00245a000, 0x1ef4800, 0xc000be7780, 0x0, 0x0, 0x0)
/Users/local/go/src/net/http/server.go:962 +0xb3b
net/http.(*conn).serve(0xc00245a000, 0x1ef4800, 0xc000be7780)
/Users/local/go/src/net/http/server.go:1817 +0x6d4
created by net/http.(*Server).Serve
/Users/local/go/src/net/http/server.go:2928 +0x384
goroutine 8724981 [IO wait]:
internal/poll.runtime_pollWait(0x7f5d3a8f84e8, 0x72, 0xffffffffffffffff)
/Users/local/go/src/runtime/netpoll.go:184 +0x55
internal/poll.(*pollDesc).wait(0xc000155d98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
/Users/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/Users/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc000155d80, 0xc00138b000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/Users/local/go/src/internal/poll/fd_unix.go:169 +0x1cf
net.(*netFD).Read(0xc000155d80, 0xc00138b000, 0x1000, 0x1000, 0xc00142b9e8, 0x4ce13d, 0xc000155d80)
/Users/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc0000dd690, 0xc00138b000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/Users/local/go/src/net/net.go:184 +0x68
net/http.(*connReader).Read(0xc001e9b050, 0xc00138b000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/Users/local/go/src/net/http/server.go:785 +0xf4
bufio.(*Reader).fill(0xc0021fe060)
/Users/local/go/src/bufio/bufio.go:100 +0x103
bufio.(*Reader).Peek(0xc0021fe060, 0x4, 0x0, 0x0, 0x0, 0x0, 0xc00142bad0)
/Users/local/go/src/bufio/bufio.go:138 +0x4f
net/http.(*conn).readRequest(0xc0008f43c0, 0x1ef4800, 0xc000503640, 0x0, 0x0, 0x0)
/Users/local/go/src/net/http/server.go:962 +0xb3b
net/http.(*conn).serve(0xc0008f43c0, 0x1ef4800, 0xc000503640)
/Users/local/go/src/net/http/server.go:1817 +0x6d4
created by net/http.(*Server).Serve
/Users/local/go/src/net/http/server.go:2928 +0x384
Exit Code: 2
根据这个问题,一个可能的解决方案是
s := new(http.Server)
// ...
s.ReadTimeout = 5 * time.Second
s.WriteTimeout = 5 * time.Second
// ...
然而,我们在grpc.NewServer
中找不到ReadTimeout
,我们是否遗漏了什么?或者如何更优雅地处理处于IO等待状态的goroutine在GRPC中?grpc版本是v1.21.1
。
英文:
Our server (grpc-gateway + grpc) is running on K8S with go 1.13, and terminate stack information
Last State: Terminated
Reason: Error
Message: o.(*Reader).fill(0xc002ec78c0)
/Users/local/go/src/bufio/bufio.go:100 +0x103
bufio.(*Reader).Peek(0xc002ec78c0, 0x4, 0x0, 0x0, 0x0, 0x0, 0xc002e79ad0)
/Users/local/go/src/bufio/bufio.go:138 +0x4f
net/http.(*conn).readRequest(0xc00245a000, 0x1ef4800, 0xc000be7780, 0x0, 0x0, 0x0)
/Users/local/go/src/net/http/server.go:962 +0xb3b
net/http.(*conn).serve(0xc00245a000, 0x1ef4800, 0xc000be7780)
/Users/local/go/src/net/http/server.go:1817 +0x6d4
created by net/http.(*Server).Serve
/Users/local/go/src/net/http/server.go:2928 +0x384
goroutine 8724981 [IO wait]:
internal/poll.runtime_pollWait(0x7f5d3a8f84e8, 0x72, 0xffffffffffffffff)
/Users/local/go/src/runtime/netpoll.go:184 +0x55
internal/poll.(*pollDesc).wait(0xc000155d98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
/Users/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/Users/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc000155d80, 0xc00138b000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/Users/local/go/src/internal/poll/fd_unix.go:169 +0x1cf
net.(*netFD).Read(0xc000155d80, 0xc00138b000, 0x1000, 0x1000, 0xc00142b9e8, 0x4ce13d, 0xc000155d80)
/Users/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc0000dd690, 0xc00138b000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/Users/local/go/src/net/net.go:184 +0x68
net/http.(*connReader).Read(0xc001e9b050, 0xc00138b000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/Users/local/go/src/net/http/server.go:785 +0xf4
bufio.(*Reader).fill(0xc0021fe060)
/Users/local/go/src/bufio/bufio.go:100 +0x103
bufio.(*Reader).Peek(0xc0021fe060, 0x4, 0x0, 0x0, 0x0, 0x0, 0xc00142bad0)
/Users/local/go/src/bufio/bufio.go:138 +0x4f
net/http.(*conn).readRequest(0xc0008f43c0, 0x1ef4800, 0xc000503640, 0x0, 0x0, 0x0)
/Users/local/go/src/net/http/server.go:962 +0xb3b
net/http.(*conn).serve(0xc0008f43c0, 0x1ef4800, 0xc000503640)
/Users/local/go/src/net/http/server.go:1817 +0x6d4
created by net/http.(*Server).Serve
/Users/local/go/src/net/http/server.go:2928 +0x384
Exit Code: 2
Per this question, one possible solution is
> s := new(http.Server)
> // ...
> s.ReadTimeout = 5 * time.Second
> s.WriteTimeout = 5 * time.Second
> // ...
However, we failed to find the ReadTimeout
from grpc.NewServer
, is there anything are we missing? or how to handle goroutine in IO wait state in GRPC more elegantly?
grpc version is v1.21.1
答案1
得分: 1
gprc.NewServer 在创建时允许使用零个或多个 ServerOption
:
func NewServer(opt ...ServerOption) *Server
虽然似乎没有与 http.Server
的 ReadTimeout
或 WriteTimeout
等效的选项,但你可以尝试使用 keepalive.ServerParameters
:
type ServerParameters struct {
MaxConnectionIdle time.Duration // 当前默认值为无限大。
MaxConnectionAge time.Duration // 当前默认值为无限大。
MaxConnectionAgeGrace time.Duration // 当前默认值为无限大。
Time time.Duration // 当前默认值为2小时。
Timeout time.Duration // 当前默认值为20秒。
}
(完整文档请参考 keepalive.ServerParameters)
你可以调整 keepalive.ServerParameters.Time
的值,将其设置为低于2小时的值:
srv := grpc.NewServer(
keepalive.ServerParameters{Time: 5 * time.Minute},
)
这将降低连接的重用率,但也会释放长时间未使用的客户端连接。
英文:
gprc.NewServer allows for zero or more ServerOption
during creation:
func NewServer(opt ...ServerOption) *Server
while there does not appear to be a ReadTimeout
or WriteTimeout
equivalent to http.Server
you could try keepalive.ServerParameters
:
type ServerParameters struct {
MaxConnectionIdle time.Duration // The current default value is infinity.
MaxConnectionAge time.Duration // The current default value is infinity.
MaxConnectionAgeGrace time.Duration // The current default value is infinity.
Time time.Duration // The current default value is 2 hours.
Timeout time.Duration // The current default value is 20 seconds.
}
(full docs keepalive.ServerParameters)
and tune say keepalive.ServerParameters.Time
to something lower than 2 hours:
srv := grpc.NewServer(
keepalive.ServerParameters{Time:5*time.Minute},
)
This will lower connection reuse - but will also free up client connections that are long since dead.
答案2
得分: -1
你可以使用带有超时的上下文(Context)。
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论