当发送大量请求时,出现”Go write unix /tmp/mysql.sock: broken pipe”错误。

huangapple go评论85阅读模式
英文:

Go write unix /tmp/mysql.sock: broken pipe when sending a lot of requests

问题

我有一个Go API端点,它执行了几个MySQL查询。当端点接收到少量请求时,它工作得很好。然而,我现在正在使用apache bench进行测试,发送了100个请求。前100个请求都成功通过了。然而,第二个100个请求导致出现以下错误:

2014/01/15 12:08:03 http: panic serving 127.0.0.1:58602: runtime error: invalid memory address or nil pointer dereference
goroutine 973 [running]:
net/http.func·009()
/usr/local/Cellar/go/1.2/libexec/src/pkg/net/http/server.go:1093 +0xae
runtime.panic(0x402960, 0x9cf419)
/usr/local/Cellar/go/1.2/libexec/src/pkg/runtime/panic.c:248 +0x106
database/sql.(*Rows).Close(0x0, 0xc2107af540, 0x69)
/usr/local/Cellar/go/1.2/libexec/src/pkg/database/sql/sql.go:1576 +0x1e
store.findProductByQuery(0xc2107af540, 0x69, 0x0, 0xb88e80, 0xc21000ac70)
/Users/dennis.suratna/workspace/session-go/src/store/product.go:83 +0xe3
store.FindProductByAppKey(0xc210337748, 0x7, 0x496960, 0x6, 0xc2105eb1b0)
/Users/dennis.suratna/workspace/session-go/src/store/product.go:28 +0x11c
api.SessionHandler(0xb9eff8, 0xc2108ee200, 0xc2108f5750, 0xc2103285a0, 0x0, ...)
/Users/dennis.suratna/workspace/session-go/src/api/session_handler.go:31 +0x2fb
api.func·001(0xb9eff8, 0xc2108ee200, 0xc2108f5750, 0xc2103285a0)
/Users/dennis.suratna/workspace/session-go/src/api/api.go:81 +0x4f
reflect.Value.call(0x3ad9a0, 0xc2101ffdb0, 0x130, 0x48d520, 0x4, ...)
/usr/local/Cellar/go/1.2/libexec/src/pkg/reflect/value.go:474 +0xe0b
reflect.Value.Call(0x3ad9a0, 0xc2101ffdb0, 0x130, 0xc2103c4a00, 0x3, ...)
/usr/local/Cellar/go/1.2/libexec/src/pkg/reflect/value.go:345 +0x9d
github.com/codegangsta/inject.(*injector).Invoke(0xc2103379c0, 0x3ad9a0, 0xc2101ffdb0, 0x4311a0, 0x1db94e, ...)

看起来这个错误不是由并发请求的数量引起的,而是由于某些未正确关闭的东西引起的。我已经在我的代码中关闭了每个准备语句,我想知道是否有人以前见过这种情况。

编辑:
这是我如何初始化我的MySQL连接:

func InitStore(environment string) error {
db, err := sql.Open("mysql", connStr(environment))
....

S = &Store{
    Mysql:       db,
    Environment: environment,
}

}

这只会在启动服务器时发生一次。

英文:

I have a Go API endpoint that makes several MySQL query. When the endpoint receives a small number of requests, it works just fine. However, I am now testing it using apache bench with 100 requests. The first 100 all went through. However, the 2nd 100 caused this error to appear

    2014/01/15 12:08:03 http: panic serving 127.0.0.1:58602: runtime error: invalid memory address or nil pointer dereference
goroutine 973 [running]:
net/http.func·009()
	/usr/local/Cellar/go/1.2/libexec/src/pkg/net/http/server.go:1093 +0xae
runtime.panic(0x402960, 0x9cf419)
	/usr/local/Cellar/go/1.2/libexec/src/pkg/runtime/panic.c:248 +0x106
database/sql.(*Rows).Close(0x0, 0xc2107af540, 0x69)
	/usr/local/Cellar/go/1.2/libexec/src/pkg/database/sql/sql.go:1576 +0x1e
store.findProductByQuery(0xc2107af540, 0x69, 0x0, 0xb88e80, 0xc21000ac70)
	/Users/dennis.suratna/workspace/session-go/src/store/product.go:83 +0xe3
store.FindProductByAppKey(0xc210337748, 0x7, 0x496960, 0x6, 0xc2105eb1b0)
	/Users/dennis.suratna/workspace/session-go/src/store/product.go:28 +0x11c
api.SessionHandler(0xb9eff8, 0xc2108ee200, 0xc2108f5750, 0xc2103285a0, 0x0, ...)
	/Users/dennis.suratna/workspace/session-go/src/api/session_handler.go:31 +0x2fb
api.func·001(0xb9eff8, 0xc2108ee200, 0xc2108f5750, 0xc2103285a0)
	/Users/dennis.suratna/workspace/session-go/src/api/api.go:81 +0x4f
reflect.Value.call(0x3ad9a0, 0xc2101ffdb0, 0x130, 0x48d520, 0x4, ...)
	/usr/local/Cellar/go/1.2/libexec/src/pkg/reflect/value.go:474 +0xe0b
reflect.Value.Call(0x3ad9a0, 0xc2101ffdb0, 0x130, 0xc2103c4a00, 0x3, ...)
	/usr/local/Cellar/go/1.2/libexec/src/pkg/reflect/value.go:345 +0x9d
github.com/codegangsta/inject.(*injector).Invoke(0xc2103379c0, 0x3ad9a0, 0xc2101ffdb0, 0x4311a0, 0x1db94e, ...)

It looks like it's not caused by the number of concurrent requests but, rather, something that is not properly closed. I am already closing every prepare statement that I create in my code. I am wondering if anyone has ever seen this before.

Edit:
This is how I am initializing my MySQL connection:

func InitStore(environment string) error {
	db, err := sql.Open("mysql", connStr(environment))
    ....

    S = &Store{
       	Mysql:       db,
    	Environment: environment,
   }
}

In this happens only once when I start the server.

答案1

得分: 2

好的,以下是翻译好的内容:

好的,所以我已经解决了这个问题,现在我可以发送大约500个请求,每个请求有10个并发,不再出现“Broken pipe”或“Too many connections error”的错误。

我认为这主要归结于遵循最佳实践。当你不希望返回多行时,使用QueryRow而不是Query,并将其与Scan链接起来。

db.QueryRow(...).Scan(...)

如果你不希望返回行,并且不打算重复使用语句,请使用Exec而不是Prepare

如果你有准备好的语句或查询多行,请不要忘记使用Close()

以上所有内容都来自于:

https://github.com/go-sql-driver/mysql/issues/111

英文:

Ok so I was able to solve this problem and now I can send ~500 requests with 10 concurrency with no more Broken pipe or Too many connections error.

I think it all comes down to following best practices. When you don't expect multiple rows to be returned user QueryRow instead of Query AND chain it with Scan

db.QueryRow(...).Scan(...)

If you don't expect rows to be returned and if you're not going to re-use your statements, use Exec not Prepare.

If you have prepared statement or querying multiple rows. Don't forget to Close()

Got all of the above from

https://github.com/go-sql-driver/mysql/issues/111

答案2

得分: 1

如果您使用Go 1.2.x版本,可以使用db.SetMaxOpenConns来告诉sql包不要打开超过X个连接。在已经打开(且繁忙)X个连接之后需要数据库连接的查询将会阻塞,直到有可用的连接。

话虽如此:接下来的"堆栈跟踪"的代码是什么?在http/server.go文件的第1093行是当您的serve函数失败时的恢复代码。看起来更像是您对一些数据处理不当导致失败,或者您漏掉了错误检查,然后尝试处理数据,而实际上返回了一个错误等等。

英文:

If you use Go 1.2.x you can use db.SetMaxOpenConns to tell the sql package to not open more than X connections. Queries that need a database connection after X connections are already open (and busy) will block until there's an available connection.

That being said: what are the next lines of the "stack trace"? Line ~1093 in http/server.go is the recover code when your serve function fails. It looks more like you are just mishandling some data and that makes it fail or you are missing an error check and then try processing data when you really were returned an error, etc.

huangapple
  • 本文由 发表于 2014年1月15日 05:40:09
  • 转载请务必保留本文链接:https://go.coder-hub.com/21124576.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定