MySQL connection err (with golang): too many connections, too many (8000 more) sleep connections running show processlist

huangapple go评论109阅读模式
英文:

MySQL connection err (with golang): too many connections, too many (8000 more) sleep connections running show processlist

问题

在循环中连续执行500,000次的插入、更新、查询和删除操作时,即使我将MySQL的最大连接数设置为10,000,也会在运行十秒钟后出现"连接过多"的错误。当运行'show processlist'时,处于睡眠状态的连接数量迅速增加到大约10,000个。(但是当单独执行CURD操作时,执行1,000,000次循环非常快)。

只执行一次循环的情况下运行正常:2017-05-16 17:10:26 <LL_INFO> 总耗时5.241008毫秒。

以下是代码:

func insertUnis(db *sql.DB, name string){
	stmt_insert, err := db.Prepare("insert into memDB1 values(?,?,'world.cnworld.cnworld.cnworld.cn',?,'ee02:123::af01:9231:df18:8998:ee02:123::af01:9231:df18:8998:abcd','2017070035|2703258|1943|0|0|10.121.205.248|088:abcd:e02:123::/64','Framed-Interface-Id`varchar(64)COLLATEutf8_binNOTNULLdefaultthah','varchar(64)COLLATEutf8_binNOTNULLdefault0xBF019231DF18899934hdas','Call-From-Id`varchar(32)COLLATEd',?,'Current-Bandwidt','NAS-Port-Idvarchar(256)COLLATEutf8_binNOTNULLdefaultsrunk 6/0/36:33.351 0/0/0/0/0/0NAS-Port-Idvarchar(256)COLLATEutf8_binNOTNULLdefaultsrunk 6/0/36:33.351 0/0/0/0/0/0NAS-Port-Idvarchar(256)COLLATEutf8_binNOTNULLdefaultsrunk 6/0/36:33.351 0/0/0/0/0/0sefiuhu',48204,57239,2458787490,2458787490,2458787490,'Static-Bandwidthvhar(24)','Bas-Bandwidth` varchar(64) COLLATE utf8_bin NOT NULL defaultegse','Session-Id varchar(64) COLLATE utf8_bin NOT NULL defaultlasefjie','Coa-Session-Id` varchar(64) COLLATE utf8_bin NOT NULL defaultsdf','macaddr`varchar(32) COLLATE utf8',?,65535,45656,'Reseverd1` varchar(32) COLLATE u',4294967292)")
	if err != nil {
		Log(LL_ERROR, "stmt_insert Prepare err:", err.Error(), ".")
	}
	defer stmt_insert.Close()
	stmt_update, err := db.Prepare("update memDB1 set user_name=? where session_key=?")
	if err != nil {
		Log(LL_ERROR, "stmt_update Prepare err:", err.Error(), ".")
	}	
	defer stmt_update.Close()
	stmt_del, err := db.Prepare("delete from memDB1 where session_key=?")
		if err != nil {
			Log(LL_ERROR, "stmt_del Prepare err:", err.Error(), ".")
	}
	defer stmt_del.Close() 
	for i:=0;i<500000;i++{
		session_key:=string(Krand(64,3))
		user_name := string(Krand(64,3))
		frame_ip := string(Krand(32,3))		
		Nas_IP := string(Krand(32,3))	
		nat_ip := string(Krand(32,3))	

		_, err = stmt_insert.Exec(session_key,user_name,frame_ip,Nas_IP,nat_ip)
		if err != nil {
			Log(LL_ERROR, "stmt_insert Exec err:", err.Error(), ".")
		}
		
		_, err = stmt_update.Exec(string(Krand(64,3)), session_key)
		if err != nil {
			Log(LL_ERROR, "stmt_update Exec err:", err.Error(), ".")
		}
		sql := "SELECT * FROM memDB1 where session_key='" + session_key+"'"
		row, err := db.Query(sql)
		defer row.Close()
		if err != nil {
			Log(LL_ERROR, "Query err:", err.Error(), ".")
		}
		_, err = stmt_del.Exec(session_key)
		 if err != nil {
			Log(LL_ERROR, "stmt_del Exec err:", err.Error(), ".")
		}
	}
	common_package.WgDay.Done()
	return
}

MySQL进程列表:

mysql> show full processlist;
...............
| 1292138 | cid  | localhost:46799 | ciddb | Sleep   |    0 |          | NULL                  |
| 1292139 | cid  | localhost:46800 | ciddb | Sleep   |    0 |          | NULL                  |
| 1292140 | cid  | localhost:46801 | ciddb | Sleep   |    0 |          | NULL                  |
| 1292141 | cid  | localhost:46802 | ciddb | Sleep   |    0 |          | NULL                  |
+---------+------+-----------------+-------+---------+------+----------+-----------------------+
8719 rows in set (0.04 sec)

MySQL错误信息:

2017-05-16 16:37:56 <LL_ERROR> stmt_insert Exec err:Error 1040: Too many connections. 
2017-05-16 16:37:56 <LL_ERROR> stmt_update Exec err:Error 1040: Too many connections. 
2017-05-16 16:37:56 <LL_ERROR> Query err:Error 1040: Too many connections. 
2017-05-16 16:37:56 <LL_ERROR> stmt_del Exec err:Error 1040: Too many connections. 
2017-05-16 16:37:56 <LL_ERROR> stmt_insert Exec err:Error 1040: Too many connections. 
2017-05-16 16:37:56 <LL_ERROR> stmt_update Exec err:Error 1040: Too many connections 

MySQL设置:max_connections 10050 set global wait_timeout = 30; set global interactive_timeout = 30;

这是一个内存表,有5个索引:KEY index1 (session_key), KEY index2 (Nas-IP), KEY index3 (user_name), KEY index4 (nat_ip), KEY index5 (frame_ip)

英文:

Execute insert, update, query and delete sequentially in for loop for 500,000 times, "too many connections" occurs after running for ten seconds even I set max connection of MySQL to 10,000. When running 'show processlist', the number of sleep status connections quickly raise up to around 10,000. (but it's fine when executing CURD separately, very quick to finish 1000,000 times of loop).

Only execute for loop for one time works fine: 2017-05-16 17:10:26 <LL_INFO> total cost time5.241008ms.

below is the code:

func insertUnis(db *sql.DB, name string){
stmt_insert, err := db.Prepare(&quot;insert into memDB1 values(?,?,&#39;world.cnworld.cnworld.cnworld.cn&#39;,?,&#39;ee02:123::af01:9231:df18:8998:ee02:123::af01:9231:df18:8998:abcd&#39;,&#39;2017070035|2703258|1943|0|0|10.121.205.248|088:abcd:e02:123::/64&#39;,&#39;Framed-Interface-Id`varchar(64)COLLATEutf8_binNOTNULLdefaultthah&#39;,&#39;varchar(64)COLLATEutf8_binNOTNULLdefault0xBF019231DF18899934hdas&#39;,&#39;Call-From-Id`varchar(32)COLLATEd&#39;,?,&#39;Current-Bandwidt&#39;,&#39;NAS-Port-Idvarchar(256)COLLATEutf8_binNOTNULLdefaultsrunk 6/0/36:33.351 0/0/0/0/0/0NAS-Port-Idvarchar(256)COLLATEutf8_binNOTNULLdefaultsrunk 6/0/36:33.351 0/0/0/0/0/0NAS-Port-Idvarchar(256)COLLATEutf8_binNOTNULLdefaultsrunk 6/0/36:33.351 0/0/0/0/0/0sefiuhu&#39;,48204,57239,2458787490,2458787490,2458787490,&#39;Static-Bandwidthvhar(24)&#39;,&#39;Bas-Bandwidth` varchar(64) COLLATE utf8_bin NOT NULL defaultegse&#39;,&#39;Session-Id varchar(64) COLLATE utf8_bin NOT NULL defaultlasefjie&#39;,&#39;Coa-Session-Id` varchar(64) COLLATE utf8_bin NOT NULL defaultsdf&#39;,&#39;macaddr`varchar(32) COLLATE utf8&#39;,?,65535,45656,&#39;Reseverd1` varchar(32) COLLATE u&#39;,4294967292)&quot;)
if err != nil {
Log(LL_ERROR, &quot;stmt_insert Prepare err:&quot;, err.Error(), &quot;.&quot;)
}
defer stmt_insert.Close()
stmt_update, err := db.Prepare(&quot;update memDB1 set user_name=? where session_key=?&quot;)
if err != nil {
Log(LL_ERROR, &quot;stmt_update Prepare err:&quot;, err.Error(), &quot;.&quot;)
}	
defer stmt_update.Close()
stmt_del, err := db.Prepare(&quot;delete from memDB1 where session_key=?&quot;)
if err != nil {
Log(LL_ERROR, &quot;stmt_del Prepare err:&quot;, err.Error(), &quot;.&quot;)
}
defer stmt_del.Close() 
for i:=0;i&lt;500000;i++{
session_key:=string(Krand(64,3))
user_name := string(Krand(64,3))
frame_ip := string(Krand(32,3))		
Nas_IP := string(Krand(32,3))	
nat_ip := string(Krand(32,3))	
_, err = stmt_insert.Exec(session_key,user_name,frame_ip,Nas_IP,nat_ip)
if err != nil {
Log(LL_ERROR, &quot;stmt_insert Exec err:&quot;, err.Error(), &quot;.&quot;)
}
_, err = stmt_update.Exec(string(Krand(64,3)), session_key)
if err != nil {
Log(LL_ERROR, &quot;stmt_update Exec err:&quot;, err.Error(), &quot;.&quot;)
}
sql := &quot;SELECT * FROM memDB1 where session_key=&#39;&quot; + session_key+&quot;&#39;&quot;
row, err := db.Query(sql)
defer row.Close()
if err != nil {
Log(LL_ERROR, &quot;Query err:&quot;, err.Error(), &quot;.&quot;)
}
_, err = stmt_del.Exec(session_key)
if err != nil {
Log(LL_ERROR, &quot;stmt_del Exec err:&quot;, err.Error(), &quot;.&quot;)
}
}
common_package.WgDay.Done()
return
}

The MySQL process list:

mysql&gt; show full processlist;
...............
| 1292138 | cid  | localhost:46799 | ciddb | Sleep   |    0 |          | NULL                  |
| 1292139 | cid  | localhost:46800 | ciddb | Sleep   |    0 |          | NULL                  |
| 1292140 | cid  | localhost:46801 | ciddb | Sleep   |    0 |          | NULL                  |
| 1292141 | cid  | localhost:46802 | ciddb | Sleep   |    0 |          | NULL                  |
+---------+------+-----------------+-------+---------+------+----------+-----------------------+
8719 rows in set (0.04 sec)

MySQL error messages:

2017-05-16 16:37:56 &lt;LL_ERROR&gt; stmt_insert Exec err:Error 1040: Too many connections. 
2017-05-16 16:37:56 &lt;LL_ERROR&gt; stmt_update Exec err:Error 1040: Too many connections. 
2017-05-16 16:37:56 &lt;LL_ERROR&gt; Query err:Error 1040: Too many connections. 2017-05-16 16:37:56 &lt;LL_ERROR&gt; stmt_del Exec err:Error 1040: Too many connections. 
2017-05-16 16:37:56 &lt;LL_ERROR&gt; stmt_insert Exec err:Error 1040: Too many connections. 
2017-05-16 16:37:56 &lt;LL_ERROR&gt; stmt_update Exec err:Error 1040: Too many connections 

MySQL setting: max_connections 10050 set global wait_timeout = 30; set global interactive_timeout = 30;

it's a memory table and had 5 indexes: KEY index1 (session_key), KEY index2 (Nas-IP), KEY index3 (user_name), KEY index4 (nat_ip), KEY index5 (frame_ip)

答案1

得分: 1

你在循环中使用了defer语句,它只是将row.Close函数推入堆栈中,并在函数返回时执行。你可能想要像这样修改代码:

for i := 0; i < 500000; i++ {
...
sql := "SELECT * FROM memDB1 where session_key='" + session_key + "'"
row, err := db.Query(sql)
if err != nil {
Log(LL_ERROR, "Query err:", err.Error(), ".")
continue
}
row.Close()
...
}

相关文档:
https://blog.golang.org/defer-panic-and-recover

英文:

You have a defer inside of a loop, which just pushes the row.Close function onto the stack, and will execute when the function returns. You probably want something like this instead:

for i:=0;i&lt;500000;i++{
...
sql := &quot;SELECT * FROM memDB1 where session_key=&#39;&quot; + session_key+&quot;&#39;&quot;
row, err := db.Query(sql)
if err != nil {
Log(LL_ERROR, &quot;Query err:&quot;, err.Error(), &quot;.&quot;)
continue
}
row.Close()
...
}

Supporting docs:
https://blog.golang.org/defer-panic-and-recover

huangapple
  • 本文由 发表于 2017年5月16日 19:33:36
  • 转载请务必保留本文链接:https://go.coder-hub.com/44000340.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定