memcached apparently resetting connections

UPDATE:

It's not memcached, it's a lot of sockets in TIME_WAIT state:

% ss -s
Total: 2494 (kernel 2784)
TCP:   43323 (estab 2314, closed 40983, orphaned 0, synrecv 0, timewait 40982/0), ports 16756

BTW, I have modified previous version (below) to use Brad Fitz's memcache client and to reuse the same memcache connection:

http://dpaste.com/1387307/

OLD VERSION:

I have thrown together the most basic webserver in Go that has handler function doing only one thing:

• retrieving a key from memcached
• sending it as http response to client

Here's the code: http://dpaste.com/1386559/

The problem is I'm getting a lot of connection resets on memcached:

2013/09/18 20:20:11 http: panic serving [::1]:19990: dial tcp 127.0.0.1:11211: connection reset by peer
goroutine 20995 [running]:
net/http.func·007()
		/usr/local/go/src/pkg/net/http/server.go:1022 +0xac
main.maybe_panic(0xc200d2e570, 0xc2014ebd80)
		/root/go/src/http_server.go:19 +0x4d
main.get_memc_val(0x615200, 0x7, 0x60b5c0, 0x6, 0x42ee58, ...)
		/root/go/src/http_server.go:25 +0x64
main.func·001(0xc200149b40, 0xc2017b3380, 0xc201888b60)
		/root/go/src/http_server.go:41 +0x35
net/http.HandlerFunc.ServeHTTP(0x65e950, 0xc200149b40, 0xc2017b3380, 0xc201888b60)
		/usr/local/go/src/pkg/net/http/server.go:1149 +0x3e
net/http.serverHandler.ServeHTTP(0xc200095410, 0xc200149b40, 0xc2017b3380, 0xc201888b60)
		/usr/local/go/src/pkg/net/http/server.go:1517 +0x16c
net/http.(*conn).serve(0xc201b9b2d0)
		/usr/local/go/src/pkg/net/http/server.go:1096 +0x765
created by net/http.(*Server).Serve
		/usr/local/go/src/pkg/net/http/server.go:1564 +0x266

I have taken care to set Linux kernel networking in such way as not to get in the way (turning off SYN flooding protection etc).

...
...
And yet on testing with "ab" (below) I'm getting those errors.

ab -c 1000 -n 50000 "http://localhost:8000/"

There is no sign whatsoever anywhere I looked that it's the kernel (dmesg, /var/log).

I would guess that is because you are running out of sockets - you never close the memc here. Check with netstat while your program is running.

func get_memc_val(k string) []byte {
	memc, err := gomemcache.Connect(mc_ip, mc_port)
	maybe_panic(err)
	val, _, _ := memc.Get(k)
	return val
}

I'd use this go memcache interface if I were you - it was written by the author of memcached who now works for Google on Go related things.

Try <a href="https://github.com/valyala/ybc/tree/master/libs/go/memcache">memcache client from YBC library</a>. Unlike <a href="https://github.com/bradfitz/gomemcache">gomemcache</a> it opens and re-uses only a few connections to memcache server irregardless of the number of concurrent requests issued via the client. It achieves high performance by pipelining concurrent requests over a small number of open connections to the memcache server.

The number of connections to the memcache server can configured via <a href="http://godoc.org/github.com/valyala/ybc/libs/go/memcache#ClientConfig">ClientConfig.ConnectionsCount</a>.