Go, tcp too many open files debug

Here's a straightforward Go http (tcp) connection test script

func main() {
    ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        fmt.Fprintln(w, "Hello, client")
    }))
    defer ts.Close()
    var wg sync.WaitGroup
    for i := 0; i < 2000; i++ {
        wg.Add(1)
        go func(i int) {
            defer wg.Done()
            resp, err := http.Get(ts.URL)
            if err != nil {
                panic(err)
            }
            greeting, err := ioutil.ReadAll(resp.Body)
            resp.Body.Close()
            if err != nil {
                panic(err)
            }
            fmt.Printf("%s", i, greeting)
        }(i)
    }
    wg.Wait()
}

And If I run this in Ubuntu I get:

panic: Get http://127.0.0.1:33202: dial tcp 127.0.0.1:33202: too many open files

Other posts say to make sure Close the connection, which I am doing it all here.
And others say to increase the limit of maximum connection with ulimit or try sudo sysctl -w fs.inotify.max_user_watches=100000 but still does not work.

How do I run millions of tcp connection goroutines in a single server?
It crashes only with 2,000 connections.

Thanks,

I think you need to change your max file descriptors. I have run into the same problem on one of my development VMs before and needed to change the file descriptors max, not anything with inotify settings.

FWIW, your program runs fine on my VM.

·> ulimit -n
120000

But after I run

·> ulimit -n 500
·> ulimit -n
500

I get:

panic: Get http://127.0.0.1:51227: dial tcp 127.0.0.1:51227: socket: too many open files

** Don't fall into the trap that Praveen did **

Note ulimit != ulimit -n.

➜  cmd git:(wip-poop) ✗ ulimit -a
-t: cpu time (seconds)              unlimited
-f: file size (blocks)              unlimited
-d: data seg size (kbytes)          unlimited
-s: stack size (kbytes)             8192
-c: core file size (blocks)         0
-v: address space (kbytes)          unlimited
-l: locked-in-memory size (kbytes)  unlimited
-u: processes                       1418
-n: file descriptors                4864

Go’s http package doesn’t specify request timeouts by default. You should always include a timeout in your service. What if a client doesn't close their session? Your process will keep alive old sessions hitting ulimits. A bad actor could intentionally open thousands of sessions, DOSing your server. Heavy load services should adjust ulimits as well but timeouts for backstop.

Ensure you specify a timeout:

http.DefaultClient.Timeout = time.Minute * 10

You can validate before and after by monitoring files opened by your process:

lsof -p [PID_ID]

If you want to run millions of go routines that open/read/close a socket, well you better up your ulimit, or open/read/close the socket and pass the value read in to the go-routine, but I would use a buffered channel to control how many file descriptors you want to be able to open.

const (
	// this is where you can specify how many maxFileDescriptors
	// you want to allow open
	maxFileDescriptors = 100
)

func main() {
	ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		fmt.Fprintln(w, "Hello, client")
	}))
	defer ts.Close()
	var wg sync.WaitGroup
	maxChan := make(chan bool, maxFileDescriptors)
	for i := 0; i < 1000; i++ {
		maxChan <- true
	    wg.Add(1)
		go func(url string, i int, maxChan chan bool, wg *sync.WaitGroup) {
			defer wg.Done()
			defer func(maxChan chan bool) { <-maxChan }(maxChan)
			resp, err := http.Get(url)
			if err != nil {
				panic(err)
			}
			greeting, err := ioutil.ReadAll(resp.Body)
			if err != nil {
				panic(err)
			}
			err = resp.Body.Close()
			if err != nil {
				panic(err)
			}
			fmt.Printf("%d: %s", i, string(greeting))
		}(ts.URL, i, maxChan, &wg)
	}
	wg.Wait()
}

HTTP/1.1 uses persistent connections by default:
A significant difference between HTTP/1.1 and earlier versions of HTTP is that persistent connections are the default behavior of any HTTP connection.
http://www.w3.org/Protocols/rfc2616/rfc2616-sec8.html
The solution was to inform the server that the client wants to close the connection after the transaction is complete. This can be done by setting the Connection header,
req.Header.Set("Connection", "close") or by setting the Close property to true on the http.Request:
req.Close = true After doing that, the “too many open files” issue went away as the program was no longer keeping HTTP connections open and thus not using up file descriptors.

I solved this by adding req.Close = true and req.Header.Set("Connection", "close"). I think it's better than changing ulimit.

source: http://craigwickesser.com/2015/01/golang-http-to-many-open-files/

I had to also to manually set the close connection header to avoid the file descriptor issue:

r, _ := http.NewRequest(http.MethodDelete, url, nil)
r.Close = true
res, err := c.Do(r)
res.Body.Close();

Without r.Close = true and res.Body.Close() I hit the file descriptor limit. With both I could fire off as many as I needed.

To temporarily allow more open files:

View your current settings by running:

sudo launchctl limit maxfiles

Increase the limit to 65535 files by running the following commands. If your site has fewer files, you can set choose to set lower soft (65535) and hard (200000) limits.

sudo launchctl limit maxfiles 65535 200000
ulimit -n 65535
sudo sysctl -w kern.maxfiles=200000
sudo sysctl -w kern.maxfilesperproc=65535

Note that you might need to set these limits for each new shell.

https://www.google.com/search?q=mac+os+launchctl+limit+maxfiles+site%3Aapple.stackexchange.com&oq=mac+os+launchctl+limit+maxfiles+site%3Aapple.stackexchange.com

change the ulimit to avoid the error "too many open files"
by default max ulimit is 4096 for linux and 1024 for mac,
u can change ulimit to 4096 by typing
ulimit -n 4096
for beyond 4096 you need to modify limits.conf in etc/security folder for linux and set hard limit to 100000 by adding this line "* hard core 100000"