Go retry for 403 forbidden http request?

I got this message from my Go http request:

&{Status:403 Forbidden StatusCode:403 Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Content-Type:[text/html] Content-Length:[345] Date:[Wed, 26 Nov 2014 07:46:36 GMT] Server:[lighttpd]] Body:0xc2155b4860 ContentLength:345 TransferEncoding:[] Close:true Trailer:map[] Request:0xc2160829c0 TLS:0xc21604eb00}

I send request with this:

resp, err := client.Do(req)
if resp.StatusCode > 400 {

And my code was supposed to just retry the request initializing the httpclient different than the one that gets this forbidden message. And I get this message:

goroutine 780 [IO wait, 479 minutes]:
net.(*pollDesc).Wait(0xc2082b6a00, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc2082b6a00, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).Read(0xc2082b69a0, 0xc2080e5000, 0x1000, 0x1000, 0x0, 0x7fd302e42fd8, 0xc20c8359a0)
    /usr/local/go/src/net/fd_unix.go:242 +0x40f
net.(*conn).Read(0xc2080c20d8, 0xc2080e5000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:121 +0xdc
crypto/tls.(*block).readFromUntil(0xc2081c0f60, 0x7fd302e44538, 0xc2080c20d8, 0x5, 0x0, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:454 +0xe6
crypto/tls.(*Conn).readRecord(0xc2082a6840, 0x17, 0x0, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:539 +0x2da
crypto/tls.(*Conn).Read(0xc2082a6840, 0xc2082e3000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:904 +0x166
net/http.noteEOFReader.Read(0x7fd302e48ee8, 0xc2082a6840, 0xc20805d238, 0xc2082e3000, 0x1000, 0x1000, 0x7612c0, 0x0, 0x0)
    /usr/local/go/src/net/http/transport.go:1270 +0x6e
net/http.(*noteEOFReader).Read(0xc2081701e0, 0xc2082e3000, 0x1000, 0x1000, 0xc208014400, 0x0, 0x0)
    <autogenerated>:125 +0xd4
bufio.(*Reader).fill(0xc2082cf620)
    /usr/local/go/src/bufio/bufio.go:97 +0x1ce
bufio.(*Reader).Peek(0xc2082cf620, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:132 +0xf0
net/http.(*persistConn).readLoop(0xc20805d1e0)
    /usr/local/go/src/net/http/transport.go:842 +0xa4
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:660 +0xc9f

goroutine 17 [syscall, 489 minutes, locked to thread]:
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:2232 +0x1

goroutine 798 [IO wait, 479 minutes]:
net.(*pollDesc).Wait(0xc2082b6a70, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc2082b6a70, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).Read(0xc2082b6a10, 0xc2080e4000, 0x1000, 0x1000, 0x0, 0x7fd302e42fd8, 0xc20c8359b0)
    /usr/local/go/src/net/fd_unix.go:242 +0x40f
net.(*conn).Read(0xc2080c20b8, 0xc2080e4000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:121 +0xdc
crypto/tls.(*block).readFromUntil(0xc2081ef5f0, 0x7fd302e44538, 0xc2080c20b8, 0x5, 0x0, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:454 +0xe6
crypto/tls.(*Conn).readRecord(0xc2082a6580, 0x17, 0x0, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:539 +0x2da
crypto/tls.(*Conn).Read(0xc2082a6580, 0xc20828d000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:904 +0x166
net/http.noteEOFReader.Read(0x7fd302e48ee8, 0xc2082a6580, 0xc208248cb8, 0xc20828d000, 0x1000, 0x1000, 0x7612c0, 0x0, 0x0)
    /usr/local/go/src/net/http/transport.go:1270 +0x6e
net/http.(*noteEOFReader).Read(0xc208167300, 0xc20828d000, 0x1000, 0x1000, 0xc208014400, 0x0, 0x0)
    <autogenerated>:125 +0xd4
bufio.(*Reader).fill(0xc208132c60)
    /usr/local/go/src/bufio/bufio.go:97 +0x1ce
bufio.(*Reader).Peek(0xc208132c60, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:132 +0xf0
net/http.(*persistConn).readLoop(0xc208248c60)
    /usr/local/go/src/net/http/transport.go:842 +0xa4
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:660 +0xc9f

goroutine 9904 [select, 57 minutes]:
net/http.(*persistConn).readLoop(0xc21629a8f0)
    /usr/local/go/src/net/http/transport.go:928 +0x9ce
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:660 +0xc9f

goroutine 799 [select, 479 minutes]:
net/http.(*persistConn).writeLoop(0xc208248c60)
    /usr/local/go/src/net/http/transport.go:945 +0x41d
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:661 +0xcbc

goroutine 781 [select, 479 minutes]:
net/http.(*persistConn).writeLoop(0xc20805d1e0)
    /usr/local/go/src/net/http/transport.go:945 +0x41d
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:661 +0xcbc

goroutine 9905 [select, 57 minutes]:
net/http.(*persistConn).writeLoop(0xc21629a8f0)
    /usr/local/go/src/net/http/transport.go:945 +0x41d
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:661 +0xcbc

goroutine 10958 [runnable]:
net/http.(*persistConn).writeLoop(0xc217483810)
    /usr/local/go/src/net/http/transport.go:945 +0x41d
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:661 +0xcbc

goroutine 10957 [runnable]:
net/http.(*persistConn).readLoop(0xc217483810)
    /usr/local/go/src/net/http/transport.go:928 +0x9ce
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:660 +0xc9f

What does it mean? Is it timing out? I have no clue where to start debugging...

I have encountered this problem. This basically means that you have reached the maximum amount of opened files per process. For Linux it is around 1024. This is a very easy thing to do with go since you can have hundreds of thousands of go routines.

Please note that you open files for a lot of stuff -

=> Actual log file

=> /dev files

=> UNIX Sockets

=> Network sockets

=> Library files /lib /lib64

=> Executables and other programs etc