ioutil.ReadAll(response.Body) blocks forever - Golang

tr := &http.Transport{
	TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
}
client := &http.Client{Transport: tr}
response, err := client.Get(link)
if err != nil {
	fmt.Println(err)
}
defer response.Body.Close()

//block forever at the next line
content, _ = ioutil.ReadAll(response.Body)

The above is my code to read content from a webpage which resides in a loop. I found sometimes the line ioutil.ReadAll(response.Body) will block forever. This happens randomly, however, it almost always happens on this webpage: http://xkcd.com/55 . It's very interesting that when I do curl http://xkcd.com/55, it returns nothing, however, wget http://xkcd.com/55 returns the whole webpage.

I suspect your problem is that you try to read the response body even if there's an error:

if err != nil {
    fmt.Println(err)
}

You should either have an else after this, or you should return or continue or something else. Your ReadAll() line is undefined behavior.

(If you originally copied this from the Get() example code, note that it includes a log.Fatalf() in the error leg, which terminates the program.)

I suspect that, as you say, occasionally you are getting a network error for one reason or another. Are you checking the output for the result of the Println()? The way you've done it, I could imagine it easily getting buried in the output.

As @twotwotwo notes, this URL returns a redirect to the same URL with a trailing slash. Get() will automatically handle this for you, so that's not the problem. By default curl does not follow redirects while wget does. You can see the header information by passing -i to curl.

Other things to verify:

• Make sure your defer is actually being called. Remember, defer is called at the end of the function, not the end of the current scope. So if you're in a loop (as you mention), you will just accumulate defer blocks and never actually close these responses.

• If the server in fact never closes the connection, then io.ReadAll() will never return. This is a feature. If you want a timeout, you need to handle that yourself. You should be able to test this hypothesis with tools like curl. For some solutions, see:

• https://stackoverflow.com/questions/20990332/golang-http-timeout-and-goroutines-accumulation

• https://stackoverflow.com/questions/16895294/how-to-set-timeout-for-http-get-requests-in-golang

• http.Transport.ResponseHeaderTimeout

Additionally, avoid read response Body in ReadAll without memory/buffer limits control, example:

googleResponse := GoogleResponse{}
err = json.NewDecoder(io.LimitReader(resp.Body, MAX_MEMORY)).Decode(&googleResponse)
if err != nil {
    return nil, err
}

Read more about it in good blog posts:<br/>
Crossing Streams: a Love Letter to io.Reader by Jason Moiron<br/>
ioutil.ReadAll(httpResponse.Body) memory consumption<br/>
Golang Slices And The Case Of The Missing Memory

Your code should work as expected. I am guessing, its a network issue. Try setting a higher timeout.

package main

import (
	"crypto/tls"
	"fmt"
	"io/ioutil"
	"net/http"
)

func main() {

	link := "http://xkcd.com/55"

	tr := &http.Transport{
		TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
	}
	client := &http.Client{Transport: tr}
	response, err := client.Get(link)
	if err != nil {
		fmt.Println(err)
	}
	defer response.Body.Close()

	//block forever at the next line
	content, _ := ioutil.ReadAll(response.Body)

	fmt.Println(string(content))

}

I probably have found the solution by adding DisableKeepAlives: true, to the `&http.Transport, like this:

tr := &http.Transport{
	TLSClientConfig:   &tls.Config{InsecureSkipVerify: true},
	DisableKeepAlives: true,
}

Since I made this change, I haven't encountered any long blocking, yet. But I'm not 100% sure this is the solution. I will leave the new code running one day or two. If there will be no blocking, I think this problem is solved.