Why doesn't net/http respect timeout durations longer than 30 seconds?

Using golang 1.20.1.

In golang's net/http and context packages, I'm unable to set a timeout longer than thirty seconds. Setting a shorter timeout works fine, e.g.

Code:

	log.Infof("Elasticsearch URL is %v", elasticsearchURL)
    client := &http.Client{Timeout: time.Duration(time.Second * 60)}
	req, err := http.NewRequest("GET", listBackupsURL, nil)
	if err != nil {
		internalError(w, fmt.Sprintf("error creating request: %v", err))
		return
	}
	req.SetBasicAuth(username, password)
	resp, err := client.Do(req)
	if err != nil {
		// handle error
		internalError(w, fmt.Sprintf("error accessing elasticsearch: %v", err))
		return
	}

Logs:

I0503 23:01:55.973821       1 somecode.go:85] URL is http://elasticsearch.example.ingest:9200
E0503 23:02:25.976345       1 caller_handler.go:63] 500 Internal Server Error: error accessing elasticsearch: Get "http://elasticsearch.example.ingest:9200/_cat/snapshots/object-store-repo?v&s=id": dial tcp 1.2.3.4:9200: i/o timeout

The timeout is thirty seconds, not sixty.

I get the same behavior if I use http.NewRequestWithContext(...) and use a context setting the same timeouts:

Code:

   	log.Infof("Elasticsearch URL is %v", elasticsearchURL)
    ctx, cancel := context.WithTimeout(context.Background(), time.Duration(time.Second * 60)) 
    defer cancel()
	req, err := http.NewRequestWithContext(ctx, "GET", listBackupsURL, nil)
	if err != nil {
		internalError(w, fmt.Sprintf("error creating request: %v", err))
		return
	}
	req.SetBasicAuth(username, password)
	resp, err := client.Do(req)
	if err != nil {
		// handle error
		internalError(w, fmt.Sprintf("error accessing elasticsearch: %v", err))
		return
	}

Logs:

I0503 23:31:10.941169       1 somecode.go:85] Elasticsearch URL is http://elasticsearch.example.ingest:9200
E0503 23:31:40.941642       1 caller_handler.go:63] 500 Internal Server Error: error accessing elasticsearch: Get "http://elasticsearch.example.ingest:9200/_cat/snapshots/object-store-repo?v&s=id": dial tcp 1.2.3.4:9200: i/o timeout

However, if I change the timeout to three seconds (time.Duration(time.Second * 3)) in either method, it works as expected:

I0503 23:44:17.622121       1 somecode.go:85] Elasticsearch URL is http://elasticsearch.example.ingest:9200
E0503 23:44:20.624795       1 caller_handler.go:63] 500 Internal Server Error: error accessing elasticsearch: Get "http://elasticsearch.example.ingest:9200/_cat/snapshots/object-store-repo?v&s=id": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

I'd isolate the problem to eliminate possibilities to help narrow what is causing it to happen. If possible, use a small piece of code so you deal with only what you want.

To test your infrastructure httpstat will help you simulate a remote timeout. Example:

func main() {

	client := &http.Client{Timeout: time.Duration(time.Second * 200)}
	req, err := http.NewRequest("GET", "https://httpstat.us/504?sleep=120000", nil)
	if err != nil {
		log.Fatal(err)
		return
	}

	resp, err := client.Do(req)
	if err != nil {
		log.Fatal(err)
		return
	}

	fmt.Println(resp)
}

If you're getting dial tcp IP:PORT: i/o timeout timeout at this point, then you need to check your OS and and the firewall(s). Whatever timeout you set in Go should override the OS defaults, and if you're getting timeout in that way it is possible that there is a firewall (local or remote) causing that.

Alternatively, if you're able to connect externally, then there is a possibility that ES is timing out, although according to the docs you should expect an informative error coming from ES directly. You can se the timeout for ES directly in the URL and test it as well.

Hope this helps.