How to disable HTTP/2 in Golang's standard http.Client, or avoid tons of INTERNAL_ERRORs from Stream ID=N?

I want to send a fairly large number (several thousand) of HTTP requests ASAP, without putting too much load on the CDN (has an https: URL, and ALPN selects HTTP/2 during the TLS phase) So, staggering (i.e. time shifting) the requests is an option, but I don't want to wait TOO long (minimize errors AND total round-trip time) and I'm not being rate limited by the server at the scale I'm operating yet.

The problem I'm seeing originates from h2_bundle.go and specifically in either writeFrame or onWriteTimeout when about 500-1k requests are in-flight, which manifests during io.Copy(fileWriter, response.Body) as:

http2ErrCodeInternal = "INTERNAL_ERROR" // also IDs a Stream number
// ^ then io.Copy observes the reader encountering "unexpected EOF"

I'm fine sticking with HTTP/1.x for now, but I would love an explanation re: what's going on. Clearly, people DO use Go to make a lot of round-trips happen per unit time, but most advice I can find is from the perspective of the server, not clients. I've already tried specifying all the relevant time-outs I can find, and cranking up connection pool max sizes.

Here's my best guess at what's going on:

The rate of requests is overwhelming a queue of connections or some other resource in the HTTP/2 internals. Maybe this is fix-able in general or possible to fine tune for my specific use case, but the fastest way to overcome this kind of problem is to rely on HTTP/1.1 entirely, as well as implement limited retry + rate limiting mechanisms.

> Aside, I am now using a single retry and rate.Limiter from https://pkg.go.dev/golang.org/x/time/rate#Limiter in addition to the "ugly hack" of disabled HTTP/2, so that outbound requests are able send an initial "burst" of M requests, and then "leak more gradually" at a given rate of N/sec. Ultimately, the errors from h2_bundle.go are just too ugly for end-users to parse. An expected/unexpected EOF should result in the client "giving it another try" or two, which is more pragmatic anyway.

As per the docs, the easiest way to disable h2 in Go's http.Client at runtime is env GODEBUG=http2client=0 ... which I can also achieve in other ways as well. Especially important to understand is that the "next protocol" is pre-negotiated "early" during TLS, so Go's http.Transport must manage that configuration along with a cache/memo to provide its functionality in a performant way. Therefore, use your own httpClient to .Do(req) (and don't forget to give your Request a context.Context so that it's easy to cancel) using a custom http.RoundTripper for Transport. Here's some example code:

type forwardRoundTripper struct {
	rt http.RoundTripper
}

func (my *forwardRoundTripper) RoundTrip(r *http.Request) (*http.Response, error) {
	return my.rt.RoundTrip(r) // adjust URLs, or transport as necessary per-request
}

// httpTransport is the http.RoundTripper given to a Client as Transport
// (don't forget to set up a reasonable Timeout and other behavior as desired)
var httpTransport = &customRoundTripper{rt: http.DefaultTransport}

func h2Disabled(rt *http.Transport) *http.Transport {
	log.Println("--- only using HTTP/1.x ...")
	rt.ForceAttemptHTTP2 = false // not good enough
	// at least one of the following is ALSO required:
	rt.TLSClientConfig.NextProtos = []string{"http/1.1"}
	// need to Clone() or replace the TLSClientConfig if a request already occurred
	// - Why? Because the first time the transport is used, it caches certain structures.
	// (if you do this replacement, don't forget to set a minimum TLS version)

	rt.TLSHandshakeTimeout = longTimeout // not related to h2, but necessary for stability
	rt.TLSNextProto = make(map[string]func(authority string, c *tls.Conn) http.RoundTripper)
	// ^ some sources seem to think this is necessary, but not in all cases
	// (it WILL be required if an "h2" key is already present in this map)
	return rt
}

func init() {
	h2ok := ...
	if t, ok := httpTransport.rt.(*http.Transport); ok && !h2ok {
		httpTransport.rt = h2Disabled(t.Clone())
	}
	// tweak rate limits here
}

This lets me make the volume of requests that I need to OR get more-reasonable errors in edge cases.