x/net/http2: what happens if Stream ID exhausted and what the action when the error of errStreamID is triggered

A related RFC chapter mentions this:

RFC 7540
Stream identifiers cannot be reused. Long-lived connections can
result in an endpoint exhausting the available range of stream
identifiers. A client that is unable to establish a new stream
identifier can establish a new connection for new streams. A server
that is unable to establish a new stream identifier can send a GOAWAY
frame so that the client is forced to open a new connection for new
streams.

Based on http2 RFC, from client site, when the request send out, if stream id larger then 2^31, our http2 lib will see it as invalid stream id , and return errStreamId. But it is seem that there is no additional processing for errStreamId in our http2 lib or in h2_bundle. So how client to know streamID exhausted so that it can request new connection? and legacy connection could not be shutdown by client or server, because legacy stream id still can be maintain.

Is it possible to run into a problem where HTTP2 connections where the stream id is exhausted (more than 2^31). Another words, errStreamID will be triggered.

Like this:

https://github.com/golang/net/blob/daac0cec0cf964a628a29bb4b82940c225b921ed/http2/frame.go#L1097

but not return errStreamID to werr, so werr(write err) is nil

https://github.com/golang/net/blob/daac0cec0cf964a628a29bb4b82940c225b921ed/http2/transport.go#L1637

So I feel confuse how to handle the errStreamID , and I try to test by myself by using http2 client/server echo module, which stream id is 1 ,and manually modify the code like this

func (f *Framer) WriteHeaders(p HeadersFrameParam) error {
	//if !validStreamID(p.StreamID) && !f.AllowIllegalWrites {
	//	return errStreamID
	//}
	return errStreamID

I find the client do not send any tcp connection request by capture frames using wireshark, and the code blocked at function of read in pipe.go, maybe my observations are limited.

client used :
https://github.com/posener/h2conn/blob/master/example/echo/client.go#L35

enter image description here

How golang handle errStreamId, and Why http2 lib will be blocked? May golang's http2 lib do not support this? if want to implement about stream id exhausted. Application level have to maintain stream id by itself, not rely on http2 lib, if stream id exhaust, application should request a new connection.

http2 lib's next stream id should transfer to application level and application should maintain the stream Id, if application knows next stream id is larger then 2^31, it will request a new connection by a new port, and legacy http2(tcp) connection should be kept.

This answer focus on the client side.

The stream id is guaranteed to be valid far before (*Framer).WriteHeaders (or other methods of *Framer) is called.

Entry point: (*Transport).RoundTrip

Let's start from (*Transport).RoundTrip, which could be considered as the entry point to send a request. This method calls (*Transport).RoundTripOpt in turn (full source).

func (t *Transport) RoundTripOpt(req *http.Request, opt RoundTripOpt) (*http.Response, error) {

	// ...unrelated code truncated...
	for retry := 0; ; retry++ {
		cc, err := t.connPool().GetClientConn(req, addr)
		
		// ...unrelated code truncated...
		res, err := cc.RoundTrip(req)
		// ...unrelated code truncated...
		return res, nil
	}
}

We will drill down t.connPool().GetClientConn(req, addr) and res, err := cc.RoundTrip(req).

Connection pool: (*clientConnPool).getClientConn

(*clientConnPool).GetClientConn calls (*clientConnPool).getClientConn (full source) in turn:

func (p *clientConnPool) getClientConn(req *http.Request, addr string, dialOnMiss bool) (*ClientConn, error) {
	// ...unrelated code truncated...
	for {
		p.mu.Lock()
		for _, cc := range p.conns[addr] {
			if cc.ReserveNewRequest() {
				// ...unrelated code truncated...
				p.mu.Unlock()
				return cc, nil
			}
		}
		// ...unrelated code truncated...
		call := p.getStartDialLocked(req.Context(), addr)
		p.mu.Unlock()
		<-call.done
		if shouldRetryDial(call, req) {
			continue
		}
		cc, err := call.res, call.err
		if err != nil {
			return nil, err
		}
		if cc.ReserveNewRequest() {
			return cc, nil
		}
	}
}

If it can find a connection from the pool that can be used to send the request, use it. Otherwise, dial for a new one. Here is (*ClientConn).ReserveNewRequest (full source):

func (cc *ClientConn) ReserveNewRequest() bool {
	cc.mu.Lock()
	defer cc.mu.Unlock()
	if st := cc.idleStateLocked(); !st.canTakeNewRequest {
		return false
	}
	cc.streamsReserved++
	return true
}

And (*ClientConn).idleStateLocked (full source):

func (cc *ClientConn) idleStateLocked() (st clientConnIdleState) {
	// ...unrelated code truncated...
	st.canTakeNewRequest = cc.goAway == nil && !cc.closed && !cc.closing && maxConcurrentOkay &&
		!cc.doNotReuse &&
		int64(cc.nextStreamID)+2*int64(cc.pendingRequests) < math.MaxInt32 &&
		!cc.tooIdleLocked()
	return
}

Here it is: int64(cc.nextStreamID)+2*int64(cc.pendingRequests) < math.MaxInt32. A connection will be used only when its nextStreamID is guaranteed to be valid.

Assign stream id: (*ClientConn).RoundTrip

Now let's turn back to (*ClientConn).RoundTrip to see how a stream id is assigned to clientStream.

The call sequence is:

• (*ClientConn).RoundTrip (full source)
  • (*clientStream).doRequest (full source)
    • (*clientStream).writeRequest (full source)
      • (*ClientConn).addStreamLocked (full source)

func (cc *ClientConn) addStreamLocked(cs *clientStream) {
	cs.flow.add(int32(cc.initialWindowSize))
	cs.flow.setConnFlow(&cc.flow)
	cs.inflow.init(transportDefaultStreamFlow)
	cs.ID = cc.nextStreamID
	// ^^^^^^^^^^^^^^^^^^^^ <==== Here it is.
	cc.nextStreamID += 2
	cc.streams[cs.ID] = cs
	if cs.ID == 0 {
		panic("assigned stream ID 0")
	}
}

The stream id is copied from (ClientConn).nextStreamID and (ClientConn).nextStreamID increases by 2 (Streams
initiated by a client MUST use odd-numbered stream identifiers).

What happens to the connections that can not be used to send new request?

These connections will become idle sooner or later. And they will remain idle for some time (the maximum amount is controlled by Transport.IdleConnTimeout) and then close themselves (if Transport.IdleConnTimeout is zero, they will not close themselves).

The behavior is demonstrated by the following example.

Notes:

1. the example requires to modify the standard package to make the stream id exhausted fast. This is the line to change:

   - nextStreamID:          1,
   + nextStreamID:          math.MaxInt32-2,
   

2. since it requires to modify the standard package, the go tools should be built from the source code too. Let's assume that the go tools is built to ~/src/golang/go/bin/go.

3. run the example with this command:

   GODEBUG=http2debug=2 ~/src/golang/go/bin/go run main.go >logs.txt 2>&1
   

4. the Transport in the example is not the same as the one we discussed in the previous sections. The client in the example below will call into h2_bundle.go eventually. h2_bundle.go is generated from golang.org/x/net/http2. In the previous sections, all the source codes is from this package.

Here comes the example:

package main

import (
	"io"
	"log"
	"net"
	"net/http"
	"net/http/httptest"
	"time"
)

func main() {
	ts := httptest.NewUnstartedServer(nil)
	ts.EnableHTTP2 = true
	ts.StartTLS()
	defer ts.Close()

	client := ts.Client()
	if transport, ok := client.Transport.(*http.Transport); ok {
		// The transport created by httptest is different from the default
		// transport. Let's copy the configurations from the default transport.
		// See:
		// - https://github.com/golang/go/blob/f30cd520516037b2fdb367ddd8e0851019bf3440/src/net/http/httptest/server.go#L176-L181
		// - https://github.com/golang/go/blob/f30cd520516037b2fdb367ddd8e0851019bf3440/src/net/http/transport.go#L43-L54
		transport.DialContext = (&net.Dialer{
			Timeout:   30 * time.Second,
			KeepAlive: 30 * time.Second,
		}).DialContext
		// It seems that the MaxIdleConns is ignored in HTTP2.
		transport.MaxIdleConns = 100
		// Here it is. The IdleConnTimeout.
		transport.IdleConnTimeout = 90 * time.Second
		transport.TLSHandshakeTimeout = 10 * time.Second
		transport.ExpectContinueTimeout = 1 * time.Second
	}

	for i := 0; i < 5; i++ {
		log.Printf("======= request: %d\n", i)
		resp, err := client.Get(ts.URL)
		if err != nil {
			log.Printf("client.Get: %v", err)
			continue
		}
		_, _ = io.Copy(io.Discard, resp.Body)
		resp.Body.Close()
	}

	log.Println("======= begin sleep for 100s")
	// Wait for the IdleConnTimeout to take effect.
	time.Sleep(100 * time.Second)
	log.Println("======= end sleep for 100s")
}

Below is the output. It shows that new connections are created because no cached connection was available (stream id exhausted). And after the IdleConnTimeout, the idle connections close themselves.

2023/05/10 23:48:20 ======= request: 0
2023/05/10 23:48:20 http2: Transport failed to get client conn for 127.0.0.1:46027: http2: no cached connection was available
2023/05/10 23:48:20 http2: server connection from 127.0.0.1:48362 on 0xc00014a000
<...truncated...>
2023/05/10 23:48:20 http2: Transport received DATA flags=END_STREAM stream=2147483645 len=19 data="404 page not found\n"
2023/05/10 23:48:20 ======= request: 1
2023/05/10 23:48:20 http2: Transport failed to get client conn for 127.0.0.1:46027: http2: no cached connection was available
2023/05/10 23:48:20 http2: Transport failed to get client conn for 127.0.0.1:46027: http2: no cached connection was available
2023/05/10 23:48:20 http2: Transport creating client conn 0xc000004780 to 127.0.0.1:46027
<...truncated...>
2023/05/10 23:48:20 http2: Transport received DATA flags=END_STREAM stream=2147483645 len=19 data="404 page not found\n"
2023/05/10 23:48:20 ======= request: 2
2023/05/10 23:48:20 http2: Transport failed to get client conn for 127.0.0.1:46027: http2: no cached connection was available
2023/05/10 23:48:20 http2: Transport failed to get client conn for 127.0.0.1:46027: http2: no cached connection was available
2023/05/10 23:48:20 http2: Transport creating client conn 0xc00009e480 to 127.0.0.1:46027
<...truncated...>
2023/05/10 23:48:20 http2: Transport received DATA flags=END_STREAM stream=2147483645 len=19 data="404 page not found\n"
2023/05/10 23:48:20 ======= begin sleep for 100s
2023/05/10 23:49:50 http2: Transport closing idle conn 0xc00034a180 (forSingleUse=false, maxStream=2147483645)
2023/05/10 23:49:50 http2: Transport closing idle conn 0xc000004600 (forSingleUse=false, maxStream=2147483645)
<...truncated...>
2023/05/10 23:50:00 ======= end sleep for 100s

Following source code in x/net/http, just like @Zuke Lu's answer,thank you his tracing code again.
https://stackoverflow.com/a/76209075/21847706.

I find that the error of errStreamID maybe rarely triggered from http2 client, because stream id will only be inc only after get valid connection.

func (t *Transport) RoundTripOpt(req *http.Request, opt RoundTripOpt) (*http.Response, error) {

    // ...unrelated code truncated...
    for retry := 0; ; retry++ {
        cc, err := t.connPool().GetClientConn(req, addr)
        
        // ...unrelated code truncated...
        res, err := cc.RoundTrip(req)//only in this function, new stream id will be only inc.
        // ...unrelated code truncated.
        
        return res, nil
    }
}

connection must can be found. so enter into the function (*ClientConn).idleStateLocked.

func (cc *ClientConn) idleStateLocked() (st clientConnIdleState) {
    // ...unrelated code truncated...
    st.canTakeNewRequest = cc.goAway == nil && !cc.closed && !cc.closing && maxConcurrentOkay &&
        !cc.doNotReuse &&
        int64(cc.nextStreamID)+2*int64(cc.pendingRequests) < math.MaxInt32 &&
        !cc.tooIdleLocked()
    return
}

if the st.canTakeNewRequest return false when MaxInt32 can not contain streamid. then code will go into

func(p* clientConnPool) getStartDialLocked(ctx context.Context, addr string) * diaCall {
  call := &dialCall{p, make(chan struct{}),ctx}
  ...
  go call.dial(call.ctx, addr)
}

then goto dial function

func (t *Transport) dialClientConn(addr string, singleUse bool) (*ClientConn, error) {
...
return t.newClientConn(tconn, singleUse)
}

so the newClientConnection has been call, and this function will create a new tcp connection, because a random port will be assign.

so I feel confuse again, how to client/server handle legacy tcp connection. I think that application layer make decision how to legacy tcp connection based on actual situation. server may shutdown this when it find there is no payload on legacy connection. This is just a speculation