reading escaped sequences from sys.stdin, bytes after escape are delayed until the next keystroke using select

I'm trying to process keystrokes in linux so I can handle arrow keys as well as normal alphnumeric etc keys.
This potentially simple approach using select and stdin delivers all the keys, but after pressing (for example) uparrow, I don't get the extra chars after escape until I press another key.

The extra characters are there if I read them, but if they aren't there (as in just pressing escape), then trying to read will hang the input and when I get the next char read I don 't know if it was part of an escape sequence or not (unless I look at timestamps)

I have tried other packages but they all have problems - requiring root, or only working if there is a screen present for example.

I want to run this code both directly on a PC, or over ssh to a raspberry pi with a server only build.

Here is a trivial test program:

#!/usr/bin/env python

import sys
import tty
import termios
import select, time

old_settings = termios.tcgetattr(sys.stdin)
ts = time.time()
try:
    tty.setraw(sys.stdin.fileno())
    while True:
        rlist, _, _ = select.select([sys.stdin], [], [], 2)
        if rlist:
            # Read a single character
            char = sys.stdin.read(1)
            if ord(char) == 27:
                print('at %6.2f ESCAPE!' % (time.time()-ts), '\x0d')
                # at this point any other characers can be read, but you can't check to see if read will block!!!
            else:
                print('at %6.2f gotta' % (time.time()-ts), char if ord(char) >32 else '{%d}' % ord(char), '\x0d')
                if char == 'x':
                    break
finally:
    termios.tcsetattr(sys.stdin, termios.TCSADRAIN, old_settings)
    print('byeeeee')

If I type 'abc<esc>d<uparrow>e' you can see the qualifying chars after esc for uparrow don't appear until I type e. If I run the same code without select, then the timings are correct, but the code blocks waiting for all keyboard input so I cannot respond to other events.

$ python3 inputer3.py
at   1.38 gotta a 
at   2.44 gotta b 
at   3.72 gotta c 
at   4.88 ESCAPE! 
at   9.32 gotta d 
at  10.51 ESCAPE! 
at  13.42 gotta [ 
at  13.42 gotta A 
at  13.42 gotta e 

Commenting out the rlist lines it all works as expected:
pressing uparrow gives me <esc>[A all with almost identical timestampsL

$ python3 inputer3.py
at   1.33 gotta a 
at   2.10 gotta b 
at   2.93 gotta c 
at   4.66 ESCAPE! 
at   5.58 gotta d 
at   7.88 ESCAPE! 
at   7.88 gotta [ 
at   7.88 gotta A 
at  10.44 gotta e 

Instead of sys.stdin.read, use os.read :

#!/usr/bin/env python

import os, sys
import tty
import termios
import select, time

old_settings = termios.tcgetattr(sys.stdin)
ts = time.time()
try:
    tty.setraw(sys.stdin.fileno())
    while True:
        rlist, _, _ = select.select([sys.stdin], [], [], 2)
        if rlist:
            # Read a single character
            # char = sys.stdin.read(1)
            char = os.read(sys.stdin.fileno(), 1).decode("utf-8")
            if ord(char) == 27:
                print('at %6.2f ESCAPE!' % (time.time()-ts), '\x0d')
                # at this point any other characers can be read, but you can't check to see if read will block!!!
            else:
                print('at %6.2f gotta' % (time.time()-ts), char if ord(char) >32 else '{%d}' % ord(char), '\x0d')
                if char == 'x':
                    break
finally:
    termios.tcsetattr(sys.stdin, termios.TCSADRAIN, old_settings)
    print('byeeeee')