Popen with PIPE blocks called process, when it's output large enough, how to unblock it?

I need to start subprocess, which can just exit, printing something to console, or never exits (became a web server e.g.).

So in my runner I'd like to check is process still running or already exits, and do it in completely unblocking manner, so if it exits, I'd like to capture the output.

So it looks like this

import subprocess
import sys
from subprocess import Popen
from time import sleep

with Popen(
        [
            sys.executable,
            r"...\stub1.py"
        ],
        stdout=subprocess.PIPE,
        stderr=subprocess.PIPE,
        encoding="utf-8",
) as proc:
    while True:
        ret = proc.poll()
        print(ret)
        if ret is not None: break
        sleep(1)

where stub1.py is just

print("foo"*100)

So it works fine while output of subprocess is relatively small.

None
0

But with larger output called process is blocked (on writing to filled pipe as I understand) and never exits, so poll() always returns None.

So how to unblock called process, flushing pipe in unblocking manner?

I see I can pass real file descriptors for stdout/stderr, but is it possible with PIPEs?

Apparently subprocess pipes have a size limit of 65536, which you can change simply by adding a pipesize argument to your Popen. A reasonable maximum value (for Linux, not sure about other OS) is 1024KB (1024*1024=1048576 bytes). You can apply this modification to your code like this:

import sys
from subprocess import Popen
from time import sleep

with Popen(
        [
            sys.executable,
            r"...\stub1.py"
        ],
        stdout=subprocess.PIPE,
        stderr=subprocess.PIPE,
        encoding="utf-8",
        pipesize=1024*1024
) as proc:
    while True:
        ret = proc.poll()
        print(ret)
        if ret is not None: break
        sleep(1)

I tested this with a sample program and it works just fine.

If 1024 KB still is not enough, you could just use os.set_blocking() on the Popen's stdout like this:

import os
import sys
from subprocess import Popen
from time import sleep

with Popen(
        [
            sys.executable,
            r"...\stub1.py"
        ],
        stdout=subprocess.PIPE,
        stderr=subprocess.PIPE,
        encoding="utf-8",
        pipesize=1024*1024
) as proc:
     os.set_blocking(proc.stdout.fileno(),os.O_NONBLOCK)
     while True:
         output = proc.stdout.read()
         if output:
            break
         # do stuff

This, however, works only on Linux, so if you are using Windows, you might want to check something like threading if the program outputs more than 1024KB
Read more: https://bugs.python.org/issue41586 <br>
https://docs.python.org/3/library/os.html#os.set_blocking