Multiprocessing Process Pool Executor blocking submit function

I am trying to turn my executor.submit function call into a blocking function so that it waits until the ProcessPoolExecutor pool has available workers.

In other words, initially it should print 1,2, then wait for 5 seconds, print 3,4, etc.

How can I achieve this?

import concurrent.futures
import multiprocessing
import time

def wait_f():
    time.sleep(5)
    return 1



if __name__ == '__main__':
    multiprocessing.freeze_support()
    global_results = []
    with concurrent.futures.ProcessPoolExecutor(max_workers=2) as executor:
        futures = []
        for j in range(10):
            future = executor.submit(wait_f)
            futures.append(future)
            print(j)
        for future in concurrent.futures.as_completed(futures):
            result = future.result()
            global_results.append(result)

Please do keep in mind that the Pool of workers pattern is designed to handle the concurrency for you so you don't have to. In other words, you should not worry about scheduling more tasks than workers as the Pool will already optimize the task flow for you.

This said, you can build from this answer a working solution.

from threading import Semaphore
from concurrent.futures import ProcessPoolExecutor

class TaskManager():
    def __init__(self, processes):
        self.workers = Semaphore(processes)
        self.executor = ProcessPoolExecutor(max_workers=processes)

    def new_task(self, function):
        """Start a new task, blocks if queue is full."""
        self.workers.acquire()
        future = self.executor.submit(function)
        future.add_done_callback(self.task_done)

    def task_done(self):
        """Called once task is done, releases the queue if blocked."""
        self.workers.release()


task_manager = TaskManager(2)
task_manager.new_task(wait_f)