How to make the Python subprocess wait for some input when running through SLURM script?

I am running some Python code using a SLURM script on a remote server accessed through SSH. At some point, issues related to licenses on the SLURM platform may happen, generating errors in Python and ending the subprocess. I want to use try-except to let the Python subprocess wait until the issue is fixed, after that it can keep running from where it stopped.

What are some smart implementations for that?

My most obvious solution is just keeping Python inside a loop if the error occurs and letting it read a file every X seconds, when I finally fix the error and want it to keep running from where it stopped, I would write something on the file and break the loop. I wonder if there is a smarter way to provide input to the Python subprocess while it is running through the SLURM script.

One idea might be to add a signal handler for signal USR1 to your Python script like this.

In the signal handler function, you can set a global variable or send a message or set a threading.Event that the main process is waiting on.

Then you can signal the process with:

kill -USR1 <PID>

or with the Python os.kill() equivalent.

Though I do have to agree there is something to be said for the simplicity of your process doing:

touch /tmp/blocked.$$

and your program waiting in a loop with a 1s sleep for that file to be removed. This way you can tell which process id is blocked.