英文:
mpirun, Python, and task mapping
问题
I have to use two tools together in a SLURM system with many cores, in a Python wrapper.
- The first one is complex and cannot be changed. It is spawn in Python directly, and uses task per node used, and each task uses as many CPUs as there are in each node. Example: 4 nodes of 20 cores, the command to run is
mpirun -np 4 -oversubscribe --bind-to none python run_solver1.py
. - The second one (OpenFOAM) is usually spawned by itself with mpirun and uses as many tasks are there are CPUs in total. The command for the previous configuration would be:
mpirun -np 80 solver2 -parallel
, then the solver 2 would itself handle the parallel tasks run.
What I need is to still use the same syntax as the first case, but also include the second solver in the python script: mpirun -np 4 -oversubscribe --bind-to none python run_solvers.py
.
I have been using subprocess
to spawn the task, but if I do:
subprocess.check_call('solver2', '-parallel'])
, with this configuration, I get only 4 tasks spawned instead of 80.subprocess.check_call('mpirun -np 80 solver2 -parallel'.split())
with this configuration, subprocess returns an error- Same if I add
-oversubscribe --bind-to none
to the call.
I also tried with mpi4py:
MPI.COMM_SELF.Spawn('solver2', args=['-parallel'], maxprocs=80)
yieldsmpi4py.MPI.Exception: MPI_ERR_SPAWN: could not spawn processes
.- Same for
MPI.COMM_WORLD.Spawn('solver2', args=['-parallel'], maxprocs=80)
.
Is there any way for me to make it work, and make Python understand that I need to spawn this one command with 80 processors?
Thanks!
英文:
I have to use two tools together in a SLURM system with many cores, in a Python wrapper.
- The first one is complex and cannot be changed. It is spawn in Python directly, and uses task per node used, and each task uses as many CPUs as there are in each node. Example: 4 nodes of 20 cores, the command to run is
mpirun -np 4 -oversubscribe --bind-to none python run_solver1.py
. - The second one (OpenFOAM) is usually spawned by itself with mpirun and uses as many tasks are there are CPUs in total. The command for the previous configuration would be:
mpirun -np 80 solver2 -parallel
, then the solver 2 would itself handle the parallel tasks run.
What I need is to still use the same syntax as the first case, but also include the second solver in the python script: mpirun -np 4 -oversubscribe --bind-to none python run_solvers.py
.
I have been using subprocess
to spawn the task, but if I do:
subprocess.check_call('solver2', '-parallel'])
, with this configuration, I get only 4 tasks spawned instead of 80.subprocess.check_call('mpirun -np 80 solver2 -parallel'.split())
with this configuration, subprocess returns an error- Same if I add
-oversubscribe --bind-to none
to the call.
I also tried with mpi4py:
MPI.COMM_SELF.Spawn('solver2', args=['-parallel'], maxprocs=80)
yieldsmpi4py.MPI.Exception: MPI_ERR_SPAWN: could not spawn processes
.- Same for
MPI.COMM_WORLD.Spawn('solver2', args=['-parallel'], maxprocs=80)
.
Is there any way for me to make it work, and make Python understand that I need to spawn this one command with 80 processors?
Thanks!
答案1
得分: 1
是的,可以使其工作。您可以使用Python的mpi4py模块来生成具有正确处理器数量的进程。以下代码应该可以工作:
from mpi4py import MPI
comm = MPI.COMM_SELF.Spawn('solver2', args=['-parallel'], maxprocs=80)
comm.Disconnect()
这段代码将生成80个带有给定参数的solver2进程实例。
英文:
Yes, it is possible to make it work. You can use the Python mpi4py module to spawn processes with the correct number of processors. The following code should work:
from mpi4py import MPI
comm = MPI.COMM_SELF.Spawn('solver2', args=['-parallel'], maxprocs=80)
comm.Disconnect()
This code will spawn 80 instances of the solver2 process with the given argument.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论