如何从串行python脚本使用mpi4py应用程序 [英] How to Consume an mpi4py application from a serial python script

问题描述

我试图基于mpi4py创建一个库，但是我想在串行python代码中使用它.

I tried to make a library based on mpi4py, but I want to use it in serial python code.

$ python serial_source.py

但是在 serial_source.py 内部存在一些称为 parallel_bar

but inside serial_source.py exists some function called parallel_bar

from foo import parallel_bar
# Can I to make this with mpi4py like a common python source code?
result = parallel_bar(num_proc = 5)

这个问题的动机是找到正确的方法来使用mpi4py来优化python中的程序，这些程序不一定设计为完全并行运行.

The motivation for this question is about finding the right way to use mpi4py to optimize programs in python which were not necessarily designed to be run completely in parallel.

推荐答案

这确实是可能的，并且在在MSMPI中未实现Spawn .

This is indeed possible and is in the documentation of mpi4py in the section Dynamic Process Management. What you need is the so called Spawn functionality which is not available with MSMPI (in case you are working with Windows) see also Spawn not implemented in MSMPI.

第一个文件为您的函数提供了一种包装器，以隐藏所有MPI内容，我想这是您的意图.在内部，它在4个新生成的进程中调用包含您的并行代码的实际"脚本.

The first file provides a kind of wrapper to your function to hide all the MPI stuff, which I guess is your intention. Internally it calls the "actual" script containing your parallel code in 4 newly spawned processes.

最后，您可以打开python终端并调用:

Finally, you can open a python terminal and call:

from my_prog import parallel_fun

parallel_fun()
# Hi from 0/4
# Hi from 3/4
# Hi from 1/4
# Hi from 2/4
# We got the magic number 6

my_prog.py

import sys
import numpy as np
from mpi4py import MPI

    def parallel_fun():
        comm = MPI.COMM_SELF.Spawn(
            sys.executable,
            args = ['child.py'],
            maxprocs=4)

        N = np.array(0, dtype='i')

        comm.Reduce(None, [N, MPI.INT], op=MPI.SUM, root=MPI.ROOT)

        print(f'We got the magic number {N}')

以下是带有并行代码的子文件:

Here the child file with the parallel code:

from mpi4py import MPI
import numpy as np


comm = MPI.Comm.Get_parent()

print(f'Hi from {comm.Get_rank()}/{comm.Get_size()}')
N = np.array(comm.Get_rank(), dtype='i')

comm.Reduce([N, MPI.INT], None, op=MPI.SUM, root=0)

这篇关于如何从串行python脚本使用mpi4py应用程序的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！