如何在循环中使用python multiprocessing Pool.map [英] How to use python multiprocessing Pool.map within loop

问题描述

我正在使用Runge-Kutta进行模拟.在每个时间步长，两个独立变量的两个FFT是必需的，可以并行化.我实现了这样的代码:

I am running a simulation using Runge-Kutta. At every time step two FFT of two independent variables are necessary which can be parallelized. I implemented the code like this:

from multiprocessing import Pool
import numpy as np

pool = Pool(processes=2)    # I like to calculate only 2 FFTs parallel 
                            # in every time step, therefor 2 processes

def Splitter(args):
    '''I have to pass 2 arguments'''
    return makeSomething(*args):

def makeSomething(a,b):
    '''dummy function instead of the one with the FFT'''
    return a*b

def RungeK():
    # ...
    # a lot of code which create the vectors A and B and calculates 
    # one Kunge-Kutta step for them 
    # ...

    n = 20                         # Just something for the example
    A = np.arange(50000)
    B = np.ones_like(A)

    for i in xrange(n):                  # loop over the time steps
        A *= np.mean(B)*B - A
        B *= np.sqrt(A)
        results = pool.map(Splitter,[(A,3),(B,2)])
        A = results[0]
        B = results[1]

    print np.mean(A)                                 # Some output
    print np.max(B)

if __name__== '__main__':
    RungeK()

不幸的是，python到达循环后会生成无限数量的进程.在此之前，似乎只有两个进程正在运行.我的记忆也填满了.添加

Unfortunately python generates a unlimited number of processes after reaching the loop. Before it seems that only two processes are running. Also my memory fills up. Adding a

pool.close()
pool.join()

循环后面的

不能解决我的问题，将其放在循环内对我来说毫无意义.希望您能提供帮助.

behind the loop does not solve my problem, and to put it inside the loop makes no sense for me. Hope you can help.

推荐答案

将池的创建移至RungeK函数中；

Move the creation of the pool into the RungeK function;

def RungeK():
    # ...
    # a lot of code which create the vectors A and B and calculates
    # one Kunge-Kutta step for them
    # ...

    pool = Pool(processes=2)
    n = 20                         # Just something for the example
    A = np.arange(50000)
    B = np.ones_like(A)

    for i in xrange(n):  # loop over the time steps
        A *= np.mean(B)*B - A
        B *= np.sqrt(A)
        results = pool.map(Splitter, [(A, 3), (B, 2)])
        A = results[0]
        B = results[1]
    pool.close()
    print np.mean(A)  # Some output
    print np.max(B)

或者，将其放在主块中.

Alternatively, put it in the main block.

这可能是多处理工作方式的副作用.例如.在MS Windows上，您需要能够导入没有副作用的主模块(例如创建新进程).

This is probably a side effect of how multiprocessing works. E.g. on MS windows, you need to be able to import the main module without side effects (like creating new processes).

这篇关于如何在循环中使用python multiprocessing Pool.map的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！