Python:并行修改数组的简便方法 [英] Python: easy way to modify array in parallel

问题描述

这个问题听起来很简单，但是对于Python并行化来说，我确实很挣扎.我在C ++的OpenMP中处理了并行化，这真容易得多. 我需要做的是并行修改矩阵的条目.而已.事实是，我无法使用简单的joblib库做到这一点:

The question might sound easy, but being new to parallelization in Python I am definitely struggling. I dealt with parallelization in OpenMP for C++ and that was a hell of a lot easier. What I need to do is modify entries of a matrix in parallel. That's it. Thing is, I can't do it using the simple joblib library:

from joblib import Parallel, delayed

my_array = [[ 1 ,2 ,3],[4,5,6]]

def foo(array,x):
  for i in [0,1,2]:
     array[x][i]=2
  return 0

def main(array):
  inputs = [0,1]
  if __name__ == '__main__':
    Parallel(n_jobs=2, verbose = 0)(delayed(foo)(array,i) for i in inputs)

main(my_array)

当作业数为1时此代码将起作用(因此main调用后的数组全为2)，但实际上变为真正的多处理则无效(该数组在main调用后保持不变) ). 我认为的主要问题是我无法使用Parallel函数返回任何有用的信息.我还尝试将数组放在共享内存中，但是找不到有关如何使用joblib的任何信息.

This code would work when the number of jobs is 1 (so the array after the call of main is all 2s), but it wouldn't when it actually becomes true multiprocessing (the array stays the same after the call of main). The main problem I think is that I can't return anything useful with the Parallel function. I also tried to have the array in shared memory, but I couldn't find any informations on how to do it with joblib.

有什么建议吗?

推荐答案

仅使用标准库，您可以使用共享内存，在这种情况下，可以使用Array来存储和修改您的数组:

Using only the standard library, you could use shared memory, in this case an Array to store and modify your array:

from multiprocessing import Pool, Array, Lock

lock = Lock()

my_array = [Array('i', [1, 2, 3], lock=lock),
            Array('i', [4, 5, 6], lock=lock),]

让我建议您对过程进行一些修改:列出需要对矩阵进行的所有更改的列表或明细表(为清楚起见，我将使用namedtuple)，以及一个函数映射这些更改.

Let me suggest you some modifications to your procedure: make a list or a schedule of all the changes you need to make to your matrix (To make it clear I am going to use a namedtuple), and a function to map these changes.

Change = namedtuple('Change', 'row idx value')

scheduled_changes = [Change(0, 0, 2),
                     Change(0, 1, 2),
                     Change(1, 0 ,2),
                     Change(1, 1, 2)]
# or build the scheduled changes list in any other way like using 
# for loops or list comprehensions...

def modify(change, matrix=my_array):
    matrix[change.row][change.idx] = change.value

现在您可以使用Pool将修改功能映射到更改:

Now you can use a Pool to map the modify function to the changes:

pool = Pool(4)
pool.map(modify, scheduled_changes)

for row in my_array:
    for col in row:
        print(col, end=' ')
    print()

# 2 2 3
# 2 2 6

这篇关于Python:并行修改数组的简便方法的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！