具有共享numpy数组的Python多处理 [英] Python multiprocessing with shared numpy array

问题描述

假设我创建了一个具有2维numpy数组作为属性的对象A.然后，我使用Process API创建了10个线程来随机设置A的行.

Suppose I created an object A with an 2 dimension numpy array as attributes. Then I created 10 threads using Process API to randomly set the rows of A.

我想知道是否编写以下代码，是否要在所有进程(线程)之间共享self.x，或者每个进程(线程)之间只有一个副本?

I want to know if I write the following code, whether self.x if shared among all the Process(thread), or every Process(thread) has just a copy?

如果不共享，我会丢失所有更新，对吧?

If not shared, I will lose all the updates, right?

import numpy as np
from multiprocessing import Process

class A:

   def __init__():
       self.x = np.zeros((3,4))

   def update():
        threads = []
        for i in range(10):
            trd = Process(target=self.set, args=(i,))
        threads.append(trd)
        trd.start()

        for i in range(10):
            threads[i].join()

   def set(i):
       self.x[i/3] = np.random.rand(1,4)


if ___main___:
        a = A()
        a.update()

推荐答案

不，它不是共享的.您生成多个进程，每个进程都复制父进程的文件描述符，并带有 no shared 对象.

No, it is not shared. You spawn multiple processes with each process copying the file descriptor of the parent process and with no shared object.

要创建共享共享变量，请使用 ctype 对象.

To create shared a shared variable you have use ctype objects.

因此，与其将数组声明为-

So instead of declaring the array as -

self.x = np.zeros((3,4))

您可以使用 Array -

from multiprocessing import Array
self.x = Array('i', [0]*10)

如果仍然要使numpy数组成为共享数组，请查看此出色的

If still you want to make the numpy array a shared array, have a look at this great answer.

这里的警告是，可能并不那么容易.您还必须锁定共享阵列，以避免出现任何竞争情况.

The caveat here is, it might not be that easy. You'll also have to lock the shared array to avoid any race condition.

这篇关于具有共享numpy数组的Python多处理的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！