在多处理进程之间共享大型只读 Numpy 数组 [英] Share Large, Read-Only Numpy Array Between Multiprocessing Processes

问题描述

我有一个 60GB 的 SciPy 数组(矩阵)，我必须在 5 个以上的 multiprocessing Process 对象之间共享.我看过 numpy-sharedmem 并阅读了这个讨论 在 SciPy 列表中.似乎有两种方法——numpy-sharedmem 和使用 multiprocessing.RawArray() 并将 NumPy dtypes 映射到 ctype s.现在， numpy-sharedmem 似乎是要走的路，但我还没有看到一个很好的参考示例.我不需要任何类型的锁，因为数组(实际上是一个矩阵)将是只读的.现在，由于它的大小，我想避免复制.听起来正确的方法是将数组的唯一副本创建为sharedmem数组，然后将其传递给处理 对象?几个具体问题:

1. 将 sharedmem 句柄实际传递给 sub-Process()es 的最佳方法是什么?我是否需要一个队列来传递一个数组?管道会更好吗?我可以将它作为参数传递给 Process() 子类的 init(我假设它是腌制的)?

2. 在我上面链接的讨论中，提到 numpy-sharedmem 不是 64 位安全的?我肯定在使用一些不是 32 位可寻址的结构.

3. RawArray() 方法是否存在权衡?更慢、更笨拙?

4. numpy-sharedmem 方法是否需要任何 ctype-to-dtype 映射?

5. 有没有人有一些开源代码这样做的例子?我是一个非常有实践经验的人，如果没有任何好的例子，很难让它工作.

如果我可以提供任何其他信息来帮助其他人澄清这一点，请发表评论，我会补充.谢谢！

这需要在 Ubuntu Linux 和 也许 Mac OS 上运行，但可移植性不是一个大问题.

解决方案

@Velimir Mlaker 给出了很好的答案.我想我可以添加一些评论和一个小例子.

(我找不到关于 sharedmem 的太多文档——这些是我自己实验的结果.)

1. 是否需要在子进程启动时或启动后传递句柄?如果只是前者，您可以只对 Process 使用 target 和 args 参数.这可能比使用全局变量更好.
2. 从您链接的讨论页面来看，似乎不久前已将 64 位 Linux 支持添加到 sharedmem 中，因此这可能不是问题.
3. 我不知道这个.
4. 没有.请参阅下面的示例.

示例

#!/usr/bin/env python从多处理导入过程导入共享内存导入 numpydef do_work(数据，开始):数据[开始] = 0;def split_work(num):n = 20宽度 = n/数量共享 = sharedmem.empty(n)共享[:] = numpy.random.rand(1, n)[0]打印值是 %s" % 共享processes = [Process(target=do_work, args=(shared, i*width)) for i in xrange(num)]对于进程中的 p:p.start()对于进程中的 p:p.join()打印值是 %s" % 共享打印 "type is %s" % type(shared[0])如果 __name__ == '__main__':split_work(4)

输出

值为 [ 0.81397784 0.59667692 0.10761908 0.6736734 0.46349645 0.983407180.44056863 0.10701816 0.67167752 0.29158274 0.22242552 0.142731560.34912309 0.43812636 0.58484507 0.81697513 0.57758441 0.42849590.7292129 0.06063283]值为 [ 0. 0.59667692 0.10761908 0.6736734 0.46349645 0.0.44056863 0.10701816 0.67167752 0.29158274 0. 0.142731560.34912309 0.43812636 0.58484507 0. 0.57758441 0.42849590.7292129 0.06063283]类型是<type 'numpy.float64'>

这个相关问题可能有用.>

I have a 60GB SciPy Array (Matrix) I must share between 5+ multiprocessing Process objects. I've seen numpy-sharedmem and read this discussion on the SciPy list. There seem to be two approaches--numpy-sharedmem and using a multiprocessing.RawArray() and mapping NumPy dtypes to ctypes. Now, numpy-sharedmem seems to be the way to go, but I've yet to see a good reference example. I don't need any kind of locks, since the array (actually a matrix) will be read-only. Now, due to its size, I'd like to avoid a copy. It sounds like the correct method is to create the only copy of the array as a sharedmem array, and then pass it to the Process objects? A couple of specific questions:

1. What's the best way to actually pass the sharedmem handles to sub-Process()es? Do I need a queue just to pass one array around? Would a pipe be better? Can I just pass it as an argument to the Process() subclass's init (where I'm assuming it's pickled)?

2. In the discussion I linked above, there's mention of numpy-sharedmem not being 64bit-safe? I'm definitely using some structures that aren't 32-bit addressable.

3. Are there tradeoff's to the RawArray() approach? Slower, buggier?

4. Do I need any ctype-to-dtype mapping for the numpy-sharedmem method?

5. Does anyone have an example of some OpenSource code doing this? I'm a very hands-on learned and it's hard to get this working without any kind of good example to look at.

If there's any additional info I can provide to help clarify this for others, please comment and I'll add. Thanks!

This needs to run on Ubuntu Linux and Maybe Mac OS, but portability isn't a huge concern.

解决方案

@Velimir Mlaker gave a great answer. I thought I could add some bits of comments and a tiny example.

(I couldn't find much documentation on sharedmem - these are the results of my own experiments.)

1. Do you need to pass the handles when the subprocess is starting, or after it has started? If it's just the former, you can just use the target and args arguments for Process. This is potentially better than using a global variable.
2. From the discussion page you linked, it appears that support for 64-bit Linux was added to sharedmem a while back, so it could be a non-issue.
3. I don't know about this one.
4. No. Refer to example below.

Example

#!/usr/bin/env python
from multiprocessing import Process
import sharedmem
import numpy

def do_work(data, start):
    data[start] = 0;

def split_work(num):
    n = 20
    width  = n/num
    shared = sharedmem.empty(n)
    shared[:] = numpy.random.rand(1, n)[0]
    print "values are %s" % shared

    processes = [Process(target=do_work, args=(shared, i*width)) for i in xrange(num)]
    for p in processes:
        p.start()
    for p in processes:
        p.join()

    print "values are %s" % shared
    print "type is %s" % type(shared[0])

if __name__ == '__main__':
    split_work(4)

Output

values are [ 0.81397784  0.59667692  0.10761908  0.6736734   0.46349645  0.98340718
  0.44056863  0.10701816  0.67167752  0.29158274  0.22242552  0.14273156
  0.34912309  0.43812636  0.58484507  0.81697513  0.57758441  0.4284959
  0.7292129   0.06063283]
values are [ 0.          0.59667692  0.10761908  0.6736734   0.46349645  0.
  0.44056863  0.10701816  0.67167752  0.29158274  0.          0.14273156
  0.34912309  0.43812636  0.58484507  0.          0.57758441  0.4284959
  0.7292129   0.06063283]
type is <type 'numpy.float64'>

This related question might be useful.

这篇关于在多处理进程之间共享大型只读 Numpy 数组的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！