来自numpy数组的RawArray? [英] RawArray from numpy array?

问题描述

我想在多个进程之间共享一个numpy数组.进程仅读取数据，因此我想避免进行复制.如果可以以multiprocessing.sharedctypes.RawArray开头，然后使用numpy.frombuffer创建一个numpy数组，我知道该怎么做.但是，如果最初给我一个numpy数组怎么办?有没有一种方法可以用numpy数组的数据初始化RawArray而不复制数据?还是有另一种无需复制就可以跨进程共享数据的方法?

I want to share a numpy array across multiple processes. The processes only read the data, so I want to avoid making copies. I know how to do it if I can start with a multiprocessing.sharedctypes.RawArray and then create a numpy array using numpy.frombuffer. But what if I am initially given a numpy array? Is there a way to initialize a RawArray with the numpy array's data without copying the data? Or is there another way to share the data across the processes without copying it?

推荐答案

据我所知，在将内存分配给特定进程后，不可能将其声明为共享内存.可以在此处和

To my knowledge it is not possible to declare memory as shared after it was assigned to a specific process. Similar discussions can be found here and here (more suitable).

让我快速概述一下您提到的解决方法(从RawArray开始并获得numpy.ndarray引用).

Let me quickly sketch the workaround you mentioned (starting with a RawArray and get a numpy.ndarray refference to it).

import numpy as np
from multiprocessing.sharedctypes import RawArray
# option 1
raw_arr = RawArray(ctypes.c_int, 12)
# option 2 (set is up, similar to some existing np.ndarray np_arr2)
raw_arr = RawArray(
        np.ctypeslib.as_ctypes_type(np_arr2.dtype), len(np_arr2)
        )
np_arr = np.frombuffer(raw_arr, dtype=np.dtype(raw_arr))
# np_arr: numpy array with shared memory, can be processed by multiprocessing

如果必须以numpy.ndarray开头，则除了复制数据外别无选择

If you have to start with a numpy.ndarray, you have no other choice as to copy the data

import numpy as np
from multiprocessing.sharedctypes import RawArray

np_arr = np.zeros(shape=(3, 4), dtype=np.ubyte)
# option 1
tmp = np.ctypeslib.as_ctypes(np_arr)
raw_arr = RawArray(tmp._type_, tmp)
# option 2
raw_arr = RawArray(np.ctypeslib.as_ctypes_type(np_arr.dtype), np_arr.flatten())

print(raw_arr[:])

这篇关于来自numpy数组的RawArray?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！