将multiprocessing.RawArray传递给C ++函数 [英] Passing multiprocessing.RawArray to a C++ function

问题描述

我的Python应用程序使用multiprocessing.RawArray创建一个在进程之间共享的数组.现在，为了加快计算速度，我想从C ++函数中修改此数组.将指向底层内存的指针传递给接受void *参数的C ++函数的安全方法是什么?

My Python application creates an array shared between processes using multiprocessing.RawArray. Now to speed up computation I want to modify this array from within a C++ function. What is a safe way to pass a pointer to the underlying memory to a C++ function that accepts a void * argument?

该功能在pxd文件中定义为:

The function is defined in a pxd file as:

cdef extern from 'lib/lib.hpp':
    void fun(void *buffer)

到目前为止我的幼稚尝试:

My naive attempt so far:

buffer = multiprocessing.RawArray(ctypes.c_ubyte, 10000)
clib.fun(ctypes.cast(self.queue_obj_buffer, ctypes.c_void_p))

这会使Cython编译失败，并出现以下错误:Cannot convert Python object to 'void *'我也尝试了ctypes.addressof，结果相似.

This fails Cython compilation with the following error: Cannot convert Python object to 'void *' I also tried ctypes.addressof with similar results.

我确实知道我将需要一种方法来从每个参与的进程中单独查询该指针，因为该相同的内存区域在进程地址空间中的映射方式将有所不同.但这不是问题，到目前为止，我只是在努力获取所有指针.我应该完全使用其他方法从C ++中分配共享内存，还是可以做我正在做的事情?

I do understand that I will need a method to query this pointer from every participating process individually, because this same region of memory will be mapped differently in process address spaces. But this is not an issue, so far I'm just struggling to get the pointer at all. Should I use a different approach altogether and allocate shared memory from within C++, or is it okay to do what I am doing?

推荐答案

multiprocessing.RawArray是 ctypes.Array ，因此可以通过

multiprocessing.RawArray is a ctypes.Array, so the address of the underlying buffer can be obtained via ctypes.addressof. This address can be reinterpreted as void *. Here is an example:

%%cython
# a small function for testing purposes:
cdef extern from *:
    """
    unsigned char get_first(void *ptr){
       unsigned char *ptr_as_ubytes = (unsigned char *)ptr;
       return ptr_as_ubytes[0];
    }
    """
    unsigned char get_first(void *ptr)


import ctypes
def first_element(buffer):
    cdef size_t ptr_address = ctypes.addressof(buffer) # size_t is big enough to hold the address
    return get_first(<void*> ptr_address)

使用<void*>ctypes.addressof(buffer)无效，因为Cython无法将PyObject自动转换为void *-(不太易读)oneliner将是<void*><size_t> ctypes.addressof(buffer):

Using <void*>ctypes.addressof(buffer) won't work, because Cython has no means for automatic conversion of a PyObject to void * - the (less readable) oneliner would be <void*><size_t> ctypes.addressof(buffer):

• Cython可以将Python对象转换为原始的size_t(或任何整数)C值.
• size_t C值可以用C语言重新解释为void *.

• Cython can convert a Python-object to a raw size_t (or any integer) C-value.
• a size_t C-value can be reinterpreted as void * in C-language.

这是上面示例功能的一个小测试:

Here is a small test of above example's functionality:

import multiprocessing
import ctypes
buffer = multiprocessing.RawArray(ctypes.c_ubyte, 10000)
buffer[0]=42
first_element(buffer)
# 42

---

如果C函数的签名不希望使用void *，而是希望使用unsigned char类型的连续内存，则@ oz1的方法更安全，因为它不仅可以防止数据被错误地重新解释，而且可以防止数据被错误地重新解释.还会自动检查缓冲区是否连续并且具有正确的维数(通过键入unsigned char[::1]完成).

---

If the signature of the C-function isn't expecting a void * but for example continuous memory of type unsigned char, so the approach from @oz1 is safer, as it not only protects data from being wrongly reinterpreted but also automatically checks that the buffer is continuous and has the right number of dimensions (done via typing as unsigned char[::1]).

这篇关于将multiprocessing.RawArray传递给C ++函数的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！