如何获取python的multiprocessing array'pointer并将其传递给Cpp程序? [英] How to get python's multiprocessing array'pointer and pass it to Cpp program?

问题描述

我现在需要在python中请求数组，并将其传递到Cpp程序中，但是python仍然需要处理它们.但是我发现当我使用多重处理时，数组的adderess将会被更改.

I now need to request arrays in python, and pass them into a Cpp program, yet python still needs to process them. But I found that when I use multiprocessing, the array's adderess would be changed.

下面是我的代码:

//export dll in test.h, this is test.cpp
#include "test.h"
#include <iostream>
using namespace std;

void someWork(double* data, long* flag){
    cout << "cpp flag address:" << flag << endl;
    //do some work
}

# test.py
import multiprocessing as mp
from multiprocessing.sharedctypes import RawArray
import ctypes

data = RawArray(ctypes.c_double, 2000)
flag = RawArray(ctypes.c_long, 20)
pkg = ctypes.cdll.LoadLibrary(r"test.dll")
pkg.someWork.argtypes = [
    ctypes.POINTER(ctypes.c_double * 2000),# dataArray
    ctypes.POINTER(ctypes.c_long * 20)#flagArray
]

def proc_py():
    idx = 0
    while True:
        if flag[idx] == 1:
            # do something
            flag[idx] = 0
            idx = (idx + 1) % 20

def proc_cpp():
    pkg.someWork(ctypes.pointer(data), ctypes.pointer(flag))

def main():
    p_cpp = mp.Process(target=proc_cpp, args=())
    p_py = mp.Process(target=proc_py, args=())
    p_cpp .start()
    p_py .start()
    p_cpp .join()
    p_py .join()

if __name__ == '__main__':
    print("py flag address:", ctypes.byref(flag))
    # proc_cpp()
    main()

结果是:当我在python中运行 proc_cpp 时，地址是相同的:

The result is: when I just run proc_cpp in python, the address are the same:

py flag address: <cparam 'P' (0000019DA8282400)>
cpp flag address:   0000019DA8282400

但是当我运行 main 时，地址是不同的:

But when I run main, the address are different:

py flag address: <cparam 'P' (000001CB42A32400)>
cpp flag address:   0000012F1E152400

我知道python的多重处理必须使用共享内存才能在处理之间共享内存，但是通过使用 mp.Array / Array.get_obj()和 mp.sharedctypes.RawArray / ctypes.pointer().有什么办法可以解决我的问题?

I know that python's multiprocessing must use shared memory to share memory between processings, but I failed both by using mp.Array/Array.get_obj() and mp.sharedctypes.RawArray/ctypes.pointer(). Is there any way to solve my question?

推荐答案

请勿在运行一次"之外创建 RawArray .主代码，或者您正在制作不同数组.在主进程中一次创建 RawArray ，然后将该 RawArray 作为参数传递给新进程的目标函数.每个进程看到"虚拟地址.会有所不同，但物理内存将相同.

Do not create the RawArray outside of the "run once" main code or you are making different arrays. Create the RawArray once in the main process, and pass that RawArray as a parameter to the target function of the new processes. The virtual address each process "sees" will be different, but the physical memory will be the same.

这是一个例子:

test.cpp:

这将显示指针地址，然后更改共享数组中的指定索引.

This will display the pointer address and then change the specified index in the shared array.

#include <iostream>
#include <cstdint>
using namespace std;

#define API __declspec(dllexport)

extern "C" API
void set(double* data, int index, double value) {
    cout << data << ' ' << index << ' ' << value << endl;
    data[index] = value;
}

test.py:

这会将共享数组传递给每个进程.主要过程还将更改元素.使用该锁是因为RawArray不同步，否则C ++代码中的打印会混乱，因此该代码不会真正并行运行，但确实说明了这些进程获得了不同的虚拟地址，但是共享相同的数据.

This passes the shared array to each process. The main process will also change an element. The lock is used because RawArray is not synchronized and the printing in the C++ code will mess up otherwise, so this code won't really run in parallel but it does illustrate that the processes get different virtual addresses but share the same data.

import multiprocessing as mp
from multiprocessing.sharedctypes import RawArray
from ctypes import *

dll = CDLL('./test')
dll.set.argtypes = POINTER(c_double),c_int,c_double
dll.set.restype = None

def call(lock,data,index,value):
    with lock:
        dll.set(data,index,value)

if __name__ == '__main__':

    # This code runs once in the main process.
    # The lock and shared data are created once only and passed to other processes.

    lock = mp.Lock()
    data = RawArray(c_double, 3)
    data[0] = 0.5
    p1 = mp.Process(target=call, args=(lock,data,1,1.25))
    p2 = mp.Process(target=call, args=(lock,data,2,2.5))
    p1.start()
    p2.start()
    p1.join()
    p2.join()
    print(list(data))

输出(不同的地址，相同的共享数据):

Output (different addresses, same shared data):

00000269D66E0000 1 1.25
00000187F0B90000 2 2.5
[0.5, 1.25, 2.5]

这篇关于如何获取python的multiprocessing array'pointer并将其传递给Cpp程序?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！