多重处理和带指针的ctypes [英] multiprocessing and ctypes with pointers

问题描述

我有multiProcessing.Process个对象，它们的目标函数带有输入和输出队列.

I have multiProcessing.Process objects whose target functions take input and output queue.

他们将一些数据放入输出队列，该数据是带有内部指针的包装ctypes结构.当然，应该序列化数据的pickle模块会中断:

To the output queue they put some data, that is a wrapped ctypes structure with internal pointers. Of course, the pickle module, that should serialize the data, breaks:

ValueError:不能腌制包含指针的ctypes对象

ValueError: ctypes objects containing pointers cannot be pickled

我能以某种方式从子进程中获取带有指针的ctypes结构而不将其转储到文件中吗?

Can I somehow get my ctypes structures with pointers out of my child processes without dumping them to files?

代码在下面

# -*- coding: utf-8 -*-
import multiprocessing as mp

from liblinear import *
from liblinearutil import *


def processTarget(inQueue, outQueue):
    while(not inQueue.empty()):
        inVal = inQueue.get()

        #training model
        y, x = [1,-1], [{1:inVal, 3:3*inVal}, {1:-1,3:-1}]
        prob  = problem(y, x)
        param = parameter('-c 4 -B 1')
        m = train(prob, param)


        outQueue.put((inVal * 2, m))
        print "done", inVal
        inQueue.task_done()

def Main():
    processes = []
    inQueue = mp.JoinableQueue()
    for i in xrange(10):
        inQueue.put(i)

    outQueue = mp.JoinableQueue()
    for i in xrange(5):
        process = mp.Process(target=processTarget, args=(inQueue, outQueue))
        print "starting", i
        process.start()
        print "started", i
    inQueue.join()

    print "JOINED"

    while(not outQueue.empty()):
        print outQueue.get()



if __name__ == '__main__':
    Main()

推荐答案

使用多处理时，每个进程都有其自己的地址空间.该指针在另一个进程中将无效.

When you use multiprocessing every process has its own address space. The pointer would not be valid in another process.

将对象转换为python对象或不带指针的ctypes类型，它应该可以工作.

translate the object to a python object or to a ctypes type without pointers and it should work.

请记住，除非将对象发送回队列，否则其他过程中发生的对象更改不会反映在父对象中.

Keep in mind that changes to the object that occur in the other process will not be reflected in the parent unless you send the object back on the queue.

这篇关于多重处理和带指针的ctypes的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！