如何更改多处理模块使用的序列化方法? [英] How to change the serialization method used by the multiprocessing module?

问题描述

如何更改Python multiprocessing库使用的序列化方法?特别是，默认的序列化方法将pickle库与该Python版本的默认pickle协议版本一起使用.默认的pickle协议是Python 2.7中的版本2和Python 3.6中的版本3.如何在Python 3.6中将协议版本设置为2，以便可以使用multiprocessing库中的某些类(例如Client和Listener)在Python 2.7运行的服务器处理与客户端之间进行通信由Python 3.6运行的进程?

How can I change the serialization method used by the Python multiprocessing library? In particular, the default serialization method uses the pickle library with the default pickle protocol version for that version of Python. The default pickle protocol is version 2 in Python 2.7 and version 3 in Python 3.6. How can I set the protocol version to 2 in Python 3.6, so I can use some of the classes (like Client and Listener) in the multiprocessing library to communicate between a server processing run by Python 2.7 and a client process run by Python 3.6?

(旁注:作为测试，我修改了 multiprocessing/connection.py 的第206行，方法是在dump()调用中添加protocol=2以将协议版本强制为2，并且我的客户端/服务器进程在我的有限测试中正常工作，服务器运行2.7，客户端运行3.6).

(Side note: as a test, I modified line 206 of multiprocessing/connection.py by adding protocol=2 to the dump() call to force the protocol version to 2 and my client/server processes worked in my limited testing with the server run by 2.7 and the client by 3.6).

在Python 3.6中，已合并补丁以设置序列化程序，但该补丁是没有记录，我还没有弄清楚如何使用它.这是我尝试使用它的方法(我也将其发布到了链接到的Python票证上):

In Python 3.6, a patch was merged to let the serializer be set, but the patch was undocumented, and I haven't figured out how to use it. Here is how I tried to use it (I posted this also to the Python ticket that I linked to):

pickle2reducer.py:

pickle2reducer.py:

from multiprocessing.reduction import ForkingPickler, AbstractReducer

class ForkingPickler2(ForkingPickler):
    def __init__(self, *args):
        if len(args) > 1:
            args[1] = 2
        else:
            args.append(2)
        super().__init__(*args)

    @classmethod
    def dumps(cls, obj, protocol=2):
        return ForkingPickler.dumps(obj, protocol)


def dump(obj, file, protocol=2):
    ForkingPickler2(file, protocol).dump(obj)


class Pickle2Reducer(AbstractReducer):
    ForkingPickler = ForkingPickler2
    register = ForkingPickler2.register
    dump = dump

和我的客户中

import pickle2reducer
multiprocessing.reducer = pickle2reducer.Pickle2Reducer()

使用multiprocessing进行其他操作之前，请先将

放在顶部.这样做时，我仍然在由Python 2.7运行的服务器上看到ValueError: unsupported pickle protocol: 3.

at the top before doing anything else with multiprocessing. I still see ValueError: unsupported pickle protocol: 3 on the server run by Python 2.7 when I do this.

推荐答案

如果您使用的是多处理程序，我相信您所指的补丁程序是有效的.

I believe the patch you're referring to works if you're using a multiprocessing "context" object.

使用pickle2reducer.py，您的客户应以:

Using your pickle2reducer.py, your client should start with:

import pickle2reducer
import multiprocessing as mp

ctx = mp.get_context()
ctx.reducer = pickle2reducer.Pickle2Reducer()

ctx和multiprocessing具有相同的API.

And ctx has the same API as multiprocessing.

希望有帮助！

这篇关于如何更改多处理模块使用的序列化方法?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！