为什么multiprocessing.Process在Windows和Linux上对于全局对象和函数参数的行为有所不同 [英] Why multiprocessing.Process behave differently on windows and linux for global object and function arguments

问题描述

以下代码在Windows和Linux(均与python2.7一起运行)上运行时具有不同的输出

The following code has different output when running on windows and linux (both with python2.7)

'''import_mock.py'''
to_mock = None

'''test.py'''
import import_mock
from multiprocessing import Process

class A(object):
    def __init__(self):
        self.a = 1
        self.b = 2
        self.c = 3

    def __getstate__(self):
        print '__getstate__'
        return { 'a': self.a, 'b': self.b,
                 'c':0 }

def func():
    import_mock.to_mock = 1
    a = A()
    return a

def func1(a):
    print a.a, a.b, a.c
    print import_mock.to_mock


if __name__ == '__main__':
    a = func()
    p = Process(target=func1, args=(a,))
    p.start()
    p.join()

在Windows上，输出为:

On windows, the output is:

__getstate__
1 2 0
None

我期望的是什么

在linux上是:

1 2 3
1

不克隆全局对象和传递的参数.

Which not clone the global object and the passed args.

我的问题是为什么他们的行为有所不同?以及如何使linux代码的行为与Windows相同?

My question is why they behave differently? And how to make the linux code behave the same as windows one?

推荐答案

添加到@Blckknght的答案:在Windows上，每个进程从头开始"导入原始模块，而在Unix-y系统上，仅主进程运行整个进程模块，而所有其他进程在使用fork()时都可以看到存在的内容(新的，您是自己不是在调用fork()，multiprocessing内部人员会在任何时候调用它)创建一个新进程).

Adding to @Blckknght's answer: on Windows, each process imports the original module "from scratch", while on Unix-y systems only the main process runs the whole module, while all other processes see whatever exists at the time fork() is used to create the new processes (no, you're not calling fork() yourself - multiprocessing internals call it whenever it creates a new process).

详细说明，用于您的import_mock:

• 在所有平台上，主进程调用func()，它将import_mock.to_mock设置为1.

在Unix-y平台上，这是所有新进程所看到的:fork()在此之后发生，因此1是所有新进程继承的状态.

On Unix-y platforms, that's what all new processes see: the fork() occurs after that, so 1 is the state all new processes inherit.

在Windows上，所有新进程都从头开始"运行整个模块.因此，他们每个人都导入自己的全新版本的import_mock.只有主进程调用func()，因此只有主进程看到to_mock更改为1.所有其他进程看到最新的None状态.

On Windows, all new processes run the entire module "from scratch". So they each import their own, brand new version of import_mock. Only the main process calls func(), so only the main process sees to_mock change to 1. All other processes see the fresh None state.

这是预料之中的，并且第二次实际上很容易理解；-)

That's all expected, and actually easy to understand the second time ;-)

传递a的过程是微妙的，因为它更多地取决于multiprocessing实现细节.实现可以从一开始就选择在所有平台上腌制参数，但事实并非如此，现在进行更改而又不破坏 some 平台上的内容为时已晚.

What's going on with passing a is subtler, because it depends more on multiprocessing implementation details. The implementation could have chosen to pickle arguments on all platforms from the start, but it didn't, and now it's too late to change without breaking stuff on some platforms.

由于写时复制fork()语义，在Unix-y系统上腌制Process()自变量不是必需的，因此实现从未如此.但是，如果没有fork()，则必须在Windows上对它们进行腌制-这样实现就可以了.

Because of copy-on-write fork() semantics, it wasn't necessary to pickle Process() arguments on Unix-y systems, and so the implementation never did. However, without fork() it is necessary to pickle them on Windows - and so the implementation does.

在使用Python 3.4(允许您在所有平台上强制执行"Windows实现"(spawn))之前，没有机械方法来避免可能出现的跨平台意外情况.

Before Python 3.4, which allows you to force "the Windows implementation" (spawn) on all platforms, there's no mechanical way to avoid possible cross-platform surprises.

但是实际上，我很少为此感到困扰. 知道，例如，多处理可能严重依赖于酸洗，我完全不知道要和酸黄瓜一起玩花招.传递A()实例时出现问题"的唯一原因是您正在玩腌菜(通过覆盖默认的__getstate__()).

But in practice, I've rarely been bothered by this. Knowing that, for example, multiprocessing can depend heavily on pickling, I stay completely clear of getting anywhere near playing tricks with pickles. The only reason you had "a problem" passing an A() instance is that you are playing pickle tricks (via overriding the default __getstate__()).

这篇关于为什么multiprocessing.Process在Windows和Linux上对于全局对象和函数参数的行为有所不同的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！