Windows上的多处理中断 [英] Multiprocessing on Windows breaks

问题描述

我在Linux上使用Python开发，却从未真正在Windows上看到过这种问题.我正在使用multiprocessing库来加快计算速度，这在Linux上对我来说效果很好.

I develop with Python on Linux and have never really seen this sort of problem with Windows. I'm using the multiprocessing library to speed up computations, which works very well for me on Linux.

但是，在Windows上，运行并不顺利:

On Windows, however, things don't run as smoothly:

 * [INFO] Parsing 1 file using 2 threads

Traceback (most recent call last):
  File "main.py", line 170, in <module>
    master = ParsingMaster(parser, list(input_file), output_list, threads=num_threads)
Traceback (most recent call last):
  File "main.py", line 39, in __init__
  File "<string>", line 1, in <module>
    self.input_process.start()
  File "C:\Python26\lib\multiprocessing\forking.py", line 342, in main
  File "C:\Python26\lib\multiprocessing\process.py", line 104, in start
        self._popen = Popen(self)
self = load(from_parent)
  File "C:\Python26\lib\multiprocessing\forking.py", line 239, in __init__
  File "C:\Python26\lib\pickle.py", line 1370, in load
    dump(process_obj, to_child, HIGHEST_PROTOCOL)
  File "C:\Python26\lib\multiprocessing\forking.py", line 162, in dump
    ForkingPickler(file, protocol).dump(obj)
  File "C:\Python26\lib\pickle.py", line 224, in dump
    return Unpickler(file).load()
    self.save(obj)
 File "C:\Python26\lib\pickle.py", line 858, in load
  File "C:\Python26\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Python26\lib\pickle.py", line 419, in save_reduce
    dispatch[key](self)
  File "C:\Python26\lib\pickle.py", line 880, in load_eof
    save(state)
  File "C:\Python26\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
    r aise EOFError
 File "C:\Python26\lib\pickle.py", line 649, in save_dict
EOFError
    self._batch_setitems(obj.iteritems())
  File "C:\Python26\lib\pickle.py", line 681, in _batch_setitems
    save(v)
  File "C:\Python26\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python26\lib\multiprocessing\forking.py", line 40, in dispatcher
    self.save_reduce(obj=obj, *rv)
  File "C:\Python26\lib\pickle.py", line 401, in save_reduce
    save(args)
  File "C:\Python26\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python26\lib\pickle.py", line 548, in save_tuple
    save(element)
  File "C:\Python26\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Python26\lib\pickle.py", line 419, in save_reduce
    save(state)
  File "C:\Python26\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python26\lib\pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "C:\Python26\lib\pickle.py", line 681, in _batch_setitems
    save(v)
  File "C:\Python26\lib\pickle.py", line 306, in save
    rv = reduce(self.proto)
  File "C:\Python26\lib\multiprocessing\managers.py", line 458, in __reduce__
    return type(self).from_address, \
AttributeError: type object 'SyncManager' has no attribute 'from_address'

我正在Windows 7上同时在python 2.6和2.7上进行测试，并且反复出现相同的错误.有人知道这是什么意思吗?

I'm testing on both Python 2.6 and 2.7 on Windows 7 and get this same error over and over. Does anybody know what it means?

推荐答案

在Windows上有限制，以下是与您看到的错误有关的部分:

There are restrictions on Windows, here is the relevant parts to the errors you are seeing:

由于Windows缺少os.fork()，因此它具有一些额外的限制:

更高的可腌性

确保Process.__init__()的所有自变量都是可腌制的.这 尤其意味着不能使用绑定方法或未绑定方法 直接作为Windows上的目标参数-只需定义一个函数 并改用它.

Ensure that all arguments to Process.__init__() are picklable. This means, in particular, that bound or unbound methods cannot be used directly as the target argument on Windows — just define a function and use that instead.

此外，如果您将Process子类化，则请确保实例将 调用Process.start()方法时可腌制.

Also, if you subclass Process then make sure that instances will be picklable when the Process.start() method is called.

这意味着无法将作为参数传递给Process.__init__()的内容被腌制或未腌制(Python中的序列化). SyncManager是什么，它抱怨无法在该对象AttributeError: type object 'SyncManager' has no attribute 'from_address'上找到属性，这可能是您的根本原因.可以真正腌制该SyncManager对象吗?是否符合泡菜规则?

This means that something that is being passed as an argument to Process.__init__() isn't able to be pickled or unpickled ( a serialization in Python ). What is SyncManager it is complaining about not being able to find attributes on that object AttributeError: type object 'SyncManager' has no attribute 'from_address', it is probably your root cause. Can that SyncManager object actually be pickled, does it meet the pickle rules?

如果您是通过Windows上的命令行运行的，则您显然不能做到这一点.

If you are running this from the command line on Windows, you can't do that either apparently.

不要那样做.将代码保存在文件中，然后使用以下命令从文件中运行代码:

Don't do that. Save the code in a file and run it from the file instead, with the command:

python myfile.py

那将解决您的问题.

这篇关于Windows上的多处理中断的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！