Windows 上的多处理中断 [英] Multiprocessing on Windows breaks

问题描述

我在 Linux 上使用 Python 进行开发，但从未真正见过 Windows 出现此类问题.我正在使用 multiprocessing 库来加速计算，这对我在 Linux 上非常有效.

I develop with Python on Linux and have never really seen this sort of problem with Windows. I'm using the multiprocessing library to speed up computations, which works very well for me on Linux.

然而，在 Windows 上，事情并不顺利:

On Windows, however, things don't run as smoothly:

 * [INFO] Parsing 1 file using 2 threads

Traceback (most recent call last):
  File "main.py", line 170, in <module>
    master = ParsingMaster(parser, list(input_file), output_list, threads=num_threads)
Traceback (most recent call last):
  File "main.py", line 39, in __init__
  File "<string>", line 1, in <module>
    self.input_process.start()
  File "C:Python26libmultiprocessingforking.py", line 342, in main
  File "C:Python26libmultiprocessingprocess.py", line 104, in start
        self._popen = Popen(self)
self = load(from_parent)
  File "C:Python26libmultiprocessingforking.py", line 239, in __init__
  File "C:Python26libpickle.py", line 1370, in load
    dump(process_obj, to_child, HIGHEST_PROTOCOL)
  File "C:Python26libmultiprocessingforking.py", line 162, in dump
    ForkingPickler(file, protocol).dump(obj)
  File "C:Python26libpickle.py", line 224, in dump
    return Unpickler(file).load()
    self.save(obj)
 File "C:Python26libpickle.py", line 858, in load
  File "C:Python26libpickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:Python26libpickle.py", line 419, in save_reduce
    dispatch[key](self)
  File "C:Python26libpickle.py", line 880, in load_eof
    save(state)
  File "C:Python26libpickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
    r aise EOFError
 File "C:Python26libpickle.py", line 649, in save_dict
EOFError
    self._batch_setitems(obj.iteritems())
  File "C:Python26libpickle.py", line 681, in _batch_setitems
    save(v)
  File "C:Python26libpickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:Python26libmultiprocessingforking.py", line 40, in dispatcher
    self.save_reduce(obj=obj, *rv)
  File "C:Python26libpickle.py", line 401, in save_reduce
    save(args)
  File "C:Python26libpickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:Python26libpickle.py", line 548, in save_tuple
    save(element)
  File "C:Python26libpickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:Python26libpickle.py", line 419, in save_reduce
    save(state)
  File "C:Python26libpickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:Python26libpickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "C:Python26libpickle.py", line 681, in _batch_setitems
    save(v)
  File "C:Python26libpickle.py", line 306, in save
    rv = reduce(self.proto)
  File "C:Python26libmultiprocessingmanagers.py", line 458, in __reduce__
    return type(self).from_address, 
AttributeError: type object 'SyncManager' has no attribute 'from_address'

我正在 Windows 7 上同时测试 Python 2.6 和 2.7，并且一遍又一遍地遇到同样的错误.有人知道是什么意思吗?

I'm testing on both Python 2.6 and 2.7 on Windows 7 and get this same error over and over. Does anybody know what it means?

推荐答案

Windows有限制，这里是你看到的错误的相关部分:

There are restrictions on Windows, here is the relevant parts to the errors you are seeing:

由于 Windows 缺少 os.fork() 它有一些额外的限制:

更易腌制

确保 Process.__init__() 的所有参数都是可挑选的.这尤其意味着不能使用绑定或未绑定的方法直接作为 Windows 上的目标参数 - 只需定义一个函数并改用它.

Ensure that all arguments to Process.__init__() are picklable. This means, in particular, that bound or unbound methods cannot be used directly as the target argument on Windows — just define a function and use that instead.

此外，如果您将 Process 子类化，请确保实例是调用 Process.start() 方法时可挑选.

Also, if you subclass Process then make sure that instances will be picklable when the Process.start() method is called.

这意味着作为参数传递给 Process.__init__() 不能被腌制或解封(Python中的序列化).什么是 SyncManager 它抱怨无法在该对象上找到属性 AttributeError: type object 'SyncManager' has no attribute 'from_address'，这可能是你的根本原因.SyncManager 对象真的可以腌制吗，是否符合pickle规则?

This means that something that is being passed as an argument to Process.__init__() isn't able to be pickled or unpickled ( a serialization in Python ). What is SyncManager it is complaining about not being able to find attributes on that object AttributeError: type object 'SyncManager' has no attribute 'from_address', it is probably your root cause. Can that SyncManager object actually be pickled, does it meet the pickle rules?

如果您是从 Windows 上的 命令行运行它，您显然也不能这样做.

If you are running this from the command line on Windows, you can't do that either apparently.

不要那样做.将代码保存在文件中，然后使用以下命令从文件中运行它:

Don't do that. Save the code in a file and run it from the file instead, with the command:

python myfile.py

这将解决您的问题.

这篇关于Windows 上的多处理中断的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！