无法在Windows 10上使用"PicklingError"启动Celery worker. [英] Can't start Celery worker on Windows 10 with "PicklingError"

问题描述

我有一个简单的测试代码，可以在Linux上成功运行，但是无法在Windows 10 x64计算机上运行.

I have a simple test code that runs successfully on Linux, but it won't run on my windows 10 x64 computer.

当我尝试启动芹菜工作者时，它抱怨无法恢复的错误:PicklingError.(Celery版本:3.1.20)

When I tried to start a celery worker, it complained about the unrecoverable error: PicklingError. (Celery version: 3.1.20)

在我的celery配置中，我已将序列化设置为'json'，但它仍然没有任何帮助.

In my celery config, I've set the serialization to 'json', but it still didn't help at all.

CELERY_RESULT_SERIALIZER = 'json'
CELERY_TASK_SERIALIZER = 'json'
CELERY_ACCEPT_CONTENT = ['json']

这是完整的错误消息:

[2016-02-09 15:11:48,532: ERROR/MainProcess] Unrecoverable error: PicklingError("Can't pickle <type 'module'>: it's not found as __builtin__.module",)

Traceback (most recent call last):
  File "C:\Python27\lib\site-packages\celery-3.1.20-py2.7.egg\celery\worker\__init__.py", line 206, in start
    self.blueprint.start(self)   
  File "C:\Python27\lib\site-packages\celery-3.1.20-py2.7.egg\celery\bootsteps.py", line 123, in start
    step.start(parent)
  File "C:\Python27\lib\site-packages\celery-3.1.20-py2.7.egg\celery\bootsteps.py", line 374, in start
    return self.obj.start()
  File "C:\Python27\lib\site-packages\celery-3.1.20-py2.7.egg\celery\concurrency\base.py", line 131, in start
    self.on_start()
  File "C:\Python27\lib\site-packages\celery-3.1.20-py2.7.egg\celery\concurrency\prefork.py", line 117, in on_start
    **self.options)
  File "C:\Python27\lib\site-packages\billiard\pool.py", line 972, in __init__
    self._create_worker_process(i)
  File "C:\Python27\lib\site-packages\billiard\pool.py", line 1068, in _create_worker_process
    w.start()
  File "C:\Python27\lib\site-packages\billiard\process.py", line 137, in start
    self._popen = Popen(self)
  File "C:\Python27\lib\site-packages\billiard\forking.py", line 263, in __init__
    dump(process_obj, to_child, HIGHEST_PROTOCOL)
  File "C:\Python27\lib\site-packages\billiard\py2\reduction.py", line 84, in dump
    ForkingPickler(file, protocol).dump(obj)
  File "C:\Python27\lib\pickle.py", line 224, in dump
    self.save(obj)
  File "C:\Python27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Python27\lib\pickle.py", line 401, in save_reduce
    save(args)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 562, in save_tuple
    save(element)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 548, in save_tuple
    save(element)
  File "C:\Python27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Python27\lib\pickle.py", line 401, in save_reduce
    save(args)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 548, in save_tuple
    save(element)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "C:\Python27\lib\pickle.py", line 681, in _batch_setitems
    save(v)
  File "C:\Python27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Python27\lib\pickle.py", line 396, in save_reduce
    save(cls)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 748, in save_global
    (obj, module, name))
PicklingError: Can't pickle <type 'module'>: it's not found as __builtin__.module

推荐答案

我遇到了同样的问题.奇怪的是，该问题仅存在于Windows上，在Linux Celery上运行时没有任何问题.原来我需要将配置模块作为名称而不是作为对象传递:

I ran into the same problem. What was weird is that the problem only existed on Windows, on Linux Celery was running without any problems. Turns out I needed to pass the configuration module as name not as object:

app.config_from_object('celeryconfig')

代替

app.config_from_object(celeryconfig)

来自 Celery文档:

提示

建议使用模块名称，因为这意味着在使用前叉池时不需要序列化该模块.如果遇到配置错误，请尝试使用模块名称.

Tip

Using the name of a module is recommended as this means that the module doesn’t need to be serialized when the prefork pool is used. If you’re experiencing configuration pickle errors then please try using the name of a module instead.

很显然，当Celery作为在Windows上失败的对象传递时，它需要使配置腌制.如果作为模块名称传递，它将起作用.感谢您为我指示正确的方向@JoyLy！

Apparently Celery needs to pickle the configuration when it's passed as an object which fails on Windows. If passed as module name it works. Thanks for pointing me in the right direction @JoyLy!

这篇关于无法在Windows 10上使用"PicklingError"启动Celery worker.的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！