通知芹菜任务工人关闭 [英] Notify celery task of worker shutdown

问题描述

我将 celery 2.4.1 与 python 2.6、rabbitmq 后端和 django 一起使用.如果工人关闭，我希望我的任务能够正确清理.据我所知，您无法提供任务析构函数，因此我尝试连接到 worker_shutdown 信号.

I am using celery 2.4.1 with python 2.6, the rabbitmq backend, and django. I would like my task to be able to clean up properly if the worker shuts down. As far as I am aware you cannot supply a task destructor so I tried hooking into the worker_shutdown signal.

注意:AbortableTask 仅适用于数据库后端，所以我不能使用它.

Note: AbortableTask only works with the database backend so I cant use that.

from celery.signals import worker_shutdown

@task
def mytask(*args)

  obj = DoStuff()

  def shutdown_hook(*args):
     print "Worker shutting down"
     # cleanup nicely
     obj.stop()

  worker_shutdown.connect(shutdown_hook)

  # blocking call that monitors a network connection
  obj.stuff()

但是，关闭挂钩永远不会被调用.Ctrl-C'ing 工人不会杀死任务，我必须从 shell 手动杀死它.

However, the shutdown hook never gets called. Ctrl-C'ing the worker doesnt kill the task and I have to manually kill it from the shell.

如果这不是正确的方法，我该如何让任务正常关闭?

So if this is not the proper way to go about it, how do I allow tasks to shutdown gracefully?

推荐答案

worker_shutdown 仅由 MainProcess 发送，而非子池工作者.所有worker_*信号除了worker_process_init，参考MainProcess.

worker_shutdown is only sent by the MainProcess, not the child pool workers. All worker_* signals except for worker_process_init, refer to the MainProcess.

但是，关闭挂钩永远不会被调用.Ctrl-C'ing 工人不会终止任务，我必须从 shell 手动终止它.

However, the shutdown hook never gets called. Ctrl-C'ing the worker doesn't kill the task and I have to manually kill it from the shell.

在正常(热)关闭的情况下，工作人员永远不会终止任务.即使任务需要几天才能完成，工作人员也不会完成关机直到完成.您可以将 --soft-time-limit 或 --time-limit 设置为告诉实例何时可以终止任务.

The worker never terminates a task under normal (warm) shutdown. Even if a task takes days to complete, the worker won't complete shutdown until it's completed. You can set --soft-time-limit, or --time-limit to to tell the instance when it's ok to terminate the task.

因此，要添加任何类型的进程清理进程，您首先需要确保任务能够真正完成.因为清理不会在此之前被调用.

So to add any kind of process cleanup process you first need to make sure that the tasks can actually complete. As the cleanup wouldn't be called before that happens.

要向池工作进程添加清理步骤，您可以使用类似:

To add a cleanup step to the pool worker processes you can use something like:

from celery import platforms
from celery.signals import worker_process_init

def cleanup_after_tasks(signum, frame):
    # reentrant code here (see http://docs.python.org/library/signal.html)

def install_pool_process_sighandlers(**kwargs):
    platforms.signals["TERM"] = cleanup_after_tasks
    platforms.signals["INT"] = cleanup_after_tasks

worker_process_init.connect(install_pool_process_sighandlers)

这篇关于通知芹菜任务工人关闭的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！