通知工人芹菜任务关闭 [英] Notify celery task of worker shutdown

问题描述

我将celery 2.4.1与python 2.6，rabbitmq后端和Django一起使用。我希望我的任务能够在工人关闭的情况下正确清理。据我所知，您无法提供任务析构函数，因此我尝试连接到 worker_shutdown 信号。

I am using celery 2.4.1 with python 2.6, the rabbitmq backend, and django. I would like my task to be able to clean up properly if the worker shuts down. As far as I am aware you cannot supply a task destructor so I tried hooking into the worker_shutdown signal.

注意： AbortableTask 仅适用于数据库后端，因此我无法使用它。

Note: AbortableTask only works with the database backend so I cant use that.

from celery.signals import worker_shutdown

@task
def mytask(*args)

  obj = DoStuff()

  def shutdown_hook(*args):
     print "Worker shutting down"
     # cleanup nicely
     obj.stop()

  worker_shutdown.connect(shutdown_hook)

  # blocking call that monitors a network connection
  obj.stuff()

但是，则永远不会调用关闭挂钩。按住Ctrl-C键可以取消该任务，而我必须从外壳中手动将其终止。

However, the shutdown hook never gets called. Ctrl-C'ing the worker doesnt kill the task and I have to manually kill it from the shell.

因此，如果这不是解决问题的正确方法，那么如何允许任务正常关闭？

So if this is not the proper way to go about it, how do I allow tasks to shutdown gracefully?

推荐答案

worker_shutdown 仅由 MainProcess 发送，而不由子池发送工人。
所有 worker _ * 信号，worker_process_init 除外，请参考 MainProcess 。

worker_shutdown is only sent by the MainProcess, not the child pool workers. All worker_* signals except for worker_process_init, refer to the MainProcess.

但是，关闭挂钩永远不会被调用。 Ctrl-C'worker
不会终止任务，我必须从外壳中手动终止它。

However, the shutdown hook never gets called. Ctrl-C'ing the worker doesn't kill the task and I have to manually kill it from the shell.

在正常（热）关机下，工作人员永远不会终止任务。
即使一项任务需要花费数天才能完成，工作人员也无法完成关闭
的任务。您可以将-soft-time-limit 或-time-limit 设置为
来告诉可以终止该任务的实例。

The worker never terminates a task under normal (warm) shutdown. Even if a task takes days to complete, the worker won't complete shutdown until it's completed. You can set --soft-time-limit, or --time-limit to to tell the instance when it's ok to terminate the task.

因此，要添加任何类型的进程清理过程，首先需要对
进行确认，以确保任务能够真正完成。由于清理不会在该事件发生之前调用
。

So to add any kind of process cleanup process you first need to make sure that the tasks can actually complete. As the cleanup wouldn't be called before that happens.

要向池工作进程添加清理步骤，可以使用
之类的方法：

To add a cleanup step to the pool worker processes you can use something like:

from celery import platforms
from celery.signals import worker_process_init

def cleanup_after_tasks(signum, frame):
    # reentrant code here (see http://docs.python.org/library/signal.html)

def install_pool_process_sighandlers(**kwargs):
    platforms.signals["TERM"] = cleanup_after_tasks
    platforms.signals["INT"] = cleanup_after_tasks

worker_process_init.connect(install_pool_process_sighandlers)

这篇关于通知工人芹菜任务关闭的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！