芹菜生产优雅重启 [英] Celery Production Graceful Restart

问题描述

我需要重新启动celery守护程序，但我需要它告诉当前的工作人员在任务完成时关闭，然后在新的工作人员仍在关闭的同时启动新的工作组.

I need to restart the celery daemon but I need it to tell the current workers to shutdown as their tasks complete and then spin up a new set of workers while the old ones are still shutting down.

守护程序上当前的优美选项在重新启动之前会等待所有任务完成，这在长时间运行的作业中没有用.

The current graceful option on the daemon waits for all tasks to complete before restarting which is not useful when you have long running jobs.

请不要建议自动重新加载，因为它在4.0.2.中目前没有记录.

Please do not suggest autoreload as it is currently undocumented in 4.0.2.

推荐答案

好吧，我最终要做的是使用有监督能力的人，并且能够对此进行管理.

Alright well what I ended up doing was using supervisord and ansible to manage this.

[program:celery_worker]
# Max concurrent task you wish to run.
numprocs=5
process_name=%(program_name)s-%(process_num)s
directory=/opt/worker/main
# Ignore this unless you want to use virtualenvs.
environment=PATH="/opt/worker/main/bin:%(ENV_PATH)s"
command=/opt/worker/bin/celery worker -n worker%(process_num)s.%%h --app=python --time-limit=3600 -c 5 -Ofair -l debug --config=celery_config -E
stdout_logfile=/var/log/celery/%(program_name)s-%(process_num)s.log
user=worker_user
autostart=true
autorestart=true
startretries=99999
startsecs=10
stopsignal=TERM
stopwaitsecs=7200
killasgroup=false

您可以使用主管来停止/启动工作程序以加载新代码，但是它将等待所有工作人员停止后再重新启动它们，这对于长时间运行的工作而言效果不佳.最好只是终止MainProcesses，它会告诉工人停止接受作业并在作业完成时关闭.

You can use supervisor to stop/start the workers to load new code but it will wait for all of them to stop before starting them again, this does not work well for long running jobs. Better just to TERM the MainProcesses which will tell the workers to stop accepting jobs and shutdown as they finish.

ps aux | grep *celery.*MainProcess | awk '{print $2}' | xargs kill -TERM

主管将在其死亡时重新启动.

Supervisor will restart them as they die.

当然，在不完全停止所有工作程序的情况下更新依赖关系是不可能的，这为使用docker之类的东西提供了一个很好的案例.;)

Of course updating dependencies without totally stopping all the workers is pretty impossible which makes a really good case for using something like docker. ;)

这篇关于芹菜生产优雅重启的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！