如何优雅地重新启动芹菜而不拖延任务 [英] How to restart Celery gracefully without delaying tasks
问题描述
我们使用Celery与我们的Django webapp来管理离线任务;这些任务中的一些可以运行长达120秒。
We use Celery with our Django webapp to manage offline tasks; some of these tasks can run up to 120 seconds.
每当我们进行任何代码修改时，我们需要重新启动Celery以重新加载新的Python代码。我们目前的解决方案是将一个SIGTERM发送到主Celery进程（ kill -s 15`cat / var / run / celeryd.pid` ），然后等待它死亡并重新启动它（ python manage.py celeryd --pidfile = / var / run / celeryd.pid [...] ）。
Whenever we make any code modifications, we need to restart Celery to have it reload the new Python code. Our current solution is to send a SIGTERM to the main Celery process (kill -s 15 `cat /var/run/celeryd.pid`), then to wait for it to die and restart it (python manage.py celeryd --pidfile=/var/run/celeryd.pid [...]).
由于长时间运行的任务，这通常意味着关闭将需要一两分钟，在此期间不会处理任何新任务，从而对当前在该站点上的用户造成明显的延迟。我正在寻找一种告诉Celery关闭的方法，但是随后立即启动一个新的Celery实例来开始运行新的任务。
Because of the long-running tasks, this usually means the shutdown will take a minute or two, during which no new tasks are processed, causing a noticeable delay to users currently on the site. I'm looking for a way to tell Celery to shutdown, but then immediately launch a new Celery instance to start running new tasks.
没有工作：
- 发送SIGHUP到主进程：这导致Celery尝试重新启动然后重新启动自己。这不仅需要很长时间，而且甚至不起作用，因为显然新的进程在旧的死机之前启动，所以新的进程会抱怨
ERROR：Pidfile（/ var / run / celeryd）。 pid）已经存在。看来我们已经在运行？ （PID：13214）并立即死亡。 （这看起来像Celery本身的一个错误;我已让他们知道。 ） - 将SIGTERM发送到主进程，然后立即启动新实例：与Pidfile相同的问题。
- 完全禁用Pidfile：没有我们无法告诉我们，当我们想要进行热关机时，30个Celery进程中的哪一个是需要发送SIGTERM的主要过程。我们也没有可靠的方法检查主流程是否还活着。
- Sending SIGHUP to the main process: this caused Celery to attempt to "restart," by doing a warm shutdown and then relaunching itself. Not only does this take a long time, it doesn't even work, because apparently the new process launches before the old one dies, so the new one complains
ERROR: Pidfile (/var/run/celeryd.pid) already exists. Seems we're already running? (PID: 13214)and dies immediately. (This looks like a bug in Celery itself; I've let them know about it.) - Sending SIGTERM to the main process and then immediately launching a new instance: same issue with the Pidfile.
- Disabling the Pidfile entirely: without it, we have no way of telling which of the 30 Celery process are the main process that needs to be sent a SIGTERM when we want it to do a warm shutdown. We also have no reliable way to check if the main process is still alive.
推荐答案
celeryd有--autoreload选项。如果启用，芹菜工人（主要过程）将检测芹菜模块中的变化并重新启动所有工作进程。与SIGHUP信号相反，当当前执行任务完成时，autoreload将独立重新启动每个进程。这意味着当一个工作进程重新启动时，剩余的进程可以执行任务。
celeryd has --autoreload option. If enabled, celery worker (main process) will detect changes in celery modules and restart all worker processes. In contrast to SIGHUP signal, autoreload restarts each process independently when the current executing task finishes. It means while one worker process is restarting the remaining processes can execute tasks.
http://celery.readthedocs.org/en/latest/userguide/workers.html#autoreloading
这篇关于如何优雅地重新启动芹菜而不拖延任务的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
