与芹菜正在进行的任务互动 [英] Interact with celery ongoing task

问题描述

我们有一个基于 rabbitMQ 和 Celery 的分布式体系结构.我们可以并行执行多个任务，而不会出现任何问题.可伸缩性很好.

We have a distributed architecture based on rabbitMQ and Celery. We can launch in parallel multiple tasks without any issue. The scalability is good.

现在，我们需要远程控制任务:暂停"，恢复"，取消".我们发现的唯一解决方案是在Celery任务中进行对另一个任务的RPC调用，该任务在数据库请求后会回复该命令.Celery任务和RPC任务不在同一台计算机上，只有RPC任务有权访问数据库.

Now we need to control the task remotely: PAUSE, RESUME, CANCEL. The only solution we found is to make in the Celery task a RPC call to another task that replies the command after a DB request. The Celery task and RPC task are not on the same machine and only the RPC task has access to the DB.

您对如何改进它以及如何轻松地与正在进行的任务进行沟通有任何建议?谢谢

Do you have any advice how to improve it and easily communicate with an ongoing task? Thank you

实际上，我们想做如下图所示的操作.进行 Blue 或 Orange 的配置很容易，但是我们不知道如何同时进行这两种操作.工作人员正在订阅一个常见的 Jobs队列，每个工作人员都在交换中声明了自己的 Admin队列.

In fact we would like to do something like in the picture below. It's easy to do the Blue configuration or the Orange, but we don't know how to do both simultaneously. Workers are subscribing to a common Jobs queue and each worker has its own Admin queue declared on an exchange.

如果使用 Celery 无法做到这一点，那么我愿意使用 python-rq 之类的其他框架作为解决方案.

IF this is not possible with Celery, I'am open to a solution with other frameworks like python-rq.

推荐答案

它看起来像 控制总线模式 .

It look like the Control Bus pattern.

为了获得更好的可伸缩性并减少RPC调用，我建议反转逻辑.状态更改发生时， PAUSE，RESUME，CANCEL 命令通过控制总线被推送到Celery任务.Celery应用程序会将Celery应用程序的当前状态存储在商店中(可能在文件系统上的内存中.).如果即使在应用程序停止/启动后仍必须保持任务状态，则为了使两个应用程序保持同步(例如，启动时的同步)，将需要进行更多工作.

For a better scalability and in order to reduce the RPC call, I recommend to reverse the logic. The PAUSE, RESUME, CANCEL command are push to the Celery tasks through a control bus when the state change occurs. The Celery app will store the current state of the Celery app in a store (could be in memory, on the filesystem..). If task states must be kept even after a stop/start of the app, It will involve more work in order to keep both app synchronized (eg. synchronization at startup).

这篇关于与芹菜正在进行的任务互动的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！