使用Celery vs.RQ的利弊 [英] Pros and cons to use Celery vs. RQ

问题描述

目前，我正在从事python项目，该项目需要实施一些后台作业（主要用于电子邮件发送和大量数据库更新）。我将Redis用于任务代理。因此，在这一点上，我有两个候选人： Celery 和 RQ 。我对这些工作队列有一定的经验，但我想请大家分享使用此工具的经验。因此。

1. 使用Celery vs. RQ有什么优缺点。


2. 任何项目示例/ task适合使用Celery vs. RQ。

Celery看起来很复杂，但是它是功能齐全的解决方案。实际上，我认为我不需要所有这些功能。从另一面看，RQ非常简单（例如，配置，集成），但是它似乎缺少一些有用的功能（例如，任务吊销，代码自动重载）

解决方案

这是我在尝试回答这个完全相同的问题时所发现的。它可能不全面，甚至在某些方面可能不准确。

简而言之，RQ旨在使整个过程变得更加简单。芹菜被设计为更坚固。他们俩都很棒。

• 文档。 RQ的文档全面而又不复杂，并且反映了该项目的整体简单性-您永远不会感到迷茫或困惑。 Celery的文档也很全面，但希望能第一次设置时，要进行很多次访问，因为内部化的选项太多了



• 监视。 芹菜的花和 RQ仪表板都很容易设置，并且至少为您提供了90％的所有信息




• 经纪人支持。芹菜无疑是赢家，RQ仅支持Redis。这意味着有关什么是经纪人的文档减少了，但也意味着如果Redis不再为您服务，则您将来将无法切换经纪人。例如， Instagram认为这两个Redis和RabbitMQ和Celery 。这很重要，因为不同的经纪人有不同的担保，例如Redis 无法（截至撰写时）保证100％传递了您的消息。




• 优先级队列。 RQ优先级队列模型既简单又有效-工作人员按顺序从队列中读取。芹菜要求纺纱多个工人从不同的队列消费。两种方法都可以使用




• OS支持。 Celery显然是赢家，因为RQ仅在支持 fork 的系统上运行。 Unix系统




• 语言支持。 RQ仅支持Python，而Celery允许您将任务从一种语言发送到另一种语言




• API。 Celery非常灵活（多个结果后端，漂亮的配置格式，工作流画布支持），但是自然地，这种功能可能会令人困惑。相比之下，RQ API很简单。




• 子任务支持。 Celery支持子任务（例如，从现有任务中创建新任务）。我不知道RQ是否可以




• 社区和稳定性。芹菜可能更成熟，但它们都是活跃的项目。截至撰写本文时，Celery在Github上拥有约3500颗星，而RQ上则有约2000颗星，并且两个项目都显示出活跃的发展

我认为，芹菜并没有它的声誉那么复杂，您可能会相信，但是您将不得不使用RTFM。

那么，为什么有人愿意将（可能功能更全的）芹菜换成RQ？在我看来，这全都归结为简单性。通过将自身限制为Redis + Unix，RQ提供了更简单的文档，更简单的代码库和更简单的API。这意味着您（以及项目的潜在贡献者）可以专注于您关心的代码，而不必在工作内存中保留有关任务队列系统的详细信息。我们每个人一次只能拥有多少个细节是有限制的，通过消除将任务队列细节保留在那的需求，RQ让我们回到您关心的代码。这种简单性是以语言间任务队列，广泛的OS支持，100％可靠的消息保证以及轻松切换消息代理的功能为代价的。

Currently I'm working on python project that requires implement some background jobs (mostly for email sending and heavily database updates). I use Redis for task broker. So in this point I have two candidates: Celery and RQ. I had some experience with these job queues, but I want to ask you guys to share you experience of using this tools. So.

1. What pros and cons to use Celery vs. RQ.
2. Any examples of projects/task suitable to use Celery vs. RQ.

Celery looks pretty complicated but it's full featured solution. Actually I don't think that I need all these features. From other side RQ is very simple (e.g configuration, integration), but it seems that it lacks some useful features (e.g task revoking, code auto-reloading)

解决方案

Here is what I have found while trying to answer this exact same question. It's probably not comprehensive, and may even be inaccurate on some points.

In short, RQ is designed to be simpler all around. Celery is designed to be more robust. They are both excellent.

• Documentation. RQ's documentation is comprehensive without being complex, and mirrors the project's overall simplicity - you never feel lost or confused. Celery's documentation is also comprehensive, but expect to be re-visiting it quite a lot when you're first setting things up as there are too many options to internalize

• Monitoring. Celery's Flower and the RQ dashboard are both very simple to setup and give you at least 90% of all information you would ever want

• Broker support. Celery is the clear winner, RQ only supports Redis. This means less documentation on "what is a broker", but also means you cannot switch brokers in the future if Redis no longer works for you. For example, Instagram considered both Redis and RabbitMQ with Celery. This is important because different brokers have different guarantees e.g. Redis cannot (as of writing) guarantee 100% that your messages are delivered.

• Priority queues. RQs priority queue model is simple and effective - workers read from queues in order. Celery requires spinning up multiple workers to consume from different queues. Both approaches work

• OS Support. Celery is the clear winner here, as RQ only runs on systems that support fork e.g. Unix systems

• Language support. RQ only supports Python, whereas Celery lets you send tasks from one language to a different language

• API. Celery is extremely flexible (multiple result backends, nice config format, workflow canvas support) but naturally this power can be confusing. By contrast, the RQ api is simple.

• Subtask support. Celery supports subtasks (e.g. creating new tasks from within existing tasks). I don't know if RQ does

• Community and Stability. Celery is probably more established, but they are both active projects. As of writing, Celery has ~3500 stars on Github while RQ has ~2000 and both projects show active development

In my opinion, Celery is not as complex as its reputation might lead you to believe, but you will have to RTFM.

So, why would anyone be willing to trade the (arguably more full-featured) Celery for RQ? In my mind, it all comes down to the simplicity. By restricting itself to Redis+Unix, RQ provides simpler documentation, simpler codebase, and a simpler API. This means you (and potential contributors to your project) can focus on the code you care about, instead of having to keep details about the task queue system in your working memory. We all have a limit on how many details can be in our head at once, and by removing the need to keep task queue details in there RQ lets get back to the code you care about. That simplicity comes at the expense of features like inter-language task queues, wide OS support, 100% reliable message guarantees, and ability to switch message brokers easily.

这篇关于使用Celery vs.RQ的利弊的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！