区分芹菜，kombu，PyAMQP和RabbitMQ / ironMQ [英] Differentiate celery, kombu, PyAMQP and RabbitMQ/ironMQ

问题描述

我想将图像上传到S3服务器，但在上传之前，我想生成3种不同大小的缩略图，并且我希望它能够在请求/响应周期内完成，因此我使用了芹菜。我已阅读文档，这是我的理解。如果我错了，请纠正。

1. Celery可以帮助您管理请求响应周期外的任务队列。 b
2. 然后有一个名为carrot / kombu的东西 - 它是一个django中间件，它包含通过芹菜创建的任务。
然后，第三层PyAMQP便于胡萝卜与经纪人。例如。 RabbitMQ，AmazonSQS，ironMQ等。

3. 经纪人坐在不同的服务器上为您做任何事情。
   

现在我的理解是 - 如果多个用户同时上传图片，芹菜会对调整大小进行排队，而调整大小实际上会发生在ironMQ服务器上，因为它提供了一个酷炫的heroku插件。

现在怀疑：

1. 但是在图片大小调整后，ironMQ会推送它到S3服务器，或者它会通知一旦进程完成..我不清楚它。

芹菜和kombu /胡萝卜，你能否生动地解释一下。 IronMQ不会为您处理您的任务;它只是作为Celery的后台，用于跟踪需要执行的工作。
所以，以下是发生的情况。假设您有两台服务器，您的Web服务器和Celery服务器。您的Web服务器负责处理请求，Celery服务器创建缩略图并将其上传到S3。以下是典型的请求：




1. 您的用户将图片上传到您的网络服务器。


2. 你在某处存储了这个图像 - 我个人建议将它放在S3上，但是你也可以将它存储在其中，例如 IronCache ，base64编码。关键是将其放置在您的Celery服务器可以访问它的地方。


3. 您在Celery上排队工作，将图像位置传递给您的Celery服务器。


4. 您的Celery服务器下载映像，生成缩略图并上传到S3。然后，它将S3 URL存储在作业结果中。 您的web服务器等待作业结束，然后才能访问结果。或者，您可以让您的Celery服务器将结果存储在数据库本身中。重点在于Celery服务器完成繁重的工作（生成缩略图），并且不会阻止请求循环。



我写了一个在Heroku上使用IronMQ的例子。你可以在这里看到： http://iron-celery-demo.herokuapp.com 。您可以在Github上查看示例 和阅读教程，该教程非常透彻，如何在Heroku上部署Celery。





清除AMQP的内容：





• IronMQ是由Iron.io开发的基于云的消息队列服务。


• AMQP是一个开放的消息传递规范
  
• RabbitMQ是最流行的AMQP规范实现（我知道）。
  
• PyAMQP是一个Python库，它允许Python客户端与任何AMQP实现进行通信，包括RabbitMQ






IronMQ和RabbitMQ / AMQP最大的区别之一是IronMQ是托管和管理的，所以您不必自己托管服务器，并担心正常运行时间。该规范提供了更多的分化方面，并且存在根本性差异，但是Celery将大部分这些差异抽象出来。因为您使用的是Celery，所以您可能会注意到的唯一区别是IronMQ已托管，因此您不必站出来管理自己的服务器。





完全披露：我受IronMio公司Iron.io雇用。




I want to upload images to S3 server, but before uploading I want to generate thumbnails of 3 different sizes, and I want it to be done out of request/response cycle hence I am using celery. I have read the docs, here is what I have understood. Please correct me if I am wrong.

1. Celery helps you manage your task queues outside the request response cycle.
2. Then there is something called carrot/kombu - its a django middleware that packages tasks that get created via celery.
3. Then the third layer PyAMQP that facilitates the communication of carrot to a broker. eg. RabbitMQ, AmazonSQS, ironMQ etc.
4. Broker sits on a different server and does stuff for you.

Now my understanding is - if multiple users upload image at the same time, celery will queue the resizing, and the resizing will actually happen at the ironMQ server, since it offers a cool addon on heroku.

Now the doubts:

1. But what after the image is resized, will ironMQ push it to the S3 server, or will it notify once the process is completed.. i am not clear about it.

2. What is the difference between celery and kombu/carrot, could you explain vividly.

解决方案

IronMQ does not process your tasks for you; it simply serves as the backend for Celery to keep track of what jobs need to be performed.

So, here's what happens. Assume you have two servers, your web server and your Celery server. Your web server is responsible for handling requests, your Celery server creates the thumbnails and uploads them to S3. Here's what a typical request looks like:

1. Your user uploads the image to your web server.
2. You store that image somewhere--I'd recommend putting it on S3 right then, personally, but you could also store it in, for example IronCache, base64-encoded. The point is to put it somewhere your Celery server can access it.
3. You queue up a job on Celery, passing the location of the image to your Celery server.
4. Your Celery server downloads the image, generates your thumbnails, and uploads them to S3. It then stores the S3 URLs in the job results.
5. Your web server waits until the job finishes, then has access to the results. Alternatively, you could have your Celery server store the results in the database itself. The point is that the Celery server does the heavy lifting (generating the thumbnails) and does not hold up the request loop while it does.

I wrote an example for using IronMQ on Heroku. You can see it here: http://iron-celery-demo.herokuapp.com. You can see the source for the example on Github and read the tutorial, which explains pretty thoroughly and step-by-step how to deploy Celery on Heroku.

To clear up the AMQP stuff:

• IronMQ is a cloud-based message queue service developed by Iron.io.
• AMQP is an open messaging specification
• RabbitMQ is the most popular implementation (that I know of) of the AMQP specification.
• PyAMQP is a Python library that lets Python clients communicate with any implementation of AMQP, including RabbitMQ

One of the biggest differences between IronMQ and RabbitMQ/AMQP is that IronMQ is hosted and managed, so you don't have to host the server yourself and worry about uptime. The spec offers a bunch more in terms of differentiation, and there are underlying differences, but Celery abstracts most of those away. Because you're using Celery, the only difference you're liable to notice is that IronMQ is hosted, so you don't have to stand up and manage your own server.

Full disclosure: I am employed by Iron.io, the company behind IronMQ.

这篇关于区分芹菜，kombu，PyAMQP和RabbitMQ / ironMQ的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！