EC2 上的 RabbitMQ 消耗大量 CPU [英] RabbitMQ on EC2 Consuming Tons of CPU

问题描述

我正在尝试让 RabbitMQ 与 Celery 和 Django 在 EC2 实例上进行一些非常基本的后台处理.我在一个大型 EC2 实例上运行 rabbitmq-server 2.5.0.

我按照

<块引用>

旧的结果不会被自动清理，所以你必须确保消费结果，否则队列的数量最终会失控.如果您正在运行 RabbitMQ 2.1.1 或更高版本，您可以利用队列的 x-expires 参数，这将在队列未使用后的特定时间限制后过期.队列过期时间可以通过 CELERY_AMQP_TASK_RESULT_EXPIRES 设置(默认不启用)来设置(以秒为单位).

I am trying to get RabbitMQ with Celery and Django going on an EC2 instance to do some pretty basic background processing. I'm running rabbitmq-server 2.5.0 on a large EC2 instance.

I downloaded and installed the test client per the instructions here (at the very bottom of the page). I have been just letting the test script go and am getting the expected output:

recving rate: 2350 msg/s, min/avg/max latency: 588078478/588352905/588588968 microseconds
recving rate: 1844 msg/s, min/avg/max latency: 588589350/588845737/589195341 microseconds
recving rate: 1562 msg/s, min/avg/max latency: 589182735/589571192/589959071 microseconds
recving rate: 2080 msg/s, min/avg/max latency: 589959557/590284302/590679611 microseconds

The problem is that it is consuming an incredible amount of CPU:

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
668 rabbitmq 20 0 618m 506m 2340 S 166 6.8 2:31.53 beam.smp
1301 ubuntu 20 0 2142m 90m 9128 S 17 1.2 0:24.75 java

I was testing on a micro instance earlier and it was completely consuming all resources on the instance.

Is this to be expected? Am I doing something wrong?

Thanks.

Edit:

The real reason for this post was that celerybeat seemed to run okay for awhile and then suddenly consume all resources on the system. I installed the rabbitmq management tools and have been investigating how the queues are created from celery and from the rabbitmq test suite. It seems to me that celery is orphaning these queues and they are not going away.

Here is the queue as generated by the test suite. One queue is created and all the messages go into it and come out:

Celerybeat creates a new queue for every time it runs the task:

It sets the auto-delete parameter to true, but I'm not entirely sure when these queues will get deleted. They seem to just slowly build up and eat resources.

Does anyone have an idea?

Thanks.

解决方案

Ok, I figured it out.

Here's the relevant piece of documentation: http://readthedocs.org/docs/celery/latest/userguide/tasks.html#amqp-result-backend

Old results will not be cleaned automatically, so you must make sure to consume the results or else the number of queues will eventually go out of control. If you’re running RabbitMQ 2.1.1 or higher you can take advantage of the x-expires argument to queues, which will expire queues after a certain time limit after they are unused. The queue expiry can be set (in seconds) by the CELERY_AMQP_TASK_RESULT_EXPIRES setting (not enabled by default).

这篇关于EC2 上的 RabbitMQ 消耗大量 CPU的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！