RabbitMQ排队的消息不断增加 [英] RabbitMQ Queued messages keep increasing

问题描述

我们有一个基于Windows的Celery/RabbitMQ服务器，该服务器为我们的Web应用程序执行进程外的长时间运行的python任务.
例如，这样做是获取一个CSV文件并处理每一行.对于每一行，它都会在我们的数据库中预订一个或多个记录.

We have a Windows based Celery/RabbitMQ server that executes long-running python tasks out-of-process for our web application.
What this does, for example, is take a CSV file and process each line. For every line it books one or more records in our database.

这似乎工作正常，我可以看到工作进程正在预订的记录.但是，当我使用管理插件(基于Web的管理工具)检查RabbitMQ服务器时，我看到排队的消息增加了，并且没有下降.

This seems to work fine, I can see the records being booked by the worker processes. However, when I check the rabbitMQ server with the management plugin (the web based management tool) I see the Queued messages increasing, and not coming back down.

在连接下，我看到116个连接，每个虚拟主机大约10-15个，都在运行"，但是当我单击鼠标时，大多数连接都以"idle"作为状态.我也想知道为什么这些连接仍然打开，以及是否需要更改使其自身关闭:

Under connections I see 116 connections, about 10-15 per virtual host, all "running" but when I click through, most of them have 'idle' as State. I'm also wondering why these connections are still open, and if there is something I need to change to make them close themselves:

在队列"下，我可以看到6200多个状态为闲置"且没有减少的项目.

Under 'Queues' I can see more than 6200 items with state 'idle', and not decreasing.

因此，我要具体询问这些是否是正常的统计信息，还是我应该担心队列增加但又不会下降，以及似乎没有关闭的持久连接...

So concretely I'm asking if these are normal statistics or if I should worry about the Queues increasing but not coming back down and the persistent connections that don't seem to close...

除了管理工具内部的简洁帮助之外，我似乎找不到任何有关这些统计数据含义以及它们好坏的信息.

Other than the rather concise help inside the management tool, I can't seem to find any information about what these stats mean and if they are good or bad.

我还想知道为什么消息仍在队列中可见，为什么不删除它们，因为任务似乎还不能很好地完成.

I'd also like to know why the messages are still visible in the queues, and why they are not removed, as the tasks seem t be completed just fine.

感谢您的帮助.

推荐答案

回答我自己的问题；

Celery将针对调用代码中的每个任务发送一条结果消息.该消息通过相同的AMPQ队列发送回去.这就是为什么任务在起作用，但是队列不断填​​满的原因.我们没有处理这些结果，甚至没有对它们感兴趣.

Celery sends a result message back for every task in the calling code. This message is sent back via the same AMPQ queue. This is why the tasks were working, but the queue kept filling up. We were not handling these results, or even interested in them.

我在celery任务中添加了 ignore_result = True ，因此该任务 not 不会将结果消息发送回队列.这是解决该问题的主要方法.

I added ignore_result=True to the celery task, so the task does not send result messages back into the queue. This was the main solution to the problem.

此外，添加了配置选项CELERY_SEND_EVENTS = False以加快芹菜的速度.如果设置为TRUE，则此选项将Celery发送事件用于外部监视工具.

Furthermore, the configuration option CELERY_SEND_EVENTS=False was added to speed up celery. If set to TRUE, this option has Celery send events for external monitoring tools.

在CELERY_TASK_RESULT_EXPIRES = 3600之上，现在可以确保即使将结果发送回去，如果未领取/确认，它们也会在一小时后过期.

On top of that CELERY_TASK_RESULT_EXPIRES=3600 now makes sure that even if results are sent back, that they expire after one hour if not picked up/acknowledged.

最后将CELERY_RESULT_PERSISTENT设置为False，这会将芹菜配置为不将这些结果消息存储在磁盘上.当服务器崩溃时，它们将消失，这对我们来说很好，因为我们不使用它们.

Finally CELERY_RESULT_PERSISTENT was set to False, this configures celery to not store these result messages on disk. They will vanish when the server crashes, which is fine in our case, as we don't use them.

简而言之；如果您在应用中不需要有关任务是否完成以及何时完成的反馈，请对celery任务使用 ignore_result = True ，这样就不会发回任何消息.如果您需要该信息，请确保您选择并处理结果，以使队列停止填充.

So in short; if you don't need feedback in your app about if and when the tasks are finished, use ignore_result=True on the celery task, so that no messages are sent back. If you do need that information, make sure you pick up and handle the results, so that the queue stops filling up.

这篇关于RabbitMQ排队的消息不断增加的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！