RabbitMQ 消费者过载 [英] RabbitMQ consumer overload

问题描述

我一直在阅读有关 AMQP 消息确认原则的信息.(https://www.rabbitmq.com/confirms.html).非常有用且写得很好的文章，但是关于消费者声明的一个特别的事情确实令人困惑，这里是引述:

<块引用><块引用>

使用自动确认模式时需要考虑的另一件事是消费者过载.

消费者超载?消息队列由代理处理并保存在 RAM 中(如果我理解正确的话).它是关于什么超载的?消费者是否有某种第二个队列?那篇文章的另一部分更令人困惑:

<块引用><块引用>

因此，消费者可能会对交付速度感到不知所措，可能会在内存中累积积压并耗尽堆或让他们的进程被操作系统终止.

什么积压?这一切如何协同工作?消费者完成了哪些工作(当然除了消费消息和处理它)?我认为代理保持队列活动并转发消息，但现在我正在阅读一些神秘的积压和消费者超载.这真的很令人困惑，有人可以解释一下或者至少指出好的来源吗?

解决方案

我相信您所指的文档涉及在我看来，AMQP 0-9-1 或 RabbitMQ 的设计缺陷实现它.

考虑以下场景:

• 一个队列中有数千条消息
• 单个消费者使用 AutoAck=true 订阅队列且未设置预取计数

会发生什么?

RabbitMQ 的实现是将任意数量的消息传递给没有预取计数的客户端.此外，使用 Auto-Ack 时，预取计数无关紧要，因为消息在传递给消费者时得到确认.

内存缓冲区:消费者的默认客户端 API 实现有一个内存缓冲区(在 .NET 中，它是某种类型的阻塞集合(如果我没记错的话).所以，在处理消息之前，但在从代理接收到消息之后，它进入这个内存中的保存区域.现在，设计缺陷就是这个保存区域.消费者别无选择，只能接受来自代理的消息，因为它是异步发布到客户端的.这是与AMQP 协议规范(参见第 53 页).

因此，此时队列中的每条消息都将立即传递给消费者，消费者将被消息淹没.假设每条消息都很小，但需要 5 分钟来处理，那么这个消费者完全有可能在任何其他消费者附加到它之前耗尽整个队列.并且由于 AutoAck 开启，broker 会在发送后立即忘记这些消息.

如果您想处理这些消息，显然这不是一个好的方案，因为它们已经离开了代理的相对安全性，现在位于消费端点的 RAM 中.假设遇到一个导致消费端点崩溃的异常 - 噗，所有消息都消失了.

如何解决这个问题?

您必须关闭 Auto-Ack，通常设置合理的预取计数也是一个好主意(通常 2-3 就足够了).

I`ve been reading about the principles of AMQP messaging confirms. (https://www.rabbitmq.com/confirms.html). Really helpful and wel written article but one particular thing about consumer aknowledgments is really confusing, here is the quote:

Another things that's important to consider when using automatic acknowledgement mode is that of consumer overload.

Consumer overload? Message queue is processed and kept in RAM by broker (if I understand it correctly). What overload is it about? Does consumer have some kind of second queue? Another part of that article is even more confusing:

Consumers therefore can be overwhelmed by the rate of deliveries, potentially accumulating a backlog in memory and running out of heap or getting their process terminated by the OS.

What backlog? How is this all works together? What part of job is done by consumer (besides consuming message and processing it of course)? I thought that broker is keeping queues alive and forwards the messages but now I am reading about some mysterious backlogs and consumer overloads. This is really confusing, can someone explain it a bit or at least point me to the good source?

解决方案

I believe the documentation you're referring to deals with what, in my opinion, is sort of a design flaw in either AMQP 0-9-1 or RabbitMQ's implementation of it.

Consider the following scenario:

• A queue has thousands of messages sitting in it
• A single consumer subscribes to the queue with AutoAck=true and no pre-fetch count set

What is going to happen?

RabbitMQ's implementation is to deliver an arbitrary number of messages to a client who has not pre-fetch count. Further, with Auto-Ack, prefetch count is irrelevant, because messages are acknowledged upon delivery to the consumer.

In-memory buffers: The default client API implementations of the consumer have an in-memory buffer (in .NET it is some type of blocking collection (if I remember correctly). So, before the message is processed, but after the message is received from the broker, it goes into this in-memory holding area. Now, the design flaw is this holding area. A consumer has no choice but to accept the message coming from the broker, as it is published to the client asynchronously. This is a flaw with the AMQP protocol specification (see page 53).

Thus, every message in the queue at that point will be delivered to the consumer immediately and the consumer will be inundated with messages. Assuming each message is small, but takes 5 minutes to process, it is entirely possible that this one consumer will be able to drain the entire queue before any other consumers can attach to it. And since AutoAck is turned on, the broker will forget about these messages immediately after delivery.

Obviously this is not a good scenario if you'd like to get those messages processed, because they've left the relative safety of the broker and are now sitting in RAM at the consuming endpoint. Let's say an exception is encountered that crashes the consuming endpoint - poof, all the messages are gone.

How to work around this?

You must turn Auto-Ack off, and generally it is also a good idea to set reasonable pre-fetch count (usually 2-3 is sufficient).

这篇关于RabbitMQ 消费者过载的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！