RabbitMQ消费者超载 [英] RabbitMQ consumer overload
问题描述
我一直在阅读有关AMQP消息传递确认的原理. ( https://www.rabbitmq.com/confirms.html ).确实很有帮助，而且写得很好，但是关于消费者致谢的一件事确实令人困惑，这是引号:
I`ve been reading about the principles of AMQP messaging confirms. (https://www.rabbitmq.com/confirms.html). Really helpful and wel written article but one particular thing about consumer aknowledgments is really confusing, here is the quote:
使用自动确认模式时需要考虑的另一件事是消费者超载.
消费者超负荷?消息队列由代理处理并保存在RAM中(如果我正确理解的话).这是什么过载?消费者是否有某种第二队列? 该文章的另一部分甚至更加令人困惑:
Consumer overload? Message queue is processed and kept in RAM by broker (if I understand it correctly). What overload is it about? Does consumer have some kind of second queue? Another part of that article is even more confusing:
因此，交付速度可能会淹没消费者，可能在内存中积压积压，耗尽内存或使进程被操作系统终止.
Consumers therefore can be overwhelmed by the rate of deliveries, potentially accumulating a backlog in memory and running out of heap or getting their process terminated by the OS.
什么积压?所有这些如何一起工作?消费者完成工作的哪一部分(当然除了消费消息并对其进行处理之外)?我以为经纪人可以保持队列活动并转发消息，但是现在我正在阅读一些神秘的积压和消费者超载信息.这真的很令人困惑，有人可以解释一下，或者至少可以指出我的好消息吗?
What backlog? How is this all works together? What part of job is done by consumer (besides consuming message and processing it of course)? I thought that broker is keeping queues alive and forwards the messages but now I am reading about some mysterious backlogs and consumer overloads. This is really confusing, can someone explain it a bit or at least point me to the good source?
推荐答案
我认为您所指的文档处理的是我认为是AMQP 0-9-1或RabbitMQ的某种设计缺陷.实施.
I believe the documentation you're referring to deals with what, in my opinion, is sort of a design flaw in either AMQP 0-9-1 or RabbitMQ's implementation of it.
请考虑以下情形:
- 队列中有成千上万条消息
- 单个使用者使用
AutoAck=true订阅队列，并且未设置预提取计数
- A queue has thousands of messages sitting in it
- A single consumer subscribes to the queue with
AutoAck=trueand no pre-fetch count set
会发生什么?
RabbitMQ的实现是将任意数量的消息传递给没有预取计数的客户端.此外，使用自动确认时，预取计数是无关紧要的，因为消息在传递给消费者时就得到了确认.
RabbitMQ's implementation is to deliver an arbitrary number of messages to a client who has not pre-fetch count. Further, with Auto-Ack, prefetch count is irrelevant, because messages are acknowledged upon delivery to the consumer.
内存缓冲区: 使用者的默认客户端API实现具有内存缓冲区(在.NET中，它是某种类型的阻塞收集(如果我没记错的话).因此，在处理消息之前，但是在从代理接收到消息之后，它进入了内存中的存放区，现在设计缺陷就是这个存放区，消费者别无选择，只能接受来自代理的消息，因为消息是异步发布到客户端的. AMQP协议规范(请参阅第53页)
In-memory buffers: The default client API implementations of the consumer have an in-memory buffer (in .NET it is some type of blocking collection (if I remember correctly). So, before the message is processed, but after the message is received from the broker, it goes into this in-memory holding area. Now, the design flaw is this holding area. A consumer has no choice but to accept the message coming from the broker, as it is published to the client asynchronously. This is a flaw with the AMQP protocol specification (see page 53).
因此，此时队列中的每个消息都将立即传递给使用者，并且使用者将被消息淹没.假设每条消息很小，但是要花费5分钟的时间，那么这个使用者完全有可能在其他使用者无法附加到该队列之前耗尽整个队列.由于AutoAck已打开，代理将在传递后立即忘记这些消息.
Thus, every message in the queue at that point will be delivered to the consumer immediately and the consumer will be inundated with messages. Assuming each message is small, but takes 5 minutes to process, it is entirely possible that this one consumer will be able to drain the entire queue before any other consumers can attach to it. And since AutoAck is turned on, the broker will forget about these messages immediately after delivery.
如果要处理这些消息，显然，这不是一个好方案，因为它们已经离开了代理的相对安全性，并且现在位于使用方端点的RAM中.假设遇到一个使使用方端点崩溃的异常-poof，所有消息都消失了.
Obviously this is not a good scenario if you'd like to get those messages processed, because they've left the relative safety of the broker and are now sitting in RAM at the consuming endpoint. Let's say an exception is encountered that crashes the consuming endpoint - poof, all the messages are gone.
如何解决此问题?
您必须关闭自动确认"，通常，设置合理的预取计数也是个好主意(通常2-3个就足够了.)
You must turn Auto-Ack off, and generally it is also a good idea to set reasonable pre-fetch count (usually 2-3 is sufficient).
这篇关于RabbitMQ消费者超载的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
