RabbitMQ 中的密钥感知消费者 [英] Key-aware consumers in RabbitMQ

问题描述

让我们考虑一个系统，其中将数千个客户端数据发布到 RabbitMQ 交换(在此阶段 client_id 已知).Exchange 将它们路由到单个队列.最后，消息由单个应用程序使用.效果很好.

Let's consider a system where thousands of clients data is published to a RabbitMQ exchange (client_id is known at this stage). Exchange routes them to a single queue. Finally, messages are consumed by a single application. Works great.

然而，随着时间的推移，消费应用程序成为瓶颈，需要水平扩展.问题是系统要求考虑特定客户端的消息由应用程序的同一实例使用.

However, over time, the consuming application becomes a bottleneck and needs to be scaled horizontally. The problem is the system requires that messages considering particular client are consumed by the same instance of the application.

我可以创建很多队列:每个客户端一个，或者使用主题交换并根据某些 client_id 前缀路由它.尽管如此，我仍然没有看到一种优雅的方式来设计消费者应用程序以使其可以水平扩展(因为它需要声明它明确使用的队列).

I can create lots of queues: either one per client or use a topic exchange and route it based on some client_id prefix. Still, I don't see an elegant way how to design the consumer application so that it can be scaled horizontally (as it requires stating queues that it consumes explicitly).

我正在寻找 RabbitMQ 方法来解决这个问题.

I'm looking for RabbitMQ way for solving this problem.

推荐答案

RabbitMQ 有 <代码>x-consistent-hash 和 x-modulus-hash 可以用来解决问题的交流.当使用这些交换时，消息会根据路由键的哈希值被划分到不同的队列中.当然，x-consistent-hash 和 x-modulus-hash 之间在分区的实现方式上存在差异，但主要思想保持不变 - 带有相同的路由键(client_id)将被分发到同一个队列，最终应该被同一个应用程序使用.

RabbitMQ has x-consistent-hash and x-modulus-hash exchanges that can be used to solve the problem. When these exchanges are used, messages get partitioned to different queues according to hash values of routing keys. Of course, there are differences between x-consistent-hash and x-modulus-hash in the way how partitioning is implemented, but main idea stays the same - messages with the same routing key (client_id) will be distributed to the same queue and eventually should be consumed by the same application.

例如，系统可以具有以下拓扑结构:每个应用程序都可以定义一个独占队列(仅由一个连接使用，该连接关闭时队列将被删除)绑定到交换(x-consistent-hash 或 x-modulus-hash).

For example, the system can have the following topology: every application can define an exclusive queue (used by only one connection and the queue will be deleted when that connection closes) that is binded to the exchange (x-consistent-hash or x-modulus-hash).

在我看来，在这种特定场景中使用分布式缓存层是个好主意，但 RabbitMQ 提供了插件来解决此类问题.

In my opinion, it is a good idea to have a distributed cache layer in this particular scenario, but RabbitMQ provides the plugins to tackle this kind of problems.

这篇关于RabbitMQ 中的密钥感知消费者的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！