Apache Kafka具有多个分区的消息顺序 [英] Apache Kafka order of messages with multiple partitions

问题描述

根据Apache Kafka文档，消息的顺序可以在分区中或一个主题中的一个分区中实现.在这种情况下，我们获得的并行性好处是什么，它等效于传统的MQ，不是吗?

As per Apache Kafka documentation, the order of the messages can be achieved within the partition or one partition in a topic. In this case, what is the parallelism benefit we are getting and it is equivalent to traditional MQs, isn't it?

推荐答案

在Kafka中，并行度等于主题的分区数.

例如，假设您的消息是根据user_id进行分区的，并考虑4条具有user_id 1、2、3和4的消息.假设您有一个带有4个分区的用户"主题.

For example, assume that your messages are partitioned based on user_id and consider 4 messages having user_ids 1,2,3 and 4. Assume that you have an "users" topic with 4 partitions.

由于分区是基于user_id的，因此假设具有user_id 1的消息将进入分区1，具有user_id 2的消息将进入分区2，依此类推.

Since partitioning is based on user_id, assume that message having user_id 1 will go to partition 1, message having user_id 2 will go to partition 2 and so on..

还假设您有4个该主题的使用者.由于您有4个使用者，因此Kafka会将每个使用者分配到一个分区.因此，在这种情况下，一旦推送了4条消息，消费者便会立即使用它们.

Also assume that you have 4 consumers for the topic. Since you have 4 consumers, Kafka will assign each consumer to one partition. So in this case as soon as 4 messages are pushed, they are immediately consumed by the consumers.

如果该主题有2个使用者而不是4个，则每个使用者将处理2个分区，消耗的吞吐量将近一半.

If you had 2 consumers for the topic instead of 4, then each consumer will be handling 2 partitions and the consuming throughput will be almost half.

要完全回答您的问题， Kafka仅按分区中的消息提供总顺序，而不提供主题中不同分区之间的消息的总顺序.

To completely answer your question, Kafka only provides a total order over messages within a partition, not between different partitions in a topic.

即，如果分区2的消耗非常慢而分区4的消耗非常快，则具有user_id 4的消息将在具有user_id 2的消息之前被消耗.这就是Kafka的设计方式.

ie, if consumption is very slow in partition 2 and very fast in partition 4, then message with user_id 4 will be consumed before message with user_id 2. This is how Kafka is designed.

这篇关于Apache Kafka具有多个分区的消息顺序的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！