当我们在运行时增加分区时，Kafka如何保证消息顺序? [英] How Kafka guarantee the messages order while we increase the partitions in runtime?

问题描述

我是 kafka 的新手，当我阅读 Kafka 文档时，我意识到使用相同键提供的消息将映射到相同的分区以保证顺序.这完全有道理.但是，我想知道如果我们在运行时增加主题分区的数量，具有相同键的新消息是否会像以前一样散列到同一个分区(旧的)?

I am new to kafka and when I read the Kafka doc, I realize that messages provided with the same key will be mapped to the same partition to guarantee the order. This totally makes sense. However, I'd like to know if we increase the number of topic partitions in runtime, will the new messages with the same key be hashed to the same partition (old one) as before?

如果是这样，如果所有消息都提供了密钥，那么它们都不会映射到新分区怎么办?这对我来说没有意义.

If so, what if all messages are provided with keys, then none of them will be mapped to new partition? This doesn't make sense to me.

如果不是，那么Kafka如何保证相同key的消息的顺序?

If not, then how Kafka guarantee the order to messages with the same key in order?

推荐答案

我想知道如果我们在运行时增加主题分区的数量，具有相同键的新消息是否会像以前一样散列到同一个分区(旧的)?

I'd like to know if we increase the number of topic partitions in runtime, will the new messages with the same key be hashed to the same partition (old one) as before?

否，新消息将根据新的分区数进行分区.旧邮件不会重新分区.

No, the new messages will be partitioned based on the new number of partitions. Old messages will not get re-partitioned.

如果不是，那么Kafka如何保证相同key的消息的顺序?

If not, then how Kafka guarantee the order to messages with the same key in order?

更改分区数时没有任何保证.在运行时增加(或减少)分区数量时，排序可以或更好地发生变化.

There are no guarantees when changing the number of partitions. When increasing (or decreasing) the number of partitions at runtime the ordering can, or better, will change.

因此，如果您依赖数据的排序，则不建议更改已包含数据的主题的分区.你能做的是

Changing a partition of a topic that already includes data is therefore not recommended if you are relying on ordering of your data. What you could do is

• 在您的生产者向其发送数据之前，将现有数据复制到具有新分区数的新主题中.
• 在您的生产者中应用自定义分区器，将基于键的消息发送到固定分区中，并且只有新键"；发送到新分区.

这篇关于当我们在运行时增加分区时，Kafka如何保证消息顺序?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！