当我们在运行时增加分区时，Kafka如何保证消息顺序? [英] How Kafka guarantee the messages order while we increase the partitions in runtime?

问题描述

我是kafka的新手，当我阅读Kafka文档时，我意识到带有相同密钥的消息将被映射到同一分区以保证顺序.这完全有道理.但是，我想知道是否在运行时增加了主题分区的数量，具有相同键的新消息是否会像以前一样被哈希到相同的分区(旧的)?

I am new to kafka and when I read the Kafka doc, I realize that messages provided with the same key will be mapped to the same partition to guarantee the order. This totally makes sense. However, I'd like to know if we increase the number of topic partitions in runtime, will the new messages with the same key be hashed to the same partition (old one) as before?

如果是这样，如果所有消息都提供了密钥怎么办，那么它们中的任何一个都不会映射到新分区?这对我来说没有意义.

If so, what if all messages are provided with keys, then none of them will be mapped to new partition? This doesn't make sense to me.

如果不是，那么Kafka如何保证按顺序发送具有相同密钥的邮件的顺序?

If not, then how Kafka guarantee the order to messages with the same key in order?

推荐答案

我想知道是否在运行时增加了主题分区的数量，是否将具有相同密钥的新消息像以前一样哈希到相同的分区(旧分区)上?

I'd like to know if we increase the number of topic partitions in runtime, will the new messages with the same key be hashed to the same partition (old one) as before?

否，新消息将根据新的分区数进行分区.旧邮件不会重新分区.

No, the new messages will be partitioned based on the new number of partitions. Old messages will not get re-partitioned.

如果不是，那么Kafka如何保证按顺序发送具有相同密钥的邮件的顺序?

If not, then how Kafka guarantee the order to messages with the same key in order?

更改分区数时无法保证.如果在运行时增加(或减少)分区数量，则顺序可能会改变，甚至会更好.

There are no guarantees when changing the number of partitions. When increasing (or decreasing) the number of partitions at runtime the ordering can, or better, will change.

因此，如果您依赖于数据的顺序，则不建议更改已经包含数据的主题的分区.你能做的就是

Changing a partition of a topic that already includes data is therefore not recommended if you are relying on ordering of your data. What you could do is

• 在将生产者向其发送数据之前，将现有数据复制到具有新分区数量的新主题中.
• 在您的生产者中应用自定义分区程序，该程序将基于密钥的消息发送到固定分区中，并且仅具有新密钥".发送到新分区.

这篇关于当我们在运行时增加分区时，Kafka如何保证消息顺序?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！