如何在Kafka主题上添加分区，并在同一分区中保留相同的键消息? [英] How to add partition to Kafka topic and keep same-key message in same partition？

问题描述

通常需要在给定的Kafka主题的相同分区中订购.也就是说，具有相同密钥的消息应该进入相同的分区.现在，如果要在正在运行的主题中添加新分区，如何制作并保持一致性?

It is common to require ordering in same partition of given Kafka topic. That is, messages with same key should go to same partition. Now, if I want to add new partition in a running topic, how to make it and kept the consistency?

据我了解，默认的分区策略是对num-of-partition进行修改.当分区数更改时(例如4到5)，某些邮件可能会与具有相同密钥的先前邮件落入不同的分区中.

To my understanding, the default partitioning strategy is to mod on num-of-partition . When the num-of-partition changes (e.g. 4 to 5), some messages might fall into different partition from previous messages with same key.

我可以为实现自定义分区行为而实现一致的哈希来进行映像，但这可能是侵入性的.

I can image to have consistent hashing implemented to customize the partitioning behavior, but it might be to intrusive.

或者，仅停止所有生产者，直到所有消息都被消耗掉为止；否则，只需停止所有生产者.然后部署新分区并重新启动所有生产者.

Or, just stop all producers until all messages are consumed up; then deploy new partition and restart all producers.

还有更好的主意吗?

推荐答案

正如您所说，当您增加一个主题中的分区数时，您肯定会松开具有相同键的消息顺序.

As you said, when you increase the number of partitions in a topic you will definitely loose the ordering of messages with the same key.

如果您尝试实现自定义分区程序以将键的一致性分配给分区，则您实际上不会使用新分区.

If you try to implement a customized partitioner to have a consistent assignment of a key to a partition, you wouldn't really use the new partition(s).

我将创建一个具有所需分区数量的新主题，并让制作人将其写入该新主题.旧主题的使用者处理完所有消息后(即消费者滞后= 0)，就可以让消费者从新主题中读取内容.

I would create a new topic with the desired amount of partitions and let the producer write into that new topic. As soon as the consumers of the old topic have processed all messages (i.e. consumer lag = 0) you could let the consumers read from the new topic.

这篇关于如何在Kafka主题上添加分区，并在同一分区中保留相同的键消息?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！