如何将分区添加到 Kafka 主题并在同一分区中保留相同的密钥消息? [英] How to add partition to Kafka topic and keep same-key message in same partition？

问题描述

要求在给定 Kafka 主题的同一分区中进行排序是很常见的.也就是说，具有相同键的消息应该进入相同的分区.现在，如果我想在正在运行的主题中添加新的分区，如何制作并保持一致性?

It is common to require ordering in same partition of given Kafka topic. That is, messages with same key should go to same partition. Now, if I want to add new partition in a running topic, how to make it and kept the consistency?

据我所知，默认的分区策略是修改 num-of-partition .当分区数量发生变化时(例如 4 到 5)，某些消息可能会与具有相同键的先前消息落入不同的分区.

To my understanding, the default partitioning strategy is to mod on num-of-partition . When the num-of-partition changes (e.g. 4 to 5), some messages might fall into different partition from previous messages with same key.

我可以想象实现一致的散列来自定义分区行为，但这可能是侵入性的.

I can image to have consistent hashing implemented to customize the partitioning behavior, but it might be to intrusive.

或者，停止所有生产者，直到所有消息都被消耗完；然后部署新分区并重启所有生产者.

Or, just stop all producers until all messages are consumed up; then deploy new partition and restart all producers.

有更好的想法吗?

推荐答案

正如您所说，当您增加主题中的分区数量时，您肯定会失去具有相同键的消息的顺序.

As you said, when you increase the number of partitions in a topic you will definitely loose the ordering of messages with the same key.

如果您尝试实现自定义分区器以一致地为分区分配键，您就不会真正使用新分区.

If you try to implement a customized partitioner to have a consistent assignment of a key to a partition, you wouldn't really use the new partition(s).

我将创建一个具有所需分区数量的新主题，并让生产者写入该新主题.一旦旧主题的消费者处理完所有消息(即消费者滞后 = 0)，您就可以让消费者从新主题中读取.

I would create a new topic with the desired amount of partitions and let the producer write into that new topic. As soon as the consumers of the old topic have processed all messages (i.e. consumer lag = 0) you could let the consumers read from the new topic.

这篇关于如何将分区添加到 Kafka 主题并在同一分区中保留相同的密钥消息?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！