是否可以将分区添加到 Kafka 0.8.2 中的现有主题 [英] Is it possible to add partitions to an existing topic in Kafka 0.8.2

问题描述

我有一个 Kafka 集群运行 2 个分区.我一直在寻找将分区数增加到 3 的方法.但是，我不想丢失有关该主题的现有消息.我尝试停止 Kafka，修改 server.properties 文件以增加分区数到 3 并重新启动 Kafka.然而，这似乎并没有改变任何事情.使用 Kafka ConsumerOffsetChecker，我仍然看到它只使用了 2 个分区.我使用的 Kafka 版本是 0.8.2.2.在 0.8.1 版本中，曾经有一个名为 kafka-add-partitions.sh 的脚本，我想它可能会起作用.但是，我在 0.8.2 中没有看到任何此类脚本.

I have a Kafka cluster running with 2 partitions. I was looking for a way to increase the partition count to 3. However, I don't want to lose existing messages on the topic. I tried stopping Kafka, modifying the server.properties file to increase the number of partitions to 3 and restart Kafka. However, that does not seem to change anything. Using Kafka ConsumerOffsetChecker, I still see it is using only 2 partitions. The Kafka version I am using is 0.8.2.2. In version 0.8.1, there used to be a script called kafka-add-partitions.sh, which I guess might do the trick. However, I don't see any such script in 0.8.2.

• 有没有办法做到这一点?

我确实尝试过创建一个全新的主题，对于那个主题，根据 server.properties 文件中的更改，它似乎确实使用了 3 个分区.但是，对于现有主题，它似乎并不关心.

I did experiment with creating a whole new topic and for that one, it does seem to use 3 partitions as per the change in the server.properties file. However, for existing topics, it doesn't seem to care.

推荐答案

看起来可以使用 this 脚本:

Looks like you can use this script instead:

bin/kafka-topics.sh --zookeeper zk_host:port/chroot --alter --topic my_topic_name 
   --partitions 40 

在代码中看起来他们做同样的事情:

In the code it looks like they do same thing:

 AdminUtils.createOrUpdateTopicPartitionAssignmentPathInZK(topic, partitionReplicaList, zkClient, true)

kafka-topics.sh 执行 这段 代码以及 AddPartitionsCommand 由 kafka-add-partition 脚本使用.

kafka-topics.sh executes this piece of code as well as AddPartitionsCommand used by kafka-add-partition script.

但是使用key时需要注意重新分区:

However you have to be aware of re-partitioning when using key:

请注意分区的一个用例是语义分区数据，并且添加分区不会改变现有数据的分区，因此如果消费者依赖它，这可能会打扰他们划分.也就是说，如果数据由 hash(key) % number_of_partitions 分区，那么这个分区可能是通过添加分区进行洗牌，但 Kafka 不会尝试以任何方式自动重新分配数据.

Be aware that one use case for partitions is to semantically partition data, and adding partitions doesn't change the partitioning of existing data so this may disturb consumers if they rely on that partition. That is if data is partitioned by hash(key) % number_of_partitions then this partitioning will potentially be shuffled by adding partitions but Kafka will not attempt to automatically redistribute data in any way.

这篇关于是否可以将分区添加到 Kafka 0.8.2 中的现有主题的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！