如何检查kafka中的键分配给哪个分区? [英] How to check which partition is a key assign to in kafka?

问题描述

我正在尝试调试一个问题，试图证明如果群集没有重新平衡，则每个唯一的密钥只会进入1个分区.

I am trying to debug a issue for which I am trying to prove that each distinct key only goes to 1 partition if the cluster is not rebalancing.

所以我想知道给定的主题，有没有办法确定将密钥发送到哪个分区?

So I was wondering for a given topic, is there a way to determine which partition a key is send to?

推荐答案

您需要byte[] keyBytes假定它不为null，然后使用org.apache.kafka.common.utils.Utils，可以运行以下命令.

You need the byte[] keyBytes assuming it isn't null, then using org.apache.kafka.common.utils.Utils, you can run the following.

Utils.toPositive(Utils.murmur2(keyBytes)) % numPartitions;

对于字符串或JSON，它是UTF8编码的，而Utils类具有帮助函数来获取该信息.
对于Avro，例如Confluent序列化值，则要复杂一些(魔术字节，然后是模式ID，然后是数据).请参见电线格式

For strings or JSON, it's UTF8 encoded, and the Utils class has helper functions to get that.
For Avro, such as Confluent serialized values, it's a bit more complicated (a magic byte, then a schema ID, then the data). See Wire format

仅进入1个分区

only goes to 1 partition

这不是保证.哈希可能会冲突.

This isn't a guarantee. Hashes can collide.

说一个给定的密钥不在一个以上的分区中更有意义.

It makes more sense to say that a given key isn't in more than one partition.

如果群集没有重新平衡

if the cluster is not rebalancing

重新平衡仍将保留分区值.

Rebalancing will still preserve a partition value.

这篇关于如何检查kafka中的键分配给哪个分区?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！