Kafka Streams - 如何更好地控制内部创建的状态存储主题的分区? [英] Kafka Streams - How to better control partitioning of internally created state store topic?

问题描述

Kafka Streams 中的状态存储是在内部创建的.状态存储按键分区，但不允许提供除键以外的分区(据我所知).

State stores in Kafka Streams are created internally. State stores are partitioned by key, but do not allow to provide partitioning other than by key (to my knowledge).

问题

如何控制 state-store 内部创建的主题的分区数?状态存储主题如何推断默认使用的分区数量和分区，以及如何覆盖?

How to control the number of partitions of a state-store internally created topic? How does the state store topic infer the number of partitions and the partitioning to use by default, and how to override?

如果您想通过除传入键值记录的键以外的其他内容来分区状态存储并进行共同分区，如何解决此问题?在这种情况下，我想通过比我的常规键更具体的东西进行分区.例如.我有一个

How to work it around if you want to partition your state-store by something other than the key of your incoming key-value record and have co-partitioning? In this case, I'd like to partition by something more specific than my regular key. E.g. I have a

case class RegularKey(fieldA: String)

我想分区

case class SpecificKey(fieldA: String, fieldB: String)

对于我传入的源主题，我在分区数上使用了 HashPartitioner.

For my incoming source topic, I am using a HashPartitioner on the number of partitions.

推荐答案

changelog 主题分区的数量取决于输入主题分区的数量，您不能更改它，因为状态是基于这个数量共享的(即，有是一个分片分区).

The number of changelog topic partitions depends in the number of input topic partitions and you cannot change it, because the state is shared based on this number (ie, there is one partitions by shard).

如果你想通过某个属性来划分变更日志主题，你必须将其设置为键.在您的情况下，您必须将 SpecificKey 设置为消息键.不允许更改分区，因为它会破坏"Kafka Streams 导致错误的结果.

If you want to partition changelog topic by some attribute, you must set it as key. In your case, you must set SpecificKey as message key. It's not allowed to change the partitioning because it would "break" Kafka Streams leading to incorrect result.

这篇关于Kafka Streams - 如何更好地控制内部创建的状态存储主题的分区?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！