Kafka - 根据主题前缀设置最小 ISR [英] Kafka - set min ISR based on topics' prefix

问题描述

我有以下问题需要解决.你们知道允许根据主题前缀设置不同的最小 ISR 值的机制吗?例如，使用前缀 app-* 创建的主题会将最小 ISR 因子值设置为 2，logs-* 等于 1，但值为 core-* 将是 4.

I have following problem to solve. Do You Guys know mechanism that would allow set different min ISR value based on topic prefix? For example topics created with prefix app-* would have min ISR factor value set to 2, logs-* equal to 1 but value for core-* would be 4.

min.insync.replicas.per.topic=topic_name_1:value,topic_name_2:value 看起来几乎是一个解决方案，但不幸的是它在 min.insync.replicas 时被遮蔽了也存在于 kafka.properties 中.所以我会探索是否有解决方案可以将这两个道具结合起来.来源

min.insync.replicas.per.topic=topic_name_1:value,topic_name_2:value looks almost as a solution but unfortunately it is shadowed when min.insync.replicas is also present in kafka.properties. So I will explore if there is solution to combine those two props. source

使用

./kafka-configs.sh --bootstrap-server localhost:9092 \
 --entity-type topics --entity-name $TOPIC --describe --all

推荐答案

为了避免 80 年代风格的解决方案(如本答案中所示)，请查看 Dirk 的答案.

我就是喜欢 bash/sh

对于已创建的主题，您必须执行分区重新分配操作.

For already created topics, you'd have to execute a partition reassignment operation.

如果您希望控制新创建主题的 ISR，您可以将 kafka-topics.sh 脚本包装在一个新脚本中，我们称之为createTopics.sh:

If you wish to control the ISR for newly created topics, you could wrap the kafka-topics.sh script in a new script, lets call it createTopics.sh:

#!/bin/bash
isr=0

if [[ $1 = \logs* ]]; then
        isr=1
elif  [[ $1 = \app* ]]; then
        isr=2
elif  [[ $1 = \core* ]]; then
        isr=4
fi

/bin/kafka-topics.sh --create --topic $1 --replication-factor $isr \
                     --partitions $3  --bootstrap-server $2

• $1 - 主题名称
• $2 - 引导服务器
• $3 - 分区
• $1 - topic name
• $2 - bootstrap-server
• $3 - partitions

例如:

  ./createTopics.sh apps-topic localhost:9092 1 && \
  ./createTopics.sh core-topic localhost:9092 1

这将创建两个具有各自复制因子的新主题:

This would create two new topics with their respective replication-factor:

• apps-topic >ISR = 2.
• core-topic >ISR = 4.
• apps-topic > ISR = 2.
• core-topic > ISR = 4.

(最大 ISR 值，假设所有副本都同步)

该脚本还将验证新主题的创建，因为它不会创建名称不以以下之一开头的任何主题:logs/app/core.Kafka会报错，复制因子必须大于0.

The script would also validate the creation of new topics, as it wouldn't create any topic which name doesn't start with one of these: logs / app / core. Kafka would show an error, Replication factor must be larger than 0.

您可以更改此行为以允许不同的主题前缀，例如，通过在脚本开头设置不同于 0 的默认 isr 值:isr=1.这将导致:

You could change this behaviour in order to allow different topic prefixes, by setting the default isr value at the start of the script different than 0: isr=1, for example. This would lead to:

#!/bin/bash
isr=1

if  [[ $1 = \app* ]]; then
       isr=2
elif  [[ $1 = \core* ]]; then
       isr=4
fi

/bin/kafka-topics.sh --create --topic $1 --replication-factor $isr \
                     --partitions $3  --bootstrap-server $2

这篇关于Kafka - 根据主题前缀设置最小 ISR的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！