Kafka-根据主题前缀设置最小ISR [英] Kafka - set min ISR based on topics' prefix

问题描述

我有以下问题要解决.你们是否知道允许根据主题前缀设置不同的最小ISR值的机制?例如，使用前缀 app-* 创建的主题的最小ISR因子值设置为2， logs-* 等于1，但 core-* 为4.

I have following problem to solve. Do You Guys know mechanism that would allow set different min ISR value based on topic prefix? For example topics created with prefix app-* would have min ISR factor value set to 2, logs-* equal to 1 but value for core-* would be 4.

min.insync.replicas.per.topic = topic_name_1:value，topic_name_2:value 看起来几乎是一种解决方案，但不幸的是，当 min.insync.replicas 也出现在 kafka.properties 中.因此，我将探讨是否存在将这两个道具结合在一起的解决方案.来源

min.insync.replicas.per.topic=topic_name_1:value,topic_name_2:value looks almost as a solution but unfortunately it is shadowed when min.insync.replicas is also present in kafka.properties. So I will explore if there is solution to combine those two props. source

使用验证最小ISR值

./kafka-configs.sh --bootstrap-server localhost:9092 \
 --entity-type topics --entity-name $TOPIC --describe --all

推荐答案

为了避免使用80年代风格的解决方案(如本答案所示)，请查看Dirk的答案.

我只是喜欢bash/sh

对于已经创建的主题，您必须执行分区重新分配操作.

For already created topics, you'd have to execute a partition reassignment operation.

如果您希望控制新创建主题的 ISR ，则可以将 kafka-topics.sh 脚本包装在新脚本中，我们称之为 createTopics.sh :

If you wish to control the ISR for newly created topics, you could wrap the kafka-topics.sh script in a new script, lets call it createTopics.sh:

#!/bin/bash
isr=0

if [[ $1 = \logs* ]]; then
        isr=1
elif  [[ $1 = \app* ]]; then
        isr=2
elif  [[ $1 = \core* ]]; then
        isr=4
fi

/bin/kafka-topics.sh --create --topic $1 --replication-factor $isr \
                     --partitions $3  --bootstrap-server $2

• $ 1 -主题名称
• $ 2 -引导服务器
• $ 3 -分区
• $1 - topic name
• $2 - bootstrap-server
• $3 - partitions

例如:

  ./createTopics.sh apps-topic localhost:9092 1 && \
  ./createTopics.sh core-topic localhost:9092 1

这将创建两个具有各自复制因子的新主题:

This would create two new topics with their respective replication-factor:

• 应用主题 > ISR = 2 .
• 核心主题 > ISR = 4 .
• apps-topic > ISR = 2.
• core-topic > ISR = 4.

(最大ISR值，假设所有副本均处于同步状态)

该脚本还将验证新主题的创建，因为它不会创建名称不以以下任何一个开头的主题: logs/app/core .Kafka将显示错误，复制因子必须大于0 .

The script would also validate the creation of new topics, as it wouldn't create any topic which name doesn't start with one of these: logs / app / core. Kafka would show an error, Replication factor must be larger than 0.

您可以更改此行为，以允许使用不同的主题前缀，例如，通过在脚本开头将默认isr值设置为不同于0的值: isr = 1 .这将导致:

You could change this behaviour in order to allow different topic prefixes, by setting the default isr value at the start of the script different than 0: isr=1, for example. This would lead to:

#!/bin/bash
isr=1

if  [[ $1 = \app* ]]; then
       isr=2
elif  [[ $1 = \core* ]]; then
       isr=4
fi

/bin/kafka-topics.sh --create --topic $1 --replication-factor $isr \
                     --partitions $3  --bootstrap-server $2

这篇关于Kafka-根据主题前缀设置最小ISR的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！