在Kafka配置中选择正确的清理策略 [英] Choosing the right cleanup policy in Kafka configuration

问题描述

我正在使用kafka_2.10-0.10.0.1.我创建了带有1个分区的主题.我知道默认的清除策略是删除".我想一直将所有记录保留在主题中(而不删除任何记录).什么是正确的方法:使"log.cleaner.enable = false"或"log.cleanup.policy = compact"?谢谢

I am using kafka_2.10-0.10.0.1. I created the topics with 1 partitions. I know that the default cleanup policy is "delete". I want to keep all the records in the topic all the time (without deleting any record). What's the right way : make "log.cleaner.enable=false" or "log.cleanup.policy=compact"? thank you

推荐答案

主题具有代理范围的配置，默认情况下适用于任何没有配置的主题，但是主题也可以具有特定于主题的配置，这些配置可以覆盖或补充经纪人范围的主题配置.

Topics have broker-wide configs that apply by default to any topic that doesn't have a config, but topics can also have topic-specific configs that override or complement broker-wide topic configs.

代理范围内的主题配置是在您的 service.properties 文件中设置的.特定于主题的配置是使用 bin/kafka-topics.sh 脚本或 AdminClient (如果使用的是Java)设置的.

Broker-wide topic configs are set in your service.properties file. Topic specific configs are set using the bin/kafka-topics.sh script or AdminClient if you are using Java.

与您相关的代理范围的配置为 log.retention.ms ，等效的主题特定的配置为 retention.ms .如果将 log.retention.ms 设置为 -1 ，则所有没有此 retention.ms 配置的主题都将具有无限的保留期.同样，如果您为特定主题的 retention.ms 设置了 -1 ，则保留期限将是无限的.

The relevant broker-wide config for you is log.retention.ms and the equivalent topic-specific config is retention.ms. If you set log.retention.ms to -1, all topics without this the retention.ms config will have an infinite retention period. Likewise, if you set -1 for retention.ms on a specific topic, it will have an infinite retention period.

要为新主题设置 retention.ms :

bin/kafka-topics.sh --zookeeper zk_host:port/chroot --create --topic my_topic_name
  --partitions 20 --replication-factor 3 --config retention.ms=-1

您还可以修改现有主题以将 retention.ms 设置为 -1 :

You can also modify an existing topic to set retention.ms to -1:

> bin/kafka-configs.sh --zookeeper zk_host:port/chroot --entity-type topics --entity-name my_topic_name --alter --add-config retention.ms=-1

在此处查看特定于主题的配置的完整列表: https://kafka.apache.org/documentation/#topicconfigs 以及有关主题操作​​的更多信息: https://kafka.apache.org/documentation/#basic_ops

See the full list of topic-specific configs here: https://kafka.apache.org/documentation/#topicconfigs and more about topic operations: https://kafka.apache.org/documentation/#basic_ops

这篇关于在Kafka配置中选择正确的清理策略的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！