在 Kafka 配置中选择正确的清理策略 [英] Choosing the right cleanup policy in Kafka configuration

问题描述

我正在使用 kafka_2.10-0.10.0.1.我创建了 1 个分区的主题.我知道默认的清理策略是删除".我想一直保留主题中的所有记录(不删除任何记录).什么是正确的方法:使log.cleaner.enable=false"或log.cleanup.policy=compact"?

I am using kafka_2.10-0.10.0.1. I created the topics with 1 partitions. I know that the default cleanup policy is "delete". I want to keep all the records in the topic all the time (without deleting any record). What's the right way : make "log.cleaner.enable=false" or "log.cleanup.policy=compact"?

推荐答案

主题具有代理范围的配置，默认情况下适用于任何没有配置的主题，但主题也可以具有特定主题的配置，这些配置会覆盖或补充代理范围的主题配置.

Topics have broker-wide configs that apply by default to any topic that doesn't have a config, but topics can also have topic-specific configs that override or complement broker-wide topic configs.

代理范围的主题配置在您的 service.properties 文件中设置.如果您使用的是 Java，则使用 bin/kafka-topics.sh 脚本或 AdminClient 来设置特定于主题的配置.

Broker-wide topic configs are set in your service.properties file. Topic specific configs are set using the bin/kafka-topics.sh script or AdminClient if you are using Java.

与您相关的代理范围配置是 log.retention.ms，等效的特定主题配置是 retention.ms.如果您将 log.retention.ms 设置为 -1，则没有此 retention.ms 配置的所有主题将具有无限保留期.同样，如果您为特定主题的 retention.ms 设置 -1，它将具有无限保留期.

The relevant broker-wide config for you is log.retention.ms and the equivalent topic-specific config is retention.ms. If you set log.retention.ms to -1, all topics without this the retention.ms config will have an infinite retention period. Likewise, if you set -1 for retention.ms on a specific topic, it will have an infinite retention period.

在新主题上设置 retention.ms:

bin/kafka-topics.sh --zookeeper zk_host:port/chroot --create --topic my_topic_name
  --partitions 20 --replication-factor 3 --config retention.ms=-1

您还可以修改现有主题以将 retention.ms 设置为 -1:

You can also modify an existing topic to set retention.ms to -1:

> bin/kafka-configs.sh --zookeeper zk_host:port/chroot --entity-type topics --entity-name my_topic_name --alter --add-config retention.ms=-1

在此处查看特定于主题的配置的完整列表:https://kafka.apache.org/documentation/#topicconfigs 以及有关主题操作​​的更多信息:https://kafka.apache.组织/文档/#basic_ops

See the full list of topic-specific configs here: https://kafka.apache.org/documentation/#topicconfigs and more about topic operations: https://kafka.apache.org/documentation/#basic_ops

这篇关于在 Kafka 配置中选择正确的清理策略的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！