在运行期间更改kafka保留期限 [英] changing kafka retention period during runtime

问题描述

使用Kafka 0.8.1.1，如何在运行时更改日志保留时间?文档表示该属性为 log.retention.hours ，但尝试对其进行更改使用 kafka-topics.sh 返回此错误

With Kafka 0.8.1.1, how do I change the log retention time while it's running? The documentation says the property is log.retention.hours, but trying to change it using kafka-topics.sh returns this error

$ bin/kafka-topics.sh --zookeeper zk.yoursite.com --alter --topic as-access --config topic.log.retention.hours=24
Error while executing topic command requirement failed: Unknown configuration "topic.log.retention.hours".
java.lang.IllegalArgumentException: requirement failed: Unknown configuration "topic.log.retention.hours".
    at scala.Predef$.require(Predef.scala:145)
    at kafka.log.LogConfig$$anonfun$validateNames$1.apply(LogConfig.scala:138)
    at kafka.log.LogConfig$$anonfun$validateNames$1.apply(LogConfig.scala:137)
    at scala.collection.Iterator$class.foreach(Iterator.scala:631)
    at scala.collection.JavaConversions$JEnumerationWrapper.foreach(JavaConversions.scala:479)
    at kafka.log.LogConfig$.validateNames(LogConfig.scala:137)
    at kafka.log.LogConfig$.validate(LogConfig.scala:145)
    at kafka.admin.TopicCommand$.parseTopicConfigsToBeAdded(TopicCommand.scala:171)
    at kafka.admin.TopicCommand$$anonfun$alterTopic$1.apply(TopicCommand.scala:95)
    at kafka.admin.TopicCommand$$anonfun$alterTopic$1.apply(TopicCommand.scala:93)
    at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:57)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:43)
    at kafka.admin.TopicCommand$.alterTopic(TopicCommand.scala:93)
    at kafka.admin.TopicCommand$.main(TopicCommand.scala:52)
    at kafka.admin.TopicCommand.main(TopicCommand.scala)

推荐答案

log.retention.hours 是代理的属性，在创建主题时用作默认值.使用 kafka-topics.sh 更改当前正在运行的主题的配置时，应指定主题级别的属性.

log.retention.hours is a property of a broker which is used as a default value when a topic is created. When you change configurations of currently running topic using kafka-topics.sh, you should specify a topic-level property.

日志保留时间的主题级属性为 retention.ms .

A topic-level property for log retention time is retention.ms.

来自Kafka 0.8.1文档中的主题级配置:

From Topic-level configuration in Kafka 0.8.1 documentation:

• 属性:retention.ms
• 默认:7天
• 服务器默认属性:log.retention.minutes
• 说明:如果我们使用删除"功能，则此配置控制了在保留旧日志段以释放空间之前，我们将保留日志的最长时间.保留政策.这表示有关消费者必须多长时间读取数据的SLA.

• Property: retention.ms
• Default: 7 days
• Server Default Property: log.retention.minutes
• Description: This configuration controls the maximum time we will retain a log before we will discard old log segments to free up space if we are using the "delete" retention policy. This represents an SLA on how soon consumers must read their data.

因此正确的命令取决于版本.最多 0.8.2 (尽管文档仍显示其使用率最高为0.10.1)在 0.9.0 ))使用 kafka-configs.sh --alter

So the correct command depends on the version. Up to 0.8.2 (although docs still show its use up to 0.10.1) use kafka-topics.sh --alter and after 0.10.2 (or perhaps from 0.9.0 going forward) use kafka-configs.sh --alter

$ bin/kafka-topics.sh --zookeeper zk.yoursite.com --alter --topic as-access --config retention.ms=86400000
 

您可以使用以下命令检查配置是否正确应用.

You can check whether the configuration is properly applied with the following command.

$ bin/kafka-topics.sh --describe --zookeeper zk.yoursite.com --topic as-access

然后您将看到类似下面的内容.

Then you will see something like below.

Topic:as-access  PartitionCount:3  ReplicationFactor:3  Configs:retention.ms=86400000

这篇关于在运行期间更改kafka保留期限的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！