kafka +如何避免磁盘存储空间不足 [英] kafka + how to avoid running out of disk storage

问题描述

我想描述以下在我们的生产集群之一中的情况

我们有具有2.6.4 HDP版本的ambari集群

集群包含3台kafka机器-而每台kafka都有5 T的磁盘

我们看到的是所有kafka磁盘都具有100％的大小，因此kafka磁盘已满，这就是所有kafka代理失败的原因

  df -h/kafka已使用的文件系统大小可用百分比已安装在/dev/sdb 5T 5T 23M 100％/var/kafka 

经过调查，我们发现 log.retention.hours = 7天

所以清除似乎是在7天之后进行的，这也许就是kafka磁盘即使容量很大(5T)也已充满100％的原因

我们现在想做的是-将来如何避免这种情况发生?

所以

我们想知道–如何避免卡夫卡磁盘上的全部已用容量

我们需要在Kafka配置中进行设置，以便根据磁盘大小清除kafka磁盘-可能吗?

以及如何知道 log.retention.hours 的正确值?是根据磁盘大小还是其他?

解决方案

在Kafka中，日志保留有两种类型；第二种是日志保留. size 和 time 保留时间.前者由 log.retention.bytes 触发，而后者由 log.retention.hours 触发.

在您的情况下，您应该注意 size 保留，有时配置起来很棘手.假设您需要 delete 清理策略，则需要将以下参数配置为

  log.cleaner.enable = truelog.cleanup.policy =删除 

然后，您需要考虑 log.retention.bytes ， log.segment.bytes 和 log.retention.check.interval.ms的配置.为此，您必须考虑以下因素:

• log.retention.bytes 是对主题的单个分区的最低保证，这意味着如果您设置 log.retention.bytes到512MB，这意味着您的磁盘上(每个分区)将始终拥有512MB的数据.

同样，如果您将 log.retention.bytes 设置为512MB，并且将 log.retention.check.interval.ms 设置为5分钟(这是默认设置)值)，在触发保留策略之前，您将在至少5分钟的时间内拥有至少 512MB的数据+生成的数据大小.

• 磁盘上的主题日志由段组成.段大小取决于 log.segment.bytes 参数.对于 log.retention.bytes = 1GB 和 log.segment.bytes = 512MB ，磁盘上将始终具有多达3个分段(2个分段分别达到保留和第三个将是当前写入数据的活动段).

最后，您应该进行数学运算并计算Kafka日志在磁盘上任何给定时间可能保留的最大大小，并相应地调整上述参数.当然，我也建议您设置一个时间保留策略，并相应地配置 log.retention.hours .如果两天后不再需要数据，则设置 log.retention.hours = 48 .

I want to described the following case that was on one of our production cluster

We have ambari cluster with HDP version 2.6.4

Cluster include 3 kafka machines – while each kafka have disk with 5 T

What we saw is that all kafka disks was with 100% size , so kafka disk was full and this is the reason that all kafka brokers was failed

df -h /kafka
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb         5T   5T   23M   100% /var/kafka

After investigation we saw that log.retention.hours=7 days

So seems that purging is after 7 days and maybe this is the reason that kafka disks are full with 100% even if they are huge – 5T

What we want to do now – is how to avoid this case in the future?

So

We want to know – how to avoid full used capacity on kafka disks

What we need to set in Kafka config in order to purge the kafka disk according to the disk size – is it possible ?

And how to know the right value of log.retention.hours ? according to the disk size or other?

解决方案

In Kafka, there are two types of log retention; size and time retention. The former is triggered by log.retention.bytes while the latter by log.retention.hours.

In your case, you should pay attention to size retention that sometimes can be quite tricky to configure. Assuming that you want a delete cleanup policy, you'd need to configure the following parameters to

log.cleaner.enable=true
log.cleanup.policy=delete

Then you need to think about the configuration of log.retention.bytes, log.segment.bytes and log.retention.check.interval.ms. To do so, you have to take into consideration the following factors:

• log.retention.bytes is a minimum guarantee for a single partition of a topic, meaning that if you set log.retention.bytes to 512MB, it means you will always have 512MB of data (per partition) in your disk.

• Again, if you set log.retention.bytes to 512MB and log.retention.check.interval.ms to 5 minutes (which is the default value) at any given time, you will have at least 512MB of data + the size of data produced within the 5 minute window, before the retention policy is triggered.

• A topic log on disk, is made up of segments. The segment size is dependent to log.segment.bytes parameter. For log.retention.bytes=1GB and log.segment.bytes=512MB, you will always have up to 3 segments on the disk (2 segments which reach the retention and the 3rd one will be the active segment where data is currently written to).

Finally, you should do the math and compute the maximum size that might be reserved by Kafka logs at any given time on your disk and tune the aforementioned parameters accordingly. Of course, I would also advice to set a time retention policy as well and configure log.retention.hours accordingly. If after 2 days you don't need your data anymore, then set log.retention.hours=48.

这篇关于kafka +如何避免磁盘存储空间不足的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！