kafka + 如何避免磁盘存储不足 [英] kafka + how to avoid running out of disk storage

问题描述

我想描述以下在我们的一个生产集群上的案例

I want to described the following case that was on one of our production cluster

我们有 HDP 版本 2.6.4 的 ambari 集群

We have ambari cluster with HDP version 2.6.4

集群包括 3 台 kafka 机器——而每个 kafka 有 5 T 的磁盘

Cluster include 3 kafka machines – while each kafka have disk with 5 T

我们看到的是所有 kafka 磁盘都是 100% 大小，所以 kafka 磁盘已满，这就是所有 kafka broker 失败的原因

What we saw is that all kafka disks was with 100% size , so kafka disk was full and this is the reason that all kafka brokers was failed

df -h /kafka
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb         5T   5T   23M   100% /var/kafka

经过调查我们发现log.retention.hours=7 days

所以看起来清除是在 7 天后，也许这就是 kafka 磁盘 100% 满的原因，即使它们很大 - 5T

So seems that purging is after 7 days and maybe this is the reason that kafka disks are full with 100% even if they are huge – 5T

我们现在想要做的——是如何在未来避免这种情况?

What we want to do now – is how to avoid this case in the future?

所以

我们想知道 – 如何避免 kafka 磁盘上的容量已满

We want to know – how to avoid full used capacity on kafka disks

我们需要在 Kafka config 中设置什么才能根据磁盘大小清除 kafka 磁盘 - 可能吗?

What we need to set in Kafka config in order to purge the kafka disk according to the disk size – is it possible ?

以及如何知道 log.retention.hours 的正确值?根据磁盘大小或其他?

And how to know the right value of log.retention.hours ? according to the disk size or other?

推荐答案

在Kafka中，日志保留有两种类型；大小和时间保留.前者由log.retention.bytes触发，后者由log.retention.hours触发.

In Kafka, there are two types of log retention; size and time retention. The former is triggered by log.retention.bytes while the latter by log.retention.hours.

在您的情况下，您应该注意 size 保留，有时配置起来非常棘手.假设您想要一个 delete 清理策略，您需要将以下参数配置为

In your case, you should pay attention to size retention that sometimes can be quite tricky to configure. Assuming that you want a delete cleanup policy, you'd need to configure the following parameters to

log.cleaner.enable=true
log.cleanup.policy=delete

然后需要考虑log.retention.bytes、log.segment.bytes和log.retention.check.interval.ms的配置.为此，您必须考虑以下因素:

Then you need to think about the configuration of log.retention.bytes, log.segment.bytes and log.retention.check.interval.ms. To do so, you have to take into consideration the following factors:

• log.retention.bytes 是主题的单个分区的最低保证，这意味着如果您设置 log.retention.bytes 到 512MB，这意味着您的磁盘中将始终有 512MB 的数据(每个分区).

• log.retention.bytes is a minimum guarantee for a single partition of a topic, meaning that if you set log.retention.bytes to 512MB, it means you will always have 512MB of data (per partition) in your disk.

同样，如果您将 log.retention.bytes 设置为 512MB，将 log.retention.check.interval.ms 设置为 5 分钟(这是默认值)value) 在任何给定时间，在触发保留策略之前，您将拥有至少 512MB 的数据 + 5 分钟窗口内生成的数据大小.

Again, if you set log.retention.bytes to 512MB and log.retention.check.interval.ms to 5 minutes (which is the default value) at any given time, you will have at least 512MB of data + the size of data produced within the 5 minute window, before the retention policy is triggered.

磁盘上的主题日志，由段组成.段大小取决于 log.segment.bytes 参数.对于 log.retention.bytes=1GB 和 log.segment.bytes=512MB，您将始终在磁盘上最多有 3 个段(2 个段达到保留和第三个将是当前写入数据的活动段).

A topic log on disk, is made up of segments. The segment size is dependent to log.segment.bytes parameter. For log.retention.bytes=1GB and log.segment.bytes=512MB, you will always have up to 3 segments on the disk (2 segments which reach the retention and the 3rd one will be the active segment where data is currently written to).

最后，您应该进行数学运算并计算在任何给定时间在您的磁盘上可能被 Kafka 日志保留的最大大小，并相应地调整上述参数.当然，我也建议设置时间保留策略并相应地配置 log.retention.hours.如果 2 天后您不再需要数据，则设置 log.retention.hours=48.

Finally, you should do the math and compute the maximum size that might be reserved by Kafka logs at any given time on your disk and tune the aforementioned parameters accordingly. Of course, I would also advice to set a time retention policy as well and configure log.retention.hours accordingly. If after 2 days you don't need your data anymore, then set log.retention.hours=48.

这篇关于kafka + 如何避免磁盘存储不足的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！