重新启动集群时，连接消费者作业被删除 [英] Connect consumers jobs are getting deleted when restarting the cluster

问题描述

我在更改与 kafka 相关的一些属性并重新启动集群时遇到以下问题.

I am facing the below issue on changing some properties related to kafka and re-starting the cluster.

In kafka Consumer, there were 5 consumer jobs are running . 

如果我们进行了一些重要的属性更改，并且在重新启动集群时，一些/所有现有的消费者作业将无法启动.

If we make some important property change , and on restarting cluster some/all the existing consumer jobs are not able to start.

Ideally all the consumer jobs should start , 

因为它将从下面的 System-topics 中获取元数据信息.

since it will take the meta-data info from the below System-topics .

config.storage.topic
offset.storage.topic
status.storage.topic

推荐答案

首先，介绍一下背景.Kafka 将其所有数据存储在 topics 中，但这些主题(或者更确切地说是组成的分区)一个主题)是仅附加的日志，除非完成某些操作，否则它将永远增长.为了防止这种情况，Kafka 可以通过两种方式清理主题:保留和压缩.配置为使用 retention 的主题将在可配置的时间长度内保留数据:代理可以自由删除任何早于此时间的日志消息.配置为使用 compaction 的主题需要每条消息都有一个密钥，代理将始终为每个不同的密钥保留最后一条已知消息.当每条消息(即键/值对)代表键的最后一个已知状态时，压缩非常方便；由于消费者正在阅读主题以获取每个键的最后一个已知状态，因此如果删除旧状态，他们最终会更快地到达最后一个状态.

First, a bit of background. Kafka stores all of its data in topics, but those topics (or rather the partitions that make up a topic) are append-only logs that would grow forever unless something is done. To prevent this, Kafka has the ability to clean up topics in two ways: retention and compaction. Topics configured to use retention will retain data for a configurable length of time: the broker is free to remove any log messages that are older than this. Topics configured to use compaction require every message have a key, and the broker will always retain the last known message for every distinct key. Compaction is extremely handy when each message (i.e., key/value pair) represents the last known state for the key; since consumers are reading the topic to get the last known state for each key, they will eventually get to that last state a bit faster if older states are removed.

代理将用于主题的清理策略取决于几个因素.默认情况下，每个隐式或显式创建的主题都将使用 retention，不过您可以通过多种方式进行更改:

Which cleanup policy a broker will use for a topic depends on several things. Every topic created implicitly or explicitly will use retention by default, though you can change a couple of ways:

• 更改全局 log.cleanup.policy 代理设置，仅影响在那之后创建的主题；或
• 在您创建或修改时指定cleanup.policy 特定于主题的设置一个话题

• change the globally log.cleanup.policy broker setting, affecting only topics created after that point; or
• specify the cleanup.policy topic-specific setting when you create or modify a topic

现在，Kafka Connect 使用多个内部主题来存储连接器配置、偏移量和状态信息.这些内部主题必须是压缩主题 以便(至少)每个连接器的最后配置、偏移量和状态始终可用.由于 Kafka Connect 从不使用旧的配置、偏移量和状态，因此代理将它们从内部主题中删除实际上是一件好事.

Now, Kafka Connect uses several internal topics to store connector configurations, offsets, and status information. These internal topics must be compacted topics so that (at least) the last configuration, offset, and status for each connector are always available. Since Kafka Connect never uses older configurations, offsets, and status, it's actually a good thing for the broker to remove them from the internal topics.

在 Kafka 0.11.0.0 之前，推荐流程是使用正确的主题特定设置手动创建这些内部主题.您可以依靠代理来自动创建它们，但由于多种原因而存在问题，其中最重要的是三个内部主题应该具有不同数量的分区.

Before Kafka 0.11.0.0, the recommended process is to manually create these internal topics using the correct topic-specific settings. You could rely upon the broker to auto-create them, but that is problematic for several reasons, not the least of which is that the three internal topics should have different numbers of partitions.

如果这些内部主题没有压缩，配置、偏移量和状态信息将在保​​留期结束后被清理和删除.默认情况下，此保留期为 24 小时！这意味着，如果您在部署/更新连接器配置后超过 24 小时重新启动 Kafka Connect，则该连接器的配置可能已被清除，并且看起来好像该连接器配置从未存在过一样.

If these internal topics are not compacted, the configurations, offsets, and status info will be cleaned up and removed after the retention period has elapsed. By default this retention period is 24 hours! That means that if you restart Kafka Connect more than 24 hours after deploying / updating a connector configuration, that connector's configuration may have been purged and it will appear as if the connector configuration never existed.

因此，如果您没有正确创建这些内部主题，只需使用主题管理工具 更新主题的设置，如文档中所述.

So, if you didn't create these internal topics correctly, simply use the topic admin tool to update the topic's settings as described in the documentation.

顺便说一句，没有正确创建这些内部主题是一个非常普遍的问题，以至于 Kafka Connect 0.11.0.0 将能够使用正确的设置自动创建这些内部主题，而无需依赖代理自动创建主题.

BTW, not properly creating these internal topics is a very common problem, so much so that Kafka Connect 0.11.0.0 will be able to automatically create these internal topics using the correct settings without relying upon broker auto-creation of topics.

在 0.11.0 中，对于源连接器写入的主题，您仍然必须依赖手动创建或代理自动创建.这并不理想，因此有一个 提案 更改 Kafka Connect 以自动为源连接器创建主题，同时让源连接器控制设置.希望这一改进使其成为 0.11.1.0，以便 Kafka Connect 更易于使用.

In 0.11.0 you will still have to rely upon manual creation or broker auto-creation for topics that source connectors write to. This is not ideal, and so there's a proposal to change Kafka Connect to automatically create the topics for the source connectors while giving the source connectors control over the settings. Hopefully that improvement makes it into 0.11.1.0 so that Kafka Connect is even easier to use.

这篇关于重新启动集群时，连接消费者作业被删除的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！