Kafka 接收器连接器:即使重启后也未分配任务 [英] Kafka sink connector: No tasks assigned, even after restart

问题描述

我在一组 Docker 容器中使用 Confluent 3.2，其中一个正在运行 kafka-connect worker.

I am using Confluent 3.2 in a set of Docker containers, one of which is running a kafka-connect worker.

由于我不清楚的原因，我的四个连接器中的两个 - 具体来说，hpgraphsl 的 MongoDB接收器连接器 - 停止工作.我能够确定主要问题:连接器没有分配任何任务，这可以通过调用 GET/connectors/{my_connector}/status 看到.其他两个(相同类型的)连接器没有受到影响，并且正在愉快地产生输出.

For reasons yet unclear to me, two of my four connectors - to be specific, hpgraphsl's MongoDB sink connector - stopped working. I was able to identify the main problem: The connectors did not have any tasks assigned, as could be seen by calling GET /connectors/{my_connector}/status. The other two connectors (of the same type) were not affected and were happily producing output.

我尝试了三种不同的方法来让我的连接器通过 REST API 再次运行:

I tried three different methods to get my connectors running again via the REST API:

• 暂停和恢复连接器
• 重新启动连接器
• 使用相同的配置删除和创建同名连接器

没有一种方法有效.我终于让我的连接器重新工作了:

None of the methods worked. I finally got my connectors working again by:

• 以不同的名称删除和创建连接器，例如 my_connector_v2 而不是 my_connector

这里发生了什么?为什么我无法重新启动现有连接器并让它开始实际任务?是否有任何关于 kafka-connect worker 的陈旧数据或需要清理的 Kafka brokers 上的一些与 kafka-connect 相关的主题?

What is going on here? Why am I not able to restart my existing connector and get it to start an actual task? Is there any stale data on the kafka-connect worker or in some kafka-connect-related topic on the Kafka brokers that needs to be cleaned?

我在特定连接器的 github 存储库上提交了一个问题，但我觉得这实际上可能是与 kafka-connect 的内在函数相关的一般错误.有什么想法吗?

I have filed an issue on the specific connector's github repo, but I feel like this might actually be general bug related to the intrinsics of kafka-connect. Any ideas?

推荐答案

我遇到过这个问题.如果 SinkTask 或 SourceTask 启动所需的资源较少，就会发生这种情况.

I have faced this issue. If the resources are less for a SinkTask or SourceTask to start, this can happen.

分配给 worker 的内存在一段时间内可能会减少.默认情况下，worker 被分配了 250MB.请增加这个.下面是一个为分布式模式运行的worker分配2GB内存的例子.

Memory allocated to the worker may be less some time. By default workers are allocated 250MB. Please increase this. Below is an example to allocate 2GB memory for the worker running in distributed mode.

KAFKA_HEAP_OPTS="-Xmx2G" sh $KAFKA_SERVICE_HOME/connect-distributed $KAFKA_CONFIG_HOME/connect-avro-distributed.properties

KAFKA_HEAP_OPTS="-Xmx2G" sh $KAFKA_SERVICE_HOME/connect-distributed $KAFKA_CONFIG_HOME/connect-avro-distributed.properties

这篇关于Kafka 接收器连接器:即使重启后也未分配任务的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！