Kafka接收器连接器:即使重新启动也未分配任务 [英] Kafka sink connector: No tasks assigned, even after restart

问题描述

我在一组Docker容器中使用Confluent 3.2，其中一个正在运行kafka-connect worker.

I am using Confluent 3.2 in a set of Docker containers, one of which is running a kafka-connect worker.

出于我尚不清楚的原因，我的四个连接器中有两个-具体来说是hpgraphsl的 MongoDB接收器连接器-停止工作.我能够确定一个主要问题:连接器没有分配任何任务，可以通过调用GET /connectors/{my_connector}/status看到.其他两个(相同类型的)连接器没有受到影响，并很乐意产生输出.

For reasons yet unclear to me, two of my four connectors - to be specific, hpgraphsl's MongoDB sink connector - stopped working. I was able to identify the main problem: The connectors did not have any tasks assigned, as could be seen by calling GET /connectors/{my_connector}/status. The other two connectors (of the same type) were not affected and were happily producing output.

我尝试了三种不同的方法来使我的连接器通过REST API重新运行:

I tried three different methods to get my connectors running again via the REST API:

• 暂停和恢复连接器
• 重新启动连接器
• 使用相同的配置以相同的名称删除和创建连接器

没有一种方法有效.我终于通过以下方法使连接器再次工作:

None of the methods worked. I finally got my connectors working again by:

• 以不同的名称删除和创建连接器，例如说my_connector_v2而不是my_connector

• Deleting and creating the connector under a different name, say my_connector_v2 instead of my_connector

这是怎么回事?为什么我不能重新启动现有的连接器并使其开始实际的任务?是否需要清除有关kafka-connect的工作程序或Kafka经纪人上与kafka-connect相关的某些主题的任何过时数据?

What is going on here? Why am I not able to restart my existing connector and get it to start an actual task? Is there any stale data on the kafka-connect worker or in some kafka-connect-related topic on the Kafka brokers that needs to be cleaned?

我已经在特定连接器的github存储库上提出了问题 ，但我觉得这实际上可能是与kafka-connect内在函数有关的一般错误.有什么想法吗?

I have filed an issue on the specific connector's github repo, but I feel like this might actually be general bug related to the intrinsics of kafka-connect. Any ideas?

推荐答案

我已经遇到了这个问题.如果启动SinkTask或SourceTask的资源较少，则可能会发生这种情况.

I have faced this issue. If the resources are less for a SinkTask or SourceTask to start, this can happen.

分配给工作程序的内存可能会少一些时间.默认情况下，为工作人员分配250MB.请增加这个.下面是为在分布式模式下运行的工作程序分配2GB内存的示例.

Memory allocated to the worker may be less some time. By default workers are allocated 250MB. Please increase this. Below is an example to allocate 2GB memory for the worker running in distributed mode.

KAFKA_HEAP_OPTS =-Xmx2G" sh $ KAFKA_SERVICE_HOME/connect-distributed $ KAFKA_CONFIG_HOME/connect-avro-distributed.properties

KAFKA_HEAP_OPTS="-Xmx2G" sh $KAFKA_SERVICE_HOME/connect-distributed $KAFKA_CONFIG_HOME/connect-avro-distributed.properties

这篇关于Kafka接收器连接器:即使重新启动也未分配任务的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！