kafka 连接异常，复制因子:3 比可用代理大:1 [英] kafka connect exception, Replication factor: 3 larger than available brokers: 1

问题描述

kafka 连接 5.4只有一个连接器，一个工人，并使用分布式连接.

kafka connect 5.4 only one connector, one worker and use connect-distributed.

以下是错误信息:

[2020-06-22 19:09:58,700] ERROR [Worker clientId=connect-1, groupId=test-cluster] 
Uncaught exception in herder work thread, exiting:  (org.apache.kafka.connect.runtime.distributed.DistributedHerder:290)
org.apache.kafka.connect.errors.ConnectException: Error while attempting to create/find topic(s) 'test-connect-offsets'
    at org.apache.kafka.connect.util.TopicAdmin.createTopics(TopicAdmin.java:262)
    at org.apache.kafka.connect.storage.KafkaOffsetBackingStore$1.run(KafkaOffsetBackingStore.java:99)
    at org.apache.kafka.connect.util.KafkaBasedLog.start(KafkaBasedLog.java:128)
    at org.apache.kafka.connect.storage.KafkaOffsetBackingStore.start(KafkaOffsetBackingStore.java:109)
    at org.apache.kafka.connect.runtime.Worker.start(Worker.java:186)
    at org.apache.kafka.connect.runtime.AbstractHerder.startServices(AbstractHerder.java:121)
    at org.apache.kafka.connect.runtime.distributed.DistributedHerder.run(DistributedHerder.java:277)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 3 larger than available brokers: 1.
    at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
    at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
    at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
    at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
    at org.apache.kafka.connect.util.TopicAdmin.createTopics(TopicAdmin.java:229)
    ... 11 more
Caused by: org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 3 larger than available brokers: 1.

推荐答案

为了让 Kafka Connect 在分布式模式下运行，它使用三个主题存储在 Kafka 集群上并保存有关配置等信息.您需要在 Kafka 中进行设置连接工作器属性:

For Kafka Connect to run in distributed mode it uses three topics that are stored on the Kafka cluster and hold information about configuration etc. You need to set in the Kafka Connect worker properties:

config.storage.replication.factor=1
offset.storage.replication.factor=1
status.storage.replication.factor=1

如果您使用的是 Docker 镜像，那么您需要设置环境变量来覆盖这些，在 Docker Compose 中如下所示:

If you're using the Docker image then you need to set the environment variables to override these, which in Docker Compose looks like:

CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: "1"
CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: "1"
CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: "1"

参考:配置Kafka Connect分布式工作者

这篇关于kafka 连接异常，复制因子:3 比可用代理大:1的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！