Kafka Connect 启动失败 [英] Kafka Connect failed to start

问题描述

我在新的 linux centos 7 上安装了 kafka confluent oss 4.0，但 kafka connect 无法启动.

I installed kafka confluent oss 4.0 on a fresh linux centos 7 but kafka connect failed to start.

重现步骤:

 - Install Oracle JDK 8
 - Copy confluent-4.0.0 folder on opt/confluent-4.0.0
 - Run /opt/confluent-4.0.0/confluent start

结果:

Starting zookeeper
zookeeper is [UP]
Starting kafka
kafka is [UP]
Starting schema-registry
schema-registry is [UP]
Starting kafka-rest
kafka-rest is [UP]
Starting connect
\Kafka Connect failed to start
connect is [DOWN]

错误日志(connect.stderr):

Error Log (connect.stderr) :

Exception in thread "main" java.lang.NoClassDefFoundError: io/confluent/connect/storage/StorageSinkConnectorConfig
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
        at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
        at org.apache.kafka.connect.runtime.isolation.PluginClassLoader.loadClass(PluginClassLoader.java:54)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.Class.getDeclaredConstructors0(Native Method)
        at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)
        at java.lang.Class.getConstructor0(Class.java:3075)
        at java.lang.Class.newInstance(Class.java:412)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.getPluginDesc(DelegatingClassLoader.java:279)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:260)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:201)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.registerPlugin(DelegatingClassLoader.java:193)
        at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:153)
        at org.apache.kafka.connect.runtime.isolation.Plugins.<init>(Plugins.java:47)
        at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:70)
Caused by: java.lang.ClassNotFoundException: io.confluent.connect.storage.StorageSinkConnectorConfig
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at org.apache.kafka.connect.runtime.isolation.PluginClassLoader.loadClass(PluginClassLoader.java:62)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 22 more

附加信息:

Java 版本:

java version "1.8.0_151"
Java(TM) SE Runtime Environment (build 1.8.0_151-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.151-b12, mixed mode

Centos 版本:

centos-release-7-4.1708.el7.centos.x86_64

编辑每个属性文件中的 plugin.path 变量并没有解决问题.

Editing plugin.path variables in every properties files didn't fix the problem.

包含plugin.path"变量的文件列表:

List of files containing 'plugin.path' variable :

./etc/schema-registry/connect-avro-distributed.properties:84:plugin.path=/opt/confluent-4.0.0/share/java
./etc/schema-registry/connect-avro-standalone.properties:51:plugin.path=/opt/confluent-4.0.0/share/java
./etc/kafka/connect-distributed.properties:95:plugin.path=/opt/confluent-4.0.0/share/java
./etc/kafka/connect-standalone.properties:50:plugin.path=/opt/confluent-4.0.0/share/java

推荐答案

使用 Confluent 4.0.0，使用 plugin.path 默认为 Kafka Connect 启用.

With Confluent 4.0.0, classloading isolation with plugin.path is enabled by default for Kafka Connect.

当你从 deb 或 rpm 包安装 Confluent Platform 时，你的 plugin.path 的默认位置是事先知道的.

When you install Confluent Platform from deb or rpm packages the default location of your plugin.path is known beforehand.

但是，当您下载并解压缩 Confluent Platform 的 zip 或 tar.gz 存档文件系统中的某个位置时，它会设置为:

However, when you download and extract the zip or tar.gz archive of Confluent Platform somewhere in your filesystem, it's set to:

plugin.path=share/java

这是一个相对路径，因为当您将 Confluent Platform 作为存档下载(zip 或 tar.gz)时，您解压存档的位置是未知的(在上面的示例中，它是 /opt/confluent-4.0.0/).

This is a relative path, because when you download Confluent Platform as an archive (zip or tar.gz), the location where you extract the archive is not known (in your example above it's /opt/confluent-4.0.0/).

如果您从解压缩 Confluent 平台的目录运行它，CLI 或 Connect 的 bin 脚本将能够猜测此位置:

The CLI or Connect's bin scripts will be able to guess this location if you run it from the directory where you extracted Confluent platform:

例如，在上面的例子中:

For instance, in the example above:

cd/opt/confluent-4.0.0

./bin/confluent start

为了让您能够从文件系统中的任何目录启动 Connect，假设 Confluent Platform 的 bin 目录在您的 PATH 中，您需要将属性 plugin.path 设置为插件的绝对路径位置:

In order for you to be able to start Connect from any directory within your filesystem, given that the bin directory for Confluent Platform is in your PATH, you will need to set the property plugin.path to the absolute path location of your plugins:

要使用 Confluent CLI

To use Confluent CLI edit:

etc/schema-registry/connect-avro-distributed.properties

并适当地设置您的plugin.path(此处:plugin.path=/opt/confluent-4.0.0/share/java)

and set your plugin.path appropriately (here: plugin.path=/opt/confluent-4.0.0/share/java)

对于常规 bin 脚本

For the regular bin scripts edit:

./etc/kafka/connect-distributed.properties

和

./etc/kafka/connect-standalone.properties

并设置您的 plugin.path 如上(同样，在您的示例中:plugin.path=/opt/confluent-4.0.0/share/java).

and set your plugin.path as above (again, in your example: plugin.path=/opt/confluent-4.0.0/share/java).

这篇关于Kafka Connect 启动失败的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！