Kafka Connect 找不到连接器 [英] Kafka Connect can't find connector

问题描述

我正在尝试使用 Kafka Connect Elasticsearch 连接器，但未成功.它因以下错误而崩溃:

I'm trying to use the Kafka Connect Elasticsearch connector, and am unsuccessful. It is crashing with the following error:

[2018-11-21 14:48:29,096] ERROR Stopping after connector error (org.apache.kafka.connect.cli.ConnectStandalone:108)
java.util.concurrent.ExecutionException: org.apache.kafka.connect.errors.ConnectException: Failed to find any class that implements Connector and which name matches io.confluent.connect.elasticsearch.ElasticsearchSinkConnector , available connectors are: PluginDesc{klass=class org.apache.kafka.connect.file.FileStreamSinkConnector, name='org.apache.kafka.connect.file.FileStreamSinkConnector', version='1.0.1', encodedVersion=1.0.1, type=sink, typeName='sink', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.file.FileStreamSourceConnector, name='org.apache.kafka.connect.file.FileStreamSourceConnector', version='1.0.1', encodedVersion=1.0.1, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.MockConnector, name='org.apache.kafka.connect.tools.MockConnector', version='1.0.1', encodedVersion=1.0.1, type=connector, typeName='connector', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.MockSinkConnector, name='org.apache.kafka.connect.tools.MockSinkConnector', version='1.0.1', encodedVersion=1.0.1, type=sink, typeName='sink', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.MockSourceConnector, name='org.apache.kafka.connect.tools.MockSourceConnector', version='1.0.1', encodedVersion=1.0.1, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.SchemaSourceConnector, name='org.apache.kafka.connect.tools.SchemaSourceConnector', version='1.0.1', encodedVersion=1.0.1, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.VerifiableSinkConnector, name='org.apache.kafka.connect.tools.VerifiableSinkConnector', version='1.0.1', encodedVersion=1.0.1, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.VerifiableSourceConnector, name='org.apache.kafka.connect.tools.VerifiableSourceConnector', version='1.0.1', encodedVersion=1.0.1, type=source, typeName='source', location='classpath'}

我在 kafka 子文件夹中解压缩了插件的构建，并且在 connect-standalone.properties 中有以下行:

I've got a build for the plugin unzipped in a kafka subfolder, and have the following line in connect-standalone.properties:

plugin.path=/opt/kafka/plugins/kafka-connect-elasticsearch-5.0.1/src/main/java/io/confluent/connect/elasticsearch

我可以看到该文件夹​​中的各种连接器，但 Kafka Connect 没有加载它们；但它确实加载了标准连接器，如下所示:

I can see the various connectors inside that folder, but Kafka Connect does not load them; but it does load the standard connectors, like this:

[2018-11-21 14:56:28,258] INFO Added plugin 'org.apache.kafka.connect.transforms.Cast$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:136)
[2018-11-21 14:56:28,259] INFO Added aliases 'FileStreamSinkConnector' and 'FileStreamSink' to plugin 'org.apache.kafka.connect.file.FileStreamSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:335)
[2018-11-21 14:56:28,260] INFO Added aliases 'FileStreamSourceConnector' and 'FileStreamSource' to plugin 'org.apache.kafka.connect.file.FileStreamSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:335)

如何正确注册连接器?

推荐答案

我昨天在没有融合平台等的 docker 中的 kafka 上手动运行了 jdbc 连接器，只是为了了解这些东西在底层是如何工作的.我不必在我身边建造 jar 或任何类似的东西.希望它与您相关 - 我所做的是(我将跳过 docker 部件如何使用连接器等安装目录):

I ran jdbc connector yesterday manually on kafka in docker without confluent platform etc just to learn how those things works underneath. I did not have to build jar on my side or anyhing like this. Hopefully it will be relevant for you - what I did is ( I will skip docker parts howto mount dir with connector etc ):

• 从https://www.confluent.io/connector/kafka-下载连接器连接jdbc/，解压zip

• 将 zip 的内容放入属性文件中配置的路径中的目录(如下图第三点所示)-

• download connector from https://www.confluent.io/connector/kafka-connect-jdbc/, unpack zip

• put contents of zip to directory in path configured in properties file ( shown below in 3rd point ) -

plugin.path=/plugins

所以树看起来像这样:

/plugins/
└── jdbcconnector
    └──assets
    └──doc
    └──etc
    └──lib

注意lib 目录，依赖项在哪里，其中之一是kafka-connect-jdbc-5.0.0.jar

Note the lib dir where are the dependencies are, one of them is kafka-connect-jdbc-5.0.0.jar

现在您可以尝试运行连接器

Now you can try to run connector

./connect-standalone.sh connect-standalone.properties jdbc-connector-config.properties

connect-standalone.properties 是 kafka-connect 所需的常用属性，就我而言:

connect-standalone.properties are common properties needed for kafka-connect, in my case:

bootstrap.servers=localhost:9092
key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=true
value.converter.schemas.enable=true
offset.storage.file.filename=/tmp/connect.offsets
offset.flush.interval.ms=10000
plugin.path=/plugins
rest.port=8086
rest.host.name=127.0.0.1

jdbc-connector-config.properties 涉及更多，因为它只是此特定连接器的配置，您需要深入研究连接器文档 - 对于 jdbc 源，它是 https://docs.confluent.io/current/connect/kafka-connect-jdbc/source-connector/source_config_options.html

jdbc-connector-config.properties is more involving, as it's just configuration for this particular connector, you need to dig into connector docs - for jdbc source it is https://docs.confluent.io/current/connect/kafka-connect-jdbc/source-connector/source_config_options.html

这篇关于Kafka Connect 找不到连接器的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！