Kafka Connect找不到连接器 [英] Kafka Connect can't find connector
问题描述
我正在尝试使用Kafka Connect Elasticsearch连接器,但未成功。它因以下错误而崩溃:
I'm trying to use the Kafka Connect Elasticsearch connector, and am unsuccessful. It is crashing with the following error:
[2018-11-21 14:48:29,096] ERROR Stopping after connector error (org.apache.kafka.connect.cli.ConnectStandalone:108)
java.util.concurrent.ExecutionException: org.apache.kafka.connect.errors.ConnectException: Failed to find any class that implements Connector and which name matches io.confluent.connect.elasticsearch.ElasticsearchSinkConnector , available connectors are: PluginDesc{klass=class org.apache.kafka.connect.file.FileStreamSinkConnector, name='org.apache.kafka.connect.file.FileStreamSinkConnector', version='1.0.1', encodedVersion=1.0.1, type=sink, typeName='sink', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.file.FileStreamSourceConnector, name='org.apache.kafka.connect.file.FileStreamSourceConnector', version='1.0.1', encodedVersion=1.0.1, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.MockConnector, name='org.apache.kafka.connect.tools.MockConnector', version='1.0.1', encodedVersion=1.0.1, type=connector, typeName='connector', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.MockSinkConnector, name='org.apache.kafka.connect.tools.MockSinkConnector', version='1.0.1', encodedVersion=1.0.1, type=sink, typeName='sink', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.MockSourceConnector, name='org.apache.kafka.connect.tools.MockSourceConnector', version='1.0.1', encodedVersion=1.0.1, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.SchemaSourceConnector, name='org.apache.kafka.connect.tools.SchemaSourceConnector', version='1.0.1', encodedVersion=1.0.1, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.VerifiableSinkConnector, name='org.apache.kafka.connect.tools.VerifiableSinkConnector', version='1.0.1', encodedVersion=1.0.1, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.VerifiableSourceConnector, name='org.apache.kafka.connect.tools.VerifiableSourceConnector', version='1.0.1', encodedVersion=1.0.1, type=source, typeName='source', location='classpath'}
我已经在kafka子文件夹中解压缩了插件的构建,并且在connect-standalone.properties中包含以下行:
I've got a build for the plugin unzipped in a kafka subfolder, and have the following line in connect-standalone.properties:
plugin.path=/opt/kafka/plugins/kafka-connect-elasticsearch-5.0.1/src/main/java/io/confluent/connect/elasticsearch
我可以看到该文件夹中的各种连接器,但是Kafka Connect不会加载它们;但确实会加载标准连接器,如下所示:
I can see the various connectors inside that folder, but Kafka Connect does not load them; but it does load the standard connectors, like this:
[2018-11-21 14:56:28,258] INFO Added plugin 'org.apache.kafka.connect.transforms.Cast$Value' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:136)
[2018-11-21 14:56:28,259] INFO Added aliases 'FileStreamSinkConnector' and 'FileStreamSink' to plugin 'org.apache.kafka.connect.file.FileStreamSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:335)
[2018-11-21 14:56:28,260] INFO Added aliases 'FileStreamSourceConnector' and 'FileStreamSource' to plugin 'org.apache.kafka.connect.file.FileStreamSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:335)
如何正确注册连接器?
推荐答案
昨天我在没有融合平台等的docker上的kafka上手动运行了jdbc连接器,只是为了了解这些东西是如何工作的。我不必在我这边或类似的东西上盖罐子。希望它对您有意义-我所做的是(我将跳过Docker部件如何使用连接器等方式挂载dir):
I ran jdbc connector yesterday manually on kafka in docker without confluent platform etc just to learn how those things works underneath. I did not have to build jar on my side or anyhing like this. Hopefully it will be relevant for you - what I did is ( I will skip docker parts howto mount dir with connector etc ):
- 下载连接器来自 https://www.confluent.io/connector/kafka-connect-jdbc/ ,解压缩zip
-
将zip的内容放到属性文件中配置的路径中的目录中(如下第3点所示)-
- download connector from https://www.confluent.io/connector/kafka-connect-jdbc/, unpack zip
put contents of zip to directory in path configured in properties file ( shown below in 3rd point ) -
plugin.path=/plugins
so树看起来像这样:
so tree looks something like this:
/plugins/
└── jdbcconnector
└──assets
└──doc
└──etc
└──lib
请注意 lib 目录的依赖项所在,其中之一是 kafka-connect-jdbc-5.0.0.jar
Note the lib dir where are the dependencies are, one of them is kafka-connect-jdbc-5.0.0.jar
现在您可以尝试运行连接器
Now you can try to run connector
./connect-standalone.sh connect-standalone.properties jdbc-connector-config.properties
connect-standalone.properties是kafka-connect所需的常见属性,在我的情况下:
connect-standalone.properties are common properties needed for kafka-connect, in my case:
bootstrap.servers=localhost:9092
key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=true
value.converter.schemas.enable=true
offset.storage.file.filename=/tmp/connect.offsets
offset.flush.interval.ms=10000
plugin.path=/plugins
rest.port=8086
rest.host.name=127.0.0.1
jdbc-connector-config.properties涉及更多,因为它只是配置这个特定的连接器,您需要深入研究连接器文档-对于jdbc源,它是 https://docs.confluent.io/current/connect/kafka-connect-jdbc/source-connector/source_config_options.html
jdbc-connector-config.properties is more involving, as it's just configuration for this particular connector, you need to dig into connector docs - for jdbc source it is https://docs.confluent.io/current/connect/kafka-connect-jdbc/source-connector/source_config_options.html
这篇关于Kafka Connect找不到连接器的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!