Kafka Connect 找不到开发的插件类 [英] Kafka Connect cant' find class of developed plugin

问题描述

我为实现 SinkConnector 的 kafka connect 创建了一个插件，我使用 gradle jar 任务将其打包到一个 jar 中:

I created a plugin for kafka connect that implements SinkConnector, I packaged it into a jar using gradle jar task:

jar {
  archiveName='name.jar'
}

我将它复制到文件夹中的 kafka 集群中，并设置 CLASSPATH=我的 jar 所在的位置.然后我执行 kafka 脚本来启动独立连接，它给我一个错误，说我的类找不到:

I copy it into the kafka cluster in a folder, and I set CLASSPATH=where my jar is. Then I execute the kafka script to start the standalone connect and it gives me an error saying that my class can't be found:

[2017-07-25 05:15:52,084] WARN could not get type for name mypackage.SplunkSinkConnector from any class loader (org.reflections.Reflections:384)
org.reflections.ReflectionsException: could not get type for name mypackage.SplunkSinkConnector
        at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:378)
        at org.reflections.ReflectionUtils.forNames(ReflectionUtils.java:397)
        at org.reflections.Reflections.getSubTypesOf(Reflections.java:367)
        at org.apache.kafka.connect.runtime.PluginDiscovery.connectorPlugins(PluginDiscovery.java:76)
        at org.apache.kafka.connect.runtime.PluginDiscovery.scanClasspathForPlugins(PluginDiscovery.java:70)
        at org.apache.kafka.connect.runtime.AbstractHerder$1.run(AbstractHerder.java:354)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: mypackage.SplunkSinkConnector
        at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:376)
        ... 6 more
[2017-07-25 05:15:52,419] INFO Reflections took 1586 ms to scan 62 urls, producing 3002 keys and 15379 values  (org.reflections.Reflections:229)
[2017-07-25 05:15:52,420] WARN could not get type for name mypackage.SplunkSinkConnector from any class loader (org.reflections.Reflections:384)
org.reflections.ReflectionsException: could not get type for name mypackage.SplunkSinkConnector
        at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:378)
        at org.reflections.ReflectionUtils.forNames(ReflectionUtils.java:397)
        at org.reflections.Reflections.getSubTypesOf(Reflections.java:367)
        at org.apache.kafka.connect.runtime.ConnectorFactory.getConnectorClass(ConnectorFactory.java:69)
        at org.apache.kafka.connect.runtime.ConnectorFactory.newConnector(ConnectorFactory.java:38)
        at org.apache.kafka.connect.runtime.AbstractHerder.getConnector(AbstractHerder.java:334)
        at org.apache.kafka.connect.runtime.AbstractHerder.validateConnectorConfig(AbstractHerder.java:233)
        at org.apache.kafka.connect.runtime.standalone.StandaloneHerder.putConnectorConfig(StandaloneHerder.java:159)
        at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:93)
Caused by: java.lang.ClassNotFoundException: mypackage.SplunkSinkConnector
        at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:376)
        ... 8 more

知道为什么不拿起我的罐子吗?

Any idea why is not picking up my jar?

谢谢

================================

================================

Kakfa Connect 10.2.1 版，根据脚本，类路径的计算方式为:CLASSPATH="$CLASSPATH":"$KAFKA_HOME/libs/*"

Kakfa Connect version 10.2.1, as per script, classpath is calculated with: CLASSPATH="$CLASSPATH":"$KAFKA_HOME/libs/*"

推荐答案

您能否检查您的 .jar 文件以确保该类在那里.使用 Scala 作为 JVM shell:

Can you check your .jar file to make sure that class is there. Using Scala as a JVM shell:

// Or Maybe try $CLASSPATH with the full classpath that you are using.
scala -classpath path-to-jar.jar
// If class is not loaded, this will trigger an error.
classOf[mypackage.SplunkSinkConnector]

仅供参考，我正是这样做的，使用自定义 .jar 插件并通过 CLASSPATH 环境变量加载了 Kafka Connect.

FYI, I am doing exactly this, using a custom .jar plugin with Kafka Connect loaded via the CLASSPATH environment variable.

UPDATE:这是我的插件的 build.gradle 文件.IMO，这是构建具有一些简单依赖项的 Java .jar 的最简单方法.我用 gradle jar 构建了 jar，它将在 ./build/libs/(project-name).jar:

UPDATE: Here is my build.gradle file for my plugin. IMO, this is the simplest way to build a Java .jar with some simple dependencies. I build the jar with gradle jar and it will be created at ./build/libs/(project-name).jar:

apply plugin: 'java'
apply plugin: 'idea'

sourceCompatibility = 1.8

repositories {
    mavenLocal()
    mavenCentral()
    maven { url "http://packages.confluent.io/maven/" }
}

configurations {
    all*.exclude group: 'org.slf4j', module: 'slf4j-log4j12'
    all*.exclude group: 'log4j'
}

dependencies {
    compile 'io.confluent:kafka-connect-storage-partitioner:3.2.1'
    compile 'org.apache.kafka:connect-api:0.10.2.1'

    testCompile 'junit:junit:4.+'
}

idea {
    project {
        languageLevel = '1.8'
    }
}

这篇关于Kafka Connect 找不到开发的插件类的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！