星火卡夫卡分流问题 [英] Spark Kafka Streaming Issue

问题描述

我使用maven

我添加了以下相关

 ＆LT;＆依赖性GT; ＆LT;！ - 星火依赖 - ＆GT;
      ＆LT;＆的groupId GT; org.apache.spark＆LT; /的groupId＆GT;
      ＆LT;＆的artifactId GT;火花streaming_2.10＆LT; / artifactId的＆GT;
      ＆LT;＆版GT; 1.1.0＆LT; /版本＆GT;
    ＆LT; /依赖性＆GT; ＆LT;＆依赖性GT; ＆LT;！ - 星火依赖 - ＆GT;
      ＆LT;＆的groupId GT; org.apache.spark＆LT; /的groupId＆GT;
      ＆LT;＆的artifactId GT;火花流-kafka_2.10＆LT; / artifactId的＆GT;
      ＆LT;＆版GT; 1.1.0＆LT; /版本＆GT;
    ＆LT; /依赖性＆GT;
 

我还增加了code罐子

  SparkConf sparkConf =新SparkConf（）setAppName（KafkaSparkTest）;
JavaSparkContext SC =新JavaSparkContext（sparkConf）;
sc.addJar(\"/home/test/.m2/repository/org/apache/spark/spark-streaming-kafka_2.10/1.0.2/spark-streaming-kafka_2.10-1.0.2.jar\");
JavaStreamingContext JSSC =新JavaStreamingContext（SC，新的持续时间（5000））;
 

据情结罚款任何错误，当我运行通过我收到以下错误火花提交，任何帮助是非常AP preciated。感谢您的时间。

 斌/火花提交--classKafkaSparkStreaming--master本地[4]尝试/简单的项目/目标/简单项目1.0.jar
 

  

异常线程mainjava.lang.NoClassDefFoundError的：组织/阿帕奇/火花/流/卡夫卡/ KafkaUtils
      在KafkaSparkStreaming.sparkStreamingTest（KafkaSparkStreaming.java:40）
      在KafkaSparkStreaming.main（KafkaSparkStreaming.java:23）
      在sun.reflect.NativeMethodAccessorImpl.invoke0（本机方法）
      在sun.reflect.NativeMethodAccessorImpl.invoke（NativeMethodAccessorImpl.java:57）
      在sun.reflect.DelegatingMethodAccessorImpl.invoke（DelegatingMethodAccessorImpl.java:43）
      在java.lang.reflect.Method.invoke（Method.java:606）
      在org.apache.spark.deploy.SparkSubmit $ .launch（SparkSubmit.scala：303）
      在org.apache.spark.deploy.SparkSubmit $。主要（SparkSubmit.scala：55）
      在org.apache.spark.deploy.SparkSubmit.main（SparkSubmit.scala）
  抛出java.lang.ClassNotFoundException：引起org.apache.spark.streaming.kafka.KafkaUtils
      在java.net.URLClassLoader的$ 1.run（URLClassLoader.java:366）

解决方案

我遇到了同样的问题，我解决它通过建立与依赖的jar。

1. 删除sc.addJar（）在code。




2. 下面添加code到pom.xml的

   ＆LT;建立＆GT;
    ＆LT; sourceDirectory＆GT;的src / main / java的＆LT; / sourceDirectory＆GT;
    ＆LT; testSourceDirectory＆GT;的src / test / java下＆LT; / testSourceDirectory＆GT;
    ＆LT;＆插件GT;
      ＆LT;！ -
                   绑定Maven的组装插件包相
        而不依赖风暴，这将创建一个jar文件
        适合部署到集群中。
        - ＆GT;
      ＆LT;＆插件GT;
        ＆LT;＆的artifactId GT; Maven的组装插件＆LT; / artifactId的＆GT;
        ＆LT;结构＆gt;
          ＆LT; descriptorRefs＆GT;
            ＆LT; descriptorRef＆GT; JAR-具有依赖性和LT; / descriptorRef＆GT;
          ＆LT; / descriptorRefs＆GT;
          ＆LT;归档和GT;
            ＆LT;＆舱单GT;
              ＆LT; mainClass＆GT;＆LT; / mainClass＆GT;
            ＆LT; /清单＆GT;
          ＆LT; /存档＆GT;
        ＆LT; /结构＆gt;
        ＆LT;＆执行GT;
          ＆LT;执行与GT;
            ＆LT;＆ID GT;使组装和LT; / ID＆GT;
            ＆LT;阶段＆gt;包装及LT; /阶段＆gt;
            ＆LT;目标＆GT;
              ＆LT;＆目标GT;单＆LT; /目标＆GT;
            ＆LT; /目标＆GT;
          ＆LT; /执行＆GT;
        ＆LT; /处决＆GT;
      ＆LT; /插件＆GT;
    ＆LT; /插件＆GT;
＆LT; /构建＆GT;





3. MVN包




4. 提交例如-JAR与 - dependencies.jar

I am using maven

i have added the following dependencies

   <dependency> <!-- Spark dependency -->
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-streaming_2.10</artifactId>
      <version>1.1.0</version>
    </dependency>   <dependency> <!-- Spark dependency -->
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-streaming-kafka_2.10</artifactId>
      <version>1.1.0</version>
    </dependency>

I have also added the jar in the code

SparkConf sparkConf = new SparkConf().setAppName("KafkaSparkTest");
JavaSparkContext sc = new JavaSparkContext(sparkConf);
sc.addJar("/home/test/.m2/repository/org/apache/spark/spark-streaming-kafka_2.10/1.0.2/spark-streaming-kafka_2.10-1.0.2.jar");
JavaStreamingContext jssc = new JavaStreamingContext(sc, new Duration(5000)); 

It comples fine with out any error , i am getting the following error when i run through spark-submit, any help is much appreciated. Thanks for your time.

bin/spark-submit --class "KafkaSparkStreaming" --master local[4] try/simple-project/target/simple-project-1.0.jar

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/streaming/kafka/KafkaUtils at KafkaSparkStreaming.sparkStreamingTest(KafkaSparkStreaming.java:40) at KafkaSparkStreaming.main(KafkaSparkStreaming.java:23) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:303) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:55) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: java.lang.ClassNotFoundException: org.apache.spark.streaming.kafka.KafkaUtils at java.net.URLClassLoader$1.run(URLClassLoader.java:366)

解决方案

I meet the same problem, I solved it by build the jar with dependencies.

1. remove "sc.addJar()" in your code.

2. add the code below to pom.xml

   <build>
       <sourceDirectory>src/main/java</sourceDirectory>
       <testSourceDirectory>src/test/java</testSourceDirectory>
       <plugins>
         <!--
                      Bind the maven-assembly-plugin to the package phase
           this will create a jar file without the storm dependencies
           suitable for deployment to a cluster.
          -->
         <plugin>
           <artifactId>maven-assembly-plugin</artifactId>
           <configuration>
             <descriptorRefs>
               <descriptorRef>jar-with-dependencies</descriptorRef>
             </descriptorRefs>
             <archive>
               <manifest>
                 <mainClass></mainClass>
               </manifest>
             </archive>
           </configuration>
           <executions>
             <execution>
               <id>make-assembly</id>
               <phase>package</phase>
               <goals>
                 <goal>single</goal>
               </goals>
             </execution>
           </executions>
         </plugin>
       </plugins>
   </build>    
   

3. mvn package

4. submit the "example-jar-with-dependencies.jar"

这篇关于星火卡夫卡分流问题的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！