为什么Spark应用程序失败，并在线程"main"中出现"Exception"? java.lang.NoClassDefFoundError:... StringDeserializer"? [英] Why does Spark application fail with "Exception in thread "main" java.lang.NoClassDefFoundError: ...StringDeserializer"?

问题描述

我正在开发一个Spark应用程序，该应用程序使用Spark和Java侦听Kafka流.

I am developing a Spark application that listens to a Kafka stream using Spark and Java.

我使用kafka_2.10-0.10.2.1.

I use kafka_2.10-0.10.2.1.

我为Kafka属性设置了各种参数:bootstrap.servers，key.deserializer，value.deserializer等.

I have set various parameters for Kafka properties: bootstrap.servers, key.deserializer, value.deserializer, etc.

我的应用程序可以正常编译，但是当我提交它时，它失败并显示以下错误:

My application compiles fine, but when I submit it, it fails with the following error:

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/kafka/common/serialization/StringDeserializer

我确实将StringDeserializer用于key.deserializer和value.deserializer，所以它确实与我编写应用程序的方式有关.

I do use StringDeserializer for key.deserializer and value.deserializer so it's indeed related to how I wrote my application.

pom.xml中使用的各种Maven依赖项:

Various maven dependencies used in pom.xml:

    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_2.11</artifactId>
        <version>2.1.1</version>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-streaming_2.10</artifactId>
        <version>2.1.1</version>
    </dependency>

    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-streaming-kafka-0-10_2.11</artifactId>
        <version>2.1.1</version>
    </dependency>

我已尝试更新Spark Streaming/Kafka的版本.我找不到任何地方.

I have tried updating the version of spark streaming/kafka. I could not find much anywhere.

推荐答案

spark-streaming_ 2.10

这取决于Scala 2.10

This is dependent upon Scala 2.10

您的其他依赖项正在使用Scala 2.11

Your other dependencies are using Scala 2.11

升级版本是针对当前错误的正确解决方案.

Upgrading the version is the correct solution for the current error.

并确保在streaming-kafka- 0-10 中，此匹配您正在运行的Kafka的版本

And make sure that within streaming-kafka-0-10, this matches the version of Kafka you're running

应用程序编译正常，但是当我尝试提交spark作业时，它显示错误:线程"main"中的异常java.lang.NoClassDefFoundError:

Application is compiling fine but when I am trying to submit the spark job, its showing error: Exception in thread "main" java.lang.NoClassDefFoundError:

默认情况下，maven不包含依赖项jar建立目标

这篇关于为什么Spark应用程序失败，并在线程"main"中出现"Exception"? java.lang.NoClassDefFoundError:... StringDeserializer"?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！